Add Claims support for recap.email

[albertisfu]

Reviewing more errors related to freelawproject/courtlistener#2243 on Sentry.

This time the reason for recent errors are notifications that belong to a Claim, some examples are below:

Their content is different from a normal pacer document. Seems the Claim number is a parallel number from the Document number in the same case.

Also contains additional data like: Creditor Name, Amount Claimed, Amount Secured, Amount Priority.

[mlissner]

Is there an easy way to stop getting errors when we get these emails?

[albertisfu]

Yeah, we could just pass an empty response from the email report when these messages come in. So that would mean a no-valid notification, so the error won't be logged into sentry. If that's ok I can apply this temporary fix.

[mlissner]

Sounds simple enough. :)

[mlissner]

We have a potential client that wants this now. It's not a blocker for them, but it'd be nice if we could do it.

[albertisfu]

Great, so should we move it to my backlog?

[mlissner]

Yeah, I haven't figured that out yet. Could you take a look at how hard this feature would be — which pieces it needs and how complicated they are? The full package for the customer will be:

• Juriscraper parser for the email (99% sure we lack this, but that it's not too hard)
• DB models to hold the data (60% sure we have something, but not sure if it's good enough)
• View code to merge it into the DB (40% sure we lack this, no idea if it's good enough)
• API to pull the data from the DB (no idea)
• Webhook support (I assume our current one would work, but in case that assumption is wrong)

[mlissner]

I'll put this on your backlog just for analysis for now.

[albertisfu]

Yes, of course, I'll do the analysis and report it here.

[albertisfu]

I've checked this, these are my findings.

• There's a ClaimsRegister report in Juriscraper used to parse HTML Claims Register pages, so I could base on that report to extend the NotificationEmail report in order to support Claims notifications, so this part seems straightforward to complete.
• About DB models, yes we have Claim and ClaimHistory we should use both to merge the data from the notification but many fields will keep empty since the Claim notification only has the following fields:

1. Case Name
2. Case Number: we can use this to identify the docket to which the Claim is linked to.
3. Creditor Name: creditor_details in DB
4. Claim Number: claim_number in DB
5. Amount Claimed: amount_claimed in DB
6. Amount Secured: secured_claimed in DB
7. Amount Priority: priority_claimed in DB

And the Claim document that we can get using the magic link and store in a ClaimHistory object.

• We have code to merge Claims Register pages, so it should be straightforward to build the merger for recap.email.

• API for Claims, we need to build this. Claims are linked to a Docket but following PACER and the models we have, Claims entries should be independent of docket entries. So I think we should build an endpoint API for Claims similar to the docket_entries one that lists the claim details, the docket to which is linked and claim history documents instead of recap documents.

• About webhooks for Claims notifications, yes we would need some changes to use the new API endpoints for claims in order to serialize claims instead of docket entries. However, since the payload structure will change maybe it would be needed to create a different type of event or add a new field to differentiate docket entries from claims.

I think everything above could be done in a week or so if there are no surprises to solve.

I have a question, Do we need to show claims to users on CourtListener site?

[mlissner]

Thanks Alberto, this sounds good, and I appreciate the work estimate as well.

I have a question, Do we need to show claims to users on CourtListener site?

Yes, in theory, we should do it, but it's one of those things we haven't gotten around to. I guess this would be the time, when we do this project.

[albertisfu]

Seems that this is the next task on my backlog, so should we start building the things that we described above (parsing claims email notifications, merging into DB, sending docket alerts and webhooks)?

And in the next step (Issue/PR) should we define how to show Claims to users on Courtslitener site?

[mlissner]

See related issues:

https://github.com/freelawproject/juriscraper/issues/543

https://github.com/freelawproject/juriscraper/issues/630

https://github.com/freelawproject/juriscraper/issues/638

https://github.com/freelawproject/courtlistener/issues/1005

https://github.com/freelawproject/juriscraper/issues/650