Hand off recap.email to @albertisfu

[albertisfu]

These are my questions for now about the Todo tasks in the project:

Create celery task for processing emails

When lambda POST to /api/rest/v3/recap-email/
EmailProcessingQueueViewSet process the POST request and triggers: do_recap_document_fetch that parses the email using S3NotificationEmail and obtain docket_entries So a notification email could have many docket entries or just one? For each docket entry, it should be one recap document?

• Where can I find a real example of a notification email? -I found this example https://github.com/freelawproject/juriscraper/issues/381#issuecomment-822978115
  but I’m not sure if is the same kind of notification email or is a different type because in this example I couldn’t see the list of dockets or recap documents links

• I found the task process_recap_email in cl/recap/task.py but I couldn’t find the code where you get the PDF document to download and processes it, so I assume that this is part of what needs to be done?

• I understand that each notification email should contain at least one first look link in order to download the PDF document, this first look link opens a page with an IFRAME where is necessary to get the file using the methods you mention: juriscraper.pacer.reports.BaseReport.download_pdf juriscraper.pacer.reports.BaseReport._query_pdf_download -So with the examples of notification emails, I think I’ll be able to understand better about the magic link and how to handle it.

• I couldn’t find the code to resend the email notification to the real user email account with recap file attached, so this is also a part of what needs to be done?

API key based permission set for Recap Email & Lambda Functions
From what I saw, right now the API doesn’t need authentication, so it’s necessary to add an auth method through a Bearer Token and use RECAPUploaders permissions, is that right?

Create homepage for recap.email
Is this page going to live outside Courtlistener?
What’s sealed content?

Develop scheme for each user to have an recap.email email address
I saw that you already have the signal to create the recap email based on username, this signal is triggered when a new user is created.
Is it already considered how to assign the recap.email for existing users?

[albertisfu]

Some updates:

• I could find an email notification in S3 where I could see the magic link but I still have the question about if a notification email could have many docket entries or just one? because the one I saw just had one.

• It would be helpful to have some real email notifications with a first look link valid.

• I found that process_recap_email calls fetch_pacer_doc_by_rd task to try to get the PACER PDF Does this work just in the case it's a free document? or what's the difference by downloading the document by the magic link?

• Do you have an example of the email to be sent to the real user email after getting the PDF?

[mlissner]

OK, sorry to leave you hanging here, Alberto. @tewen, if you can check my work on these questions from Alberto, that'd be great too.

A few replies:

So a notification email could have many docket entries or just one?

I think just one for now. IIRC, the reason for it using an array is so that the JSON format could be the same as for other types of objects coming from PACER.

Where can I find a real example of a notification email?

The example of an NEF that you found is good. You can find others in the test fixtures for Juriscraper, too:

https://github.com/freelawproject/juriscraper/tree/main/tests/examples/pacer/nef

I couldn’t find the code where you get the PDF document

Looks like you found fetch_pacer_doc_by_rd. I think that should work, but it needs to be upgraded to use the one-time "magic" links, as mentioned here: https://github.com/freelawproject/courtlistener/issues/1708#issuecomment-915632763

So with the examples of notification emails, I think I’ll be able to understand better about the magic link and how to handle it.

Yes. We don't have any usable magic links, because everybody always...uses them, and they're only usable once. That said, when we have all our other code ready to go, we have a partner lined up that is willing to turn on this system. When he does, I think we should be ready to race ahead to get his content either with code or manually. It's going to be a bit like debugging code in production since there's no other way to get unused magic links.

I couldn’t find the code to resend the email notification to the real user email account with recap file attached

Yes, this needs to be done.

Do you have an example of the email to be sent to the real user email after getting the PDF?

No, though we have a few other emails that we send, so you could use those as starting places. Our emails are not very pretty though, so if you know tools for making better (compliant) HTML emails, I'd be happy to consider them.

From what I saw, right now the API doesn’t need authentication, so it’s necessary to add an auth method through a Bearer Token and use RECAPUploaders permissions, is that right?

Precisely.

Is [https://recap.email] going to live outside Courtlistener?

Probably the best thing to do here is to set up recap.email to redirect to a page on https://free.law/. That'd keep all our products in a single place, and it'd make sure that people that saw the emails could figure out what's going on with them. I had grander visions of doing a whole website for recap.email, but it feels excessive in hindsight.

Is it already considered how to assign the recap.email for existing users?

It's considered and completed (by hand). I just closed the issue. :)

OK, I think that's it. Back to you Alberto!

[albertisfu]

Perfect, thank you for your answers @mlissner

So about the one-time link and PDF download, just to confirm, I assume that a link like this: https://ecf.ared.uscourts.gov/doc1/02715212035?caseid=120574&de_seq_num=83&magic_num=99963705 Would open a page with an IFRAME like the described here: https://github.com/freelawproject/courtlistener/issues/1708#issuecomment-915082412 This IFRAME shows the PDF that is the one that needs to be downloaded, is that right?

• So the first task I’ll be working on is to complete this PDF Download and check it’s correctly associated with a docket.

• The second task I’ll be working on is to add the authentication method to rest/v3/recap-email/ endpoint.

• The third task I’ll be working on is to compose the email notification for the recap.email user. About the attached file, I’m not familiar with the average size of a PACER PDF, so what file size threshold do you think would be good to decide if attaching the file or not? -I found this information: https://www.outlook-apps.com/maximum-email-size/ about the common file size for sent and received attached files, I’m not sure if the data is 100% accurate. -So if a PDF file size is over the threshold we define I assume instead of attaching the file we would put the S3 link to be downloaded, is that right?

• The fourth task I’ll be working on is the recap page, so if it will be a section inside free. law the code should be here https://github.com/freelawproject/free.law isn’t it? When I come to this task, it would be helpful to have an example or the actual content for the next points: -The funnel for anonymous donations -Why it's important and how it works -Info about webhooks and APIs -Info about sealed content

[mlissner]

...The first task...

Before you start building more functionality into it, I wonder if you'd benefit from testing what's there and making sure it works. The stuff that has landed so far came via a number of small(ish) PRs, and hasn't really been tested very well.

---

About the attached file, I’m not familiar with the average size of a PACER PDF, so what file size threshold do you think would be good to decide if attaching the file or not?

Yeah, this is a good question. I guess I'd say if it's greater than 750kb we should do it as a link and if smaller, then do it as an attachment? That should separate scanned docs so they don't get attached, but I mostly pulled that number out of my hat.

So if a PDF file size is over the threshold we define I assume instead of attaching the file we would put the S3 link to be downloaded, is that right?

Exactly.

The fourth task I’ll be working on is the recap page

That sounds good. We can get into that later.

[albertisfu]

@mlissner as we were commenting about django-ses it provides some webhook events for bounce email, complaint received, send email, delivery email, open email, click received.

To receive these notifications it's necessary to use Amazon SNS service additionally to Amazon SES.

• Should we handle these notifications in Courtlistener to stop sending recap email notifications to problematic addresses like bounced or complained to avoid affecting recap.email sender reputation on SES?

• If so, should we ban those problematic email addresses after one event (bounce or complaint) or should we set a threshold?

There are two types of bounce email:

• Soft Bounce: The ISP cannot deliver the email to the recipient because of a temporary condition in this case Amazon SES retries the email for an extended period of time (no bounce notification is triggered for the sender), If Amazon SES cannot deliver the email in that time period, it sends a bounce notification to the sender (I understand in this stage it converts to Hard Bounce)

• Hard Bounce: The email is rejected by the ISP because of a persistent condition or rejected by Amazon SES because the email address is on the Amazon SES suppression list. An email address is on the Amazon SES suppression list if it has recently caused a hard bounce for any Amazon SES customer. A hard bounce with an ISP can occur because the recipient's address is invalid. A hard bounce notification is sent from the ISP back to Amazon SES, which notifies the sender through email or through Amazon SNS

[mlissner]

Should we handle these notifications in Courtlistener to stop sending recap email notifications to problematic addresses like bounced or complained to avoid affecting recap.email sender reputation on SES?

Probably, yes.

If so, should we ban those problematic email addresses after one event (bounce or complaint) or should we set a threshold?

There's probably a smart way to do this, since email hosts are not always the most reliable and networks sometimes fail. Probably we focus only on hard bounces since the soft ones seem to be handled by SES.

We wouldn't want to ban somebody forever if their host was offline for 24 hours. Maybe we do exponential backoff sending? The first time it hard bounces, we don't send another for five minutes, then 10, then 20, then 40, 80, 160, 320, etc. If any goes through, we reset it to send everything? There might be more thoughtful ways of doing this and implementing it might be a pain. It's probably also worth showing a banner to the person if they're logged in that says they've got a problem that needs to be fixed?

[johnhawkinson]

There's probably a smart way to do this, since email hosts are not always the most reliable and networks sometimes fail. Probably we focus only on hard bounces since the soft ones seem to be handled by SES.

We wouldn't want to ban somebody forever if their host was offline for 24 hours. Maybe we do exponential backoff sending? The first time it hard bounces, we don't send another for five minutes, then 10, then 20, then 40, 80, 160, 320, etc. If any goes through, we reset it to send everything? There might be more thoughtful ways of doing this and implementing it might be a pain.

I don't think any of this makes sense? The email subsystem (here Amazon SES) keeps track of backoff and retries and redelivery and all that. Just let Amazon worry about all that stuff.

It's probably also worth showing a banner to the person if they're logged in that says they've got a problem that needs to be fixed?

Sure, definitely a good thing to do.

[mlissner]

I don't think any of this makes sense? The email subsystem (here Amazon SES) keeps track of backoff and retries and redelivery and all that. Just let Amazon worry about all that stuff.

The idea is to let SES do the backoff on softfails, but when it reaches a hardfail, then what? Do you keep sending emails to that address, or do you give up or what? If you give up, for how long?

[johnhawkinson]

I'm sorry, I misunderstood the proposal to be for us to handle the soft failure in some way.

As for hard failures, I have all too often (okay, once is enough) been the victim of multi-day email system outages beyond my control (e.g. over a long weekend) and been truly annoyed at the mailing lists and services that silently unsubscribed me as a result. The reality is that permanent failures are not always "permanent."

I think this really depends on Amazon's policies. In a perfect world, we would ignore hard failures and carry on. But if Amazon construes that as bad behavior and punishes us for it, then we should apply some mitigation.

[mlissner]

Yeah, so I think that lends itself towards something like an exponential backoff on hard fails, with a max of something like five or maybe ten days. I think any email outage greater than that would be such a crisis that us not sending things would be the least of their worries.

It's not just about amazon's policies around this either. If we're sending a lot of hard fails to Gmail, they may be more likely to score us as spam as a result, regardless of AWS's situation.

[albertisfu]

I've been playing with django-ses and reading documentation about SES notifications through AWS SNS, there are some Django signals to receive SNS notifications for events like Bounces, Complaint, Send, Delivery, Open and Click.

There are three types of bounces: Undetermined, Permanent and Transient and some sub-types: https://docs.aws.amazon.com/ses/latest/dg/notification-contents.html#bounce-object

Bounce notifications are for hard bounces (Permanent), and for soft bounces that SES stopped trying to deliver (Undetermined or Transient).

• Type: Undetermined -Sub-type: Undetermined The recipient's email provider sent a bounce message. The bounce message didn't contain enough information for Amazon SES to determine the reason for the bounce.

Hard Bounces

• Type: Permanent -Sub-type: General The recipient's email provider sent a hard bounce message, but didn't specify the reason for the hard bounce. AWS SES strongly recommend doesn't send email again to those recipients: Important When you receive this type of bounce notification, you should immediately remove the recipient's email address from your mailing list. Sending messages to addresses that produce hard bounces can have a negative impact on your reputation as a sender. If you continue sending email to addresses that produce hard bounces, we might pause your ability to send additional email.

• Type: Permanent -Sub-type: NoEmail The intended recipient's email provider sent a bounce message indicating that the email address doesn't exist. AWS SES strongly recommend doesn't send email again to those recipients.

• Type: Permanent -Sub-type: Suppressed The recipient's email address is on the Amazon SES suppression list because it has a recent history of producing hard bounces.

• Type: Permanent -Sub-type: OnAccountSuppressionList Amazon SES has suppressed sending to this address because it is on the account-level suppression list.

Soft Bounces

• Type: Transient -Sub-type: General The recipient's email provider sent a general bounce message. You might be able to send a message to the same recipient in the future if the issue that caused the message to bounce is resolved.

• Type: Transient -Sub-type: MailboxFull The recipient's email provider sent a bounce message because the recipient's inbox was full. You might be able to send to the same recipient in the future when the mailbox is no longer full.

• Type: Transient -Sub-type: MessageTooLarge The recipient's email provider sent a bounce message because message you sent was too large. You might be able to send a message to the same recipient if you reduce the size of the message.

• Type: Transient -Sub-type: ContentRejected The recipient's email provider sent a bounce message because the message you sent contains content that the provider doesn't allow. You might be able to send a message to the same recipient if you change the content of the message.

• Type: AttachmentRejected -Sub-type: ContentRejected The recipient's email provider sent a bounce message because the message contained an unacceptable attachment. For example, some email providers may reject messages with attachments of a certain file type, or messages with very large attachments. You might be able to send a message to the same recipient if you remove or change the content of the attachment.

• Complaints Most email client programs provide a button labeled "Mark as Spam," or similar, which moves the message to a spam folder, and forwards it to the email provider. Additionally, most email providers maintain an abuse address (e.g., abuse@example.net), where users can forward unwanted email messages and request that the email provider take action to prevent them. In both of these cases, the recipient is making a complaint. If the email provider concludes that you are a spammer, and Amazon SES has a feedback loop set up with the email provider, then the email provider will send the complaint back to Amazon SES. When Amazon SES receives such a complaint, it forwards the complaint to you either by email or by using an Amazon SNS notification, depending on how you have your system set up. We recommend that you do not make repeated delivery attempts to email addresses that generate complaints.

It's a best practice that you maintain a bounce rate under 5% and a complaint rate under 0.1%.

In SES panel exist a Suppression list, by default SES put in here all Permanent bounced email and complaints, but it's possible to change this configuration to put inside just bounced emails or complained emails, it's also possible to deactivate this suppression list, but I think it could be a good a second layer of protection to avoid sending email to 'Permanent' bounce email and complaints.

Taking into consideration previous data I think we should consider:

• If we receive a hard bounce (Permanent), we shouldn't send future emails to those email addresses.
• If we receive a complaint, we shouldn't send future emails to those email addresses. To avoid affecting sender reputation.

About not hard bounces as Undetermined and Transient bounces, we might define how to handle each type of bounces:

• Undetermined: In this type, we won't know the reason for the bounce, so should we try to send the same email in the next few days? or should we just ignore this email without taking further action about future emails for this user

• General: In this type, we won't know the exact reason for the bounce (it could be an active automatic response rule), so should we try to send the same email in the next few days? or should we just ignore this email without taking further action about future emails for this user

• MailboxFull: In this case, we depend on the recipient to solve the problem, should we try to send the same email in the next few days? should we pause all emails for this user for a few days? or should we just ignore this email without taking further action about future emails for this user

  • MessageTooLarge: I think this could be a very rare case because we are going to set a threshold to avoid sending large files attached to emails, but if it happens it could means that the recipient's email provider has set a very small email size limit, so should we discard this email and try to send it without any attachment and next emails that we send to this user also should be without attachments?

• ContentRejected: should we just ignore this email without taking further action about future emails to this user?

• AttachmentRejected: similar to MessageTooLarge, should we discard this email and try to send it without any attachment, and next emails that we send to this user also should be without attachments?

[mlissner]

This is a great summary, thanks Alberto. Let's talk tomorrow. I think my reaction is to mostly trust AWS's guidance rather than second-guess it.

[mlissner]

Topic above continued in: https://github.com/freelawproject/courtlistener/issues/1942. I'm going to close this issue because it's hitting too many topics and we can split them off in their own more focused threads.