Closed mlissner closed 6 years ago
Another wrinkle. Of the ~9k attachments we've received, only about 2k have HTML files associated with them. The remainder do not. This can be either because the HTML file was pulled off during an earlier processing run that crashed (which I've seen) or because it wasn't uploaded in the first place.
OK, this is actually kind of not too bad. First, the reason most of these didn't have documents associated with them is because they were processed successfully. I assumed that if they didn't have a valid case ID they'd fail, but that was wrong. These were getting processed just fine even with bad case ID values — it just isn't used.
The 2k items that still have documents associated with them are ones that are for dockets that we don't have for some reason. So that's a lot, but it's not a problem unique to attachment pages.
The fix here, then, is:
The rest of the old data is fine, more or less.
Looking at the client side of this, we find the following fun comment:
// pacer_case_id is not currently used by backend, but send anyway...
So...we knew it wasn't used by the backend. Regardless, the reason it wasn't getting set is because the only source we had for it was the current URL, which is typically of this form:
https://ecf.dcd.uscourts.gov/doc1/04506319222
Or the referrer URL, which is of this form:
https://ecf.dcd.uscourts.gov/cgi-bin/DktRpt.pl?415844360161432-L_1_0-1
Ain't no case numbers there, so we would just send "undefined." In the changes that just landed (above), I've added a new function that will suck the pacer_case_id value from the forms on the page, if that's possible.
This seems bad. Instead they're getting sent with the strings
undefined
ornull
for their pacer_case_id.I'll be able to fix this for old stuff by yanking the ID out of the links or the button, and https://github.com/freelawproject/juriscraper/releases/tag/1.12.3 does exactly that, but something needs to be done about the RECAP client.