RECAP sends CL's user cookies to the API endpoint

[elvey]

FYI: I logged into PACER, installed Recap for Chrome, checked my Recap preferences (Notify was&is checked), and then downloaded Document 315/Attach­ment 1/Text of Proposed Order from https://www.courtlistener.com/docket/6183591/united-states-v-manafort/?filed_after=&filed_before=&entry_gte=&entry_lte=&order_by=desc but it seems it failed to upload. FYI.
My Recap icon is blue on the page, https://ecf.dcd.uscourts.gov/doc1/04516611913
I'm wondering what's wrong. No big deal.

[mlissner]

Hm, that worked fine for me (and the doc is now uploaded). Can you reproduce this anywhere else?

[elvey]

I'm not a regular (that is, I'm a very infrequent) user, but I'd be happy to try if you point me to something. I guess if it's a common problem someone else will report it.

[elvey]

I'm a bit confused by your second sentence in your comment on bug #241. Just to be clear, I didn't DO anything to cause the doc referred to in that bug to upload; I just noticed that it had. I guess you're saying that in that bug, the doc showed as uploaded by the time you looked at it, but in this bug, the doc didn't show as uploaded 'till after you uploaded it. Just trying to be helpful there and here. I appreciate you.

[mlissner]

Yeah, in #241 I went to try to fix it, but it was already fixed so I didn't have the ability to reproduce it.

In this bug, can you try to determine if you can upload anything or if it's limited to specific documents?

[johnhawkinson]

@elvey, were you logged into PACER before you started? Or did you get prompted to login after following the "Buy on PACER" link from CourtListener?

[Pascal666]

I think I am also running into this problem. Icon is blue but I do not get the Uploaded Successfully popup and it does not appear anything actually uploads, not dockets nor documents. Errors show up in the JavaScript console. RECAP v1.2.7 on Chrome Version 67.0.3396.87 (Official Build) (64-bit) on Windows 7. I was looking at the docket and documents from near the bottom of https://www.courtlistener.com/docket/5820530/in-re-nexus-6p-products-liability-litigation/?page=2

Docket:

Accept charges:

View document:

3 documents downloaded:

[Pascal666]

I just noticed that my PACER bill shows I was charged for 3 copies of each document and the 3 copies in my Downloads folder are identical. The last JavaScript console log above appears to show that RECAP did several things 3 times.

[johnhawkinson]

@Pascal666, do you have any other browser extensions or addons installed? What you describe above isn't normal, although it's good that there are clear error messages (I haven't run down the stack traces you provided, but it's great that we do have them!). But extension incompatibilities have caused a lot of weird problems, so it's a good first place to start.

[Pascal666]

I only have 28 other extensions running right now, but I guess it is possible one of them could be causing the problem. I'll see if I can do some testing from a different environment, once the quarter changes over tomorrow. Being charged 3 times for each document used up my free allowance for the quarter really fast.

[johnhawkinson]

I'll see if I can do some testing from a different environment, once the quarter changes over tomorrow. Being charged 3 times for each document used up my free allowance for the quarter really fast.

@mlissner: We should, I think, change the extension such that users can run it against test/training servers (e.g. https://dcecf.psc.uscourts.gov/cgi-bin/DktRpt.pl?63112) (with the server debugging flag). That would help with this kind of thing.

[mlissner]

28 other extensions running right now

That sounds like a lot to me, but it's not really the number so much as the type. Look for ones that:

• Inject stuff into every page.
• Modify headers

I guess with 28 extensions you could pretty quickly divide and conquer. I think if you use the testing PACER servers, you could see if you got the errors and if you could make them go away. As a first step, you should probably just see if things work properly in a new profile that just has RECAP installed. Then try using only RECAP in your old profile, then slowly enable your other extensions until things break.

That'd be my approach, at least, and if you were able to do similar, it'd be very helpful. Can you share the extensions you have installed?

[Pascal666]

I think if you use the testing PACER servers, you could see if you got the errors and if you could make them go away.

The RECAP icon does not turn blue on the test server and the JavaScript console shows no indication that it is doing anything. johnhawkinson was just saying it would be nice if the plugin did work on test servers.

Can you share the extensions you have installed?

Sure, though I was wrong on the count. I simply counted the number of extension icons to the right of the address bar. chrome://system/ shows I have an additional nine extensions enabled that do not have icons. It does not appear to list the extensions I have installed but disabled.

ahfgeienlihckogmohjhadlkjgocpleb : Web Store : version 0_2 aknpkdffaafgjchaibgeefbgmgeghloj : Angry Birds : version 1_5_0_8 cjpalhdlnbpafiamejdnhcphjbkeiagm : uBlock Origin : version 1_16_10 dgefbojfgdddnignhmfmnencgiloojpe : Turkopticon : version 3_41 dhdgffkkebhmkfjojejmpbldmpobfkfo : Tampermonkey : version 4_6 edacconmaakjimmfgnblocblbcdcpbko : Session Buddy : version 3_6_3 emehklffcaphknhhfhadkjhpfapcbpco : Strong Password Generator : version 1_6 fdhfobojnimoedpefghekjhckolommed : Chrome2ChromeV2 : version 0_0_5 fdpohaocaechififmbbbbbknoalclacl : Full Page Screen Capture : version 4_3 figkapeodjhdgnpiamleongcmecfjccb : Stop Autoplay for Youtube™ : version 4_1_0 gcbommkclmclpchllfjekcdonpmejbdp : HTTPS Everywhere : version 2018_6_21 gdbofhhdmcladcmmfjolgndfkpobecpg : Don't track me Google : version 4_21 gfdkimpbcpahaombhbimeihdjnejgicl : Feedback : version 1_0 gofhjkjmkpinhpoiabjplobcaignabnl : FlashBlock : version 0_9_34 hcpidejphgpcgfnpiehkcckkkemgneif : Awesome Cookie Manager : version 1_0_0_0 impiikfnffjblkkefnplfonianmboaam : Call From Browser : version 1_25_0 jgpmhnmjbhgkhpbgelalfpplebgfjmbf : Smile Always : version 0_93 jnjfeinjfmenlddahdjdmgpbokiacbbb : Quick Tabs : version 2017_10_8 kbfnbcaeplbcioakkpcpgfkobkghlhen : Grammarly for Chrome : version 14_853_1708 klbibkeccnjlkjkiokjodocebajanakg : The Great Suspender : version 6_30 kmendfapggjehodndflmmgagdbamhnfd : CryptoTokenExtension : version 0_9_73 lcfdefmogcogicollfebhgjiiakbjdje : Disable Extensions Temporarily : version 1_0 mfehgcgbbipciphmccgaenjidiccnmng : Cloud Print : version 0_1 mhjfbmdgcfjbbpaeojofohoefgiehjai : Chrome PDF Viewer : version 1 midkcinmplflbiflboepnahkboeonkam : Reload All Tabs : version 4_0_1 nakplnnackehceedgkgkokbgbmfghain : Fakespot - Analyze Fake Amazon Reviews : version 0_3_2 neajdppkdcdipfabeoofebfddakdcjhd : Google Network Speech : version 1_0 njabckikapfpffapmjgojcnbfjonfjfg : cookies.txt : version 1_14_1 nkeimhogjdpnpccoofpliimaahmaaome : Google Hangouts : version 1_3_8 nkgllhigpcljnhoakjkgaieabnkmgdkb : Don't Fuck With Paste : version 2_4 nmmhkkegccagdldgiimedpiccmgmieda : Chrome Web Store Payments : version 1_0_0_4 occjjkgifpmdgodlplnacmkejpdionan : AutoScroll : version 4_8 ocfjjjjhkpapocigimmppepjgfdecjkb : Storage Area Explorer : version 0_4_2 ocpcmghnefmdhljkoiapafejjohldoga : Context Menu Search : version 2_93 oiillickanjlaeghobeeknbddaonmjnc : RECAP : version 1_2_7 pkedcjkdefgpdelpbcmbmeomcjbeemfm : Chrome Media Router : version 6718_423_0_0 pnjaodmkngahhkoihejjehlcdlnohgmp : RSS Feed Reader : version 7_4_4

[Pascal666]

Disabling Grammarly for https://ecf.cand.uscourts.gov stopped the JavaScript errors on the Docket and Accept Charges screens, and fixed the triple download problem, but the JavaScript error is still there for downloads and the docket and downloadeds still aren't getting sent to RECAP.

Disabling all extensions other than RECAP got rid of the JavaScript error on the downloads page, but the downloaded file still did not get uploaded.

PACER does not charge for opinions, so I have been testing with document 144 at https://www.courtlistener.com/docket/5820530/in-re-nexus-6p-products-liability-litigation/?page=2

I tested on Linux, with most of my extensions enabled, and the downloaded file got uploaded to RECAP just fine.

Windows, no other extensions, doesn't work:

Linux, plenty of other extensions, worked fine:

[Pascal666]

I just tested on Windows again with no extensions enabled other than RECAP, and the error on the downloads page is back:

[mlissner]

Ah, sorry, I thought the testing server worked on the upload side but not on the processing side. My mistake there.

I have an additional nine extensions enabled that do not have icons

FYI, there are two kinds of extensions, those that work globally and those that work on a specific page or pages. The ones that work globally have to have an icon you can see all the time. The ones for pages have an icon you only see in the URL bar when you're on those specific URLs.

Disabling all extensions other than RECAP got rid of the JavaScript error on the downloads page, but the downloaded file still did not get uploaded.

Can you clarify — are you seeing the upload notification and then it's not showing up in CourtListener or are you just not seeing the upload notification? If there's no JS error you should usually see the upload happen. Another thought: Do you have a fast Internet connection? I have DSL and upload is about 10× slower than download, so big things take me a while to upload.

As for the difference between Windows and Linux, that's...troublesome. The OS shouldn't matter, but so I'd expect something else to be the cause. Could you be running different versions of Chrome?

What's your dev background? You seem pretty skilled. Interested in helping more formally?

[Pascal666]

Can you clarify — are you seeing the upload notification and then it's not showing up in CourtListener or are you just not seeing the upload notification? If there's no JS error you should usually see the upload happen. Another thought: Do you have a fast Internet connection? I have DSL and upload is about 10× slower than download, so big things take me a while to upload.

On windows there is no notification and the document does not show up on CourtListener. Yes, my upload is about 10x slower than download, but my download is 500mbps. I would think an upload of 50mbps would be sufficient. I waited hours to see if it would show up too. Showed up instantly on Linux.

What's your dev background? You seem pretty skilled. Interested in helping more formally?

A few decades of programming experience in a dozen languages on as many platforms. I miss vaxen, but NetWare was fun too. Always happy to help fixing bugs. Seems I spend most of my day reporting them anymore. If you could use a laugh, I wrote this a while ago: https://what.thedailywtf.com/topic/8322/bug-report

I figured out the problem. It has nothing to do with Windows vs Linux or other extensions. This would have been a lot easier to track down if the plugin wrote errors and a success message to the javascript console. If you pull up the Chrome extensions page and enable "Developer mode" you will see a "background page" link in the RECAP box that links to chrome-extension://oiillickanjlaeghobeeknbddaonmjnc/_generated_background_page.html . When uploads are not working this log contains a lot of errors like:

Failed to load resource: the server responded with a status of 403 (FORBIDDEN) recap.js:70 RECAP: Ajax error getting document availability. Status: error. Error: FORBIDDEN error @ recap.js:70 www.courtlistener.com/api/rest/v3/recap/:1 Failed to load resource: the server responded with a status of 403 (FORBIDDEN) recap.js:129 RECAP: Ajax error uploading docket. Status: error.Error: FORBIDDEN error @ recap.js:129

Turns out I get the above errors if I am logged into courtlistener.com when RECAP tries to use the API. All I had to do to get RECAP working was log out of courtlistener.com. No idea if it is just my user ID (Pascal) that does not have access to the API or if everyone is running into the problem.

I figured out a way to test RECAP without having to pay a PACER fee. I pulled up a docket on courtlistener.com and searched for Attachments. If the line item only shows "Main Doc­ument" then the list of attachments has not yet been downloaded. Click "Buy on PACER" to see the list for free. You should get the RECAP popup and the document list should show up on courtlistener.com instantly.

Please try logging into courtlistener.com and then try using the RECAP plugin.

[mlissner]

Yep. I just tried this on Chrome 67 and you're totally right, it fails when you're logged in and returns the following JSON:

{"detail":"CSRF Failed: Referer checking failed - no Referer."}

On Firefox, I'm unable to reproduce this problem. I'm going to look into that error a bit more, but the issue seems to be that somehow Chrome isn't using the correct token authentication when uploading the item. Comparing the request in Chrome vs. Firefox, it looks like Chrome is including cookies from your user session while Firefox is not. That could be the issue, but when I do "copy as curl" on the failing request in Chrome, I can't reproduce it using curl.

I'm going to keep digging, but this looks like a major issue — thank you.

[mlissner]

OK, a bit of digging later, there are a few possible solutions:

1. You can modify the requests that chrome sends and thus remove the cookies that are sent. This is an OK solution, but if more than one extension tries to modify the request, the most-recently installed extension wins.

   That means that if somebody installs RECAP and then installs some other extension that modifies requests, RECAP breaks until they re-install. That sucks.

2. There's a property called withCredentials that you can set on XHR's which supposedly makes it so they don't send cookies. But...this doesn't seem to work on web extensions.

3. It's possible to set up a subdomain on CourtListener so that the regular www cookies don't get sent to the extension. I guess this subdomain would be just for the recap extension, and so we'd get the domain working and then update the extension to work with it.

4. The final option is to grant all CL users upload permissions in the API. The good part of this is that we'd finally know who was having problems with their uploads and be able to better help them. That's also the bad part of this.

I'm currently leaning towards option three. I guess we could set up https://recap-api.courtlistener.com, and that should/could work?

This bug also explains a related problem I identified that too many people are getting the "Thanks for using the CourtListener API" welcome emails. This email gets sent after somebody uses the API a few times and because these people are using the API with their CL credentials being sent, they get this welcome email. Not great.

[mlissner]

One other idea, call it idea number 5: The authentication for our API currently is defined as:

# Auth
'DEFAULT_AUTHENTICATION_CLASSES': (
    'rest_framework.authentication.BasicAuthentication',
    'rest_framework.authentication.SessionAuthentication',
    'rest_framework.authentication.TokenAuthentication',
),

Basic auth is the HTTP user/password auth pop up thing FLP loves so much. Session auth uses cookies, and Token auth uses tokens. Right now our requests are failing when we send both cookies and tokens because the session auth tries to do its thing. I wonder if we switch the order here if the Token auth would happen before the Session auth and if that'd fix things.

[mlissner]

This actually sounds promising:

The authentication schemes are always defined as a list of classes. REST framework will attempt to authenticate with each class in the list, and will set request.user and request.auth using the return value of the first class that successfully authenticates.

(from: http://www.django-rest-framework.org/api-guide/authentication/)

[mlissner]

Hot damn, that did it. For details see:

https://github.com/freelawproject/courtlistener/commit/cf1a00794cb2d89e3b22d4fe3bfe721b90f5865c

Thank you again, @Pascal666. Can you confirm that things are now working for you and close the bug if so?

[Pascal666]

No idea if that fixed elvey's problem, but most of what I just tested worked now, but I could not get this $0.10 docket to upload: https://ecf.jpml.uscourts.gov/cgi-bin/DktRpt.pl?112525715685928-L_1_0-1 RECAP icon is blue, but I'm thinking it may be an unrelated issue. I would have expected that docket to be linked to from https://www.courtlistener.com/opinion/2539438/in-re-nebuad-device-privacy-litigation/ but it does not appear to be.

[mlissner]

@elvey I'd be curious to hear if this fixed your issue too?

@Pascal666 I don't know if JPML is supported...should be, but I can't test right now because it keeps timing out on the login page. Grrr. In any case, I think that's a different issue, probably worth filing it if it seems like it's a JPML issue.

[mlissner]

@Pascal666 , this should have gotten its own bug, but JPML is now supported! Had to do some special work for it. https://twitter.com/RECAPtheLaw/status/1017151613255835648

I'm going to close this one down. I think we're good here.

[Pascal666]

Thank you for all your hard work @mlissner. It appears you have gotten an amazing amount done in the past day.

[mlissner]

I have my moments! JPML wasn't too tough...just had to update the data model, document the changes, update the parser, and update the upload verification rules! Done that loop a few times now...

[Pascal666]

Do you maintain a complete list of the difference between the courts supported by PACER and those supported by CL as a todo list?

[mlissner]

No, but I think JPML was the last one aside from appellate.

[elvey]

Sorry I didn't responded to you way back when.

FYI, I just installed RECAP (into Opera Version:80.0.4170.63 (arm64)) and visited: https://ecf.waed.uscourts.gov/doc1/19513521082?caseid=83623&de_seq_num=268&pdf_toggle_possible=1 The Recap icon is Blue, shows Logged into PACER, RECAP is active. Has access to this site.
But it doesn't seem to have sent the file to the free law project. It is displayed in my browser.

I suppose I should try to reproduce in Chrome, with all other plug-ins disabled prior to installing RECAP... As noted in '18, I'm not a regular (that is, I'm a very infrequent) user, and IANAL, and IIRC, have yet to contribute a file successfully.

Thanks for the laugh, @Pascal666! and effort, all!

FYI, 'till I get a round 'tuit: Console shows:

RECAP: Attaching links to all eligible documents (1 found)
utils.js:144 RECAP: Item saved in storage at tabId: 159
content_delegate.js:353 RECAP: Got results from API. Running callback on API results to insert link
content_delegate.js:653 RECAP: Got results from API. Running callback on API results to attach links and icons where appropriate.
content_delegate.js:417 RECAP: Successfully submitted RECAP "View" button form: OK
content_delegate.js:486 RECAP: Successfully got PDF as arraybuffer via ajax request.
utils.js:144 RECAP: Item saved in storage at tabId: 159
core.js:85 Uncaught TypeError: Cannot read properties of null (reading 'offsetHeight')
    at CMECF.MainMenu.setMainContentSize (core.js:85)
    at N (core.js:20)
DevTools failed to load source map: Could not load content for chrome-extension://oiillickanjlaeghobeeknbddaonmjnc/assets/js/bootstrap.bundle.js.map: HTTP error: status code 404, net::ERR_UNKNOWN_URL_SCHEME
73core.js:85 Uncaught TypeError: Cannot read properties of null (reading 'offsetHeight')
    at CMECF.MainMenu.setMainContentSize (core.js:85)
    at N (core.js:20)
CMECF.MainMenu.setMainContentSize @ core.js:85
N @ core.js:20

[mlissner]

Those logs look OK, @elvey, but Opera isn't supported. Give it a try in Chrome or Firefox (or Edge or Safari, honestly), and if it still fails, please file a new bug with the details.

Thanks for the much-belated follow up though. I appreciate the effort.