search

freelawproject / recap

This repository is for filing issues on any RECAP-related effort.
https://free.law/recap/
12 stars 4 forks source link

Large dockets fail to show [R] icons for all entries #218

Open johnhawkinson opened 7 years ago

johnhawkinson commented 7 years ago

I accidently ran a 30-page docket report (Waymo from doc 227 instead of doc 2272)! And for some reason I get no [R] boxes at all. Console logs are unhelpful:

RECAP: Attaching links to all eligible documents (9398 found)  content_delegate.js:407:3
RECAP: Saved the pacer_doc_id to pacer_case_id mappings to local storage.  content_delegate.js:42:7

and nothing (relevant to recap) in the background log.

Apparently 9400 docs are too many for the query.

Something better should happen than silent failure. And it's deeply ironic that a person most in need of crowdsourcing would lose out after they inadvertently spent too much money.

Anyhow, we should warn for this condition, not fail silently, until a better solution is found.

mlissner commented 6 years ago

Right now this is set at 300 items per page of results from the API. It's uncommon enough, we can probably push this a bit. My goal at the outset was to be careful here and then slowly expand based on usage and performance. So far, things seem pretty fine.

Other thoughts to resolve this:

  1. Paginate during scrolling (I know you don't like this, but done right, it can work very well without users having any issues). So, when a long page loads, grab the first 300 IDs and send them to the API. As the user scrolls, grab additional chunks based on viewport. Try to stay well ahead of the user's scroll point.

  2. Use pagination on the server side. That is, allow the query of all 10k items, notice that it's paginated, and then slowly paginate in those results by hitting the API as many times as needed for the page.

I actually like the first one better, because it only hits the backend as much as needed (the user never scrolls, we hit the backend once).

We do need some testing though to see how high we can push the count without the server being sad.

mlissner commented 1 year ago

Well, for the record, the nasty API query behind this is officially one of the most impactful queries on our poor, sad database. It turns out that pulling a few hundred items from the DB at one time doesn't make it very happy.

Now that we have @ERosendo on board, I think doing this via scroll makes a lot of sense. We'd want some sort of indicator to show the user that items are loading, but otherwise, it'd be great to be smarter here. I've long de-prioritized this issue since it only impacts users on really big dockets, but I guess it's worth giving it a small boost.

Eduardo, could you provide an estimate of how long it'd take to do this, please?