fixmyrights / discord-bot

🤖 Discord Bot for the Right to Repair community.
BSD 2-Clause "Simplified" License
4 stars 4 forks source link

LegiScan API results should be persisted on a 1-hour schedule #28

Open j1mbl3s opened 4 years ago

j1mbl3s commented 4 years ago

The API documentation says that, within a 1-hour schedule for search/searchRaw, it will serve unchanged cache data (see page 7).

j1mbl3s commented 4 years ago

Watched bills with changes could be displayed on this schedule while another, less frequent schedule could display the rest of the results, with the scan/query command displaying the results as needed.

x47188 commented 4 years ago

in general we track about 180,000 bills and have an average 3 hour lag between a state publishing any random update and the system picking it up.

State schedule vary wildly from near real-time to once a day, and the Timmy the Intern factor in data entry delay from Clerk staff.

And as it relates to public service Pull API calls there is a 1 hour cache on any specific set of parameters, though 1-2 checks per day is sufficient for most common identification/update approaches

– Sean, from Legiscan

finnbear commented 4 years ago

My argument for not locally caching bills is as follows: The scan command, which does the API queries, is run very infrequently (one to two times a day). It is not a common command to use. The other commands (ignore/watchlist/etc.) operate on the database.json/watchlist, which acts like a cache.