search

unitedstates / inspectors-general

Collecting reports from Inspectors General across the US federal government.
https://sunlightfoundation.com/blog/2014/11/07/opengov-voices-opening-up-government-reports-through-teamwork-and-open-data/
Creative Commons Zero v1.0 Universal
107 stars 21 forks source link

Revise Amtrak scraper #249

Closed divergentdave closed 9 years ago

divergentdave commented 9 years ago

The Amtrak website was redesigned recently, this takes care of the changes. (Yay, no tables!) I spot-checked the contents, and it looks like all the reports are still there, just organized and presented differently.

Much like with the USPS site, pagination of reports is not stable. If multiple reports with the same published date span two pages of search results, then those reports will show up in different orders from page load to page load. Thus, fetching each page once may miss some reports. For this scraper, I solved the issue by getting all of the results for each query on one page.

konklone commented 9 years ago

I love it when scrapers get simpler. Down with pagination!