search

openaustralia / theyvoteforyou

Making parliamentary voting information accessible, understandable, and easy to use so that you can hold your elected representatives to account.
https://theyvoteforyou.org.au/
Other
134 stars 30 forks source link

It appears that aph.gov.au might be blocking our scraper #1477

Open mlandauer opened 5 months ago

mlandauer commented 5 months ago

We wrote the following message to aph by filling in the form at https://www.aph.gov.au/Help/Contact_Us

I’m Matthew Landauer, one of the founders of the charity the OpenAustralia Foundation, a library of Parliamentary, Government and Public information, including OpenAustralia (https://www.openaustralia.org.au) and They Vote For You (https://theyvoteforyou.org.au/).

Both of these services depend on Hansard data that we scrape from aph.gov.au and parlinfo.aph.gov.au. We appreciate the ongoing cooperation and support of APH in facilitating access to Hansard running and maintaining those scrapers for the last 15 years. Most of the time this works seamlessly, and from time to time we report errors to APH and work with notifications of errors to correct errors in the records.

We recently discovered a problem that is preventing the Hansard from being scraped. This problem started around the beginning of December 2023.

We are now seeing regular but intermittent 403 (forbidden) errors on scraping parlinfo urls which work fine when checked in the browser. Here is one such url as an example:

https://parlinfo.aph.gov.au/parlInfo/search/display/display.w3p;adv=yes;orderBy=_fragment_number,doc_date-rev;page=0;query=Dataset%3Ahansards,hansards80%20Date%3A21%2F1%2F2024;rec=0;resCount=Default

Is there by any chance any anti-scraping or anti-bot technology that’s recently been put in place that could be interfering with our scrapers? Could you please unblock us? Alternatively, if it's stemming from another intermittent technical problem, we kindly request your guidance. For example, If there's a workaround available, we would greatly appreciate any guidance you can offer.

We’d really appreciate your help trying to resolve this issue

All the best, Matthew Landauer

I put in some very hacky changes to the parser which makes things a little better but the parser needs to be constantly restarted in https://github.com/openaustralia/openaustralia-parser/commit/906431b2e4ef92d840a5827e35d1ae147e7535da