search

planningalerts-scrapers / issues

Only for keeping track of all issues related to scraping
0 stars 0 forks source link

Brisbane City Council: No data received since 30 Sep 2021 #449

Closed mlandauer closed 2 years ago

mlandauer commented 3 years ago

This issue has been automatically created by PlanningAlerts for Brisbane City Council

It uses the scraper planningalerts-scrapers/BCC_DEVELOPMENT_APPLICATIONS

Only close this issue once the authority is working again on PlanningAlerts. Otherwise a new issue will just automatically created. Also, if there is a duplicate issue close the other one in favour of this one.

Front logo Front conversations

MutazAshhab commented 2 years ago

The Brisbane scraper works locally.

The issue is really it is not able to resolve an old node.js version. Im guessing updating the node version in the package.json and re-generating the package-lock.json file will do it.

katska commented 2 years ago

@mlandauer this scraper covers a large populated area! Also local community group has been in touch (among others) to report it's not working. Any update?

katska commented 2 years ago

@Taz17 Is the problem keeping this out of production fixed now? A lot of people are covered by this one!

MutazAshhab commented 2 years ago

@katska no, I suspected it was an issue with morph but when I went to test it I was not able to add a new scraper on morph (bug, ran into a CanCan::AccessDenied: You are not authorized to access this page error). Here is the slack conversation

katska commented 2 years ago

@Taz17 thanks. I see that. Is that what you could see via honeybadger.io? That's the last mention I see in the conversation.

mlandauer commented 2 years ago

Fixing this involves fixing nodejs on morph.io (or working around the problem by rewriting the scraper in something other than nodejs)

mlandauer commented 2 years ago

Now finally fixed. In trying to fix another problem on morph I realised that the unit tests weren't passing and that was because the nodejs stuff was broken. So decided to try to fix that first.

It turned out it couldn't find the version of nodejs because an api lookup was failing. It turned out it was failing because there was an expired root certificate for the ancient version of ubuntu that is running the scrapers. So https://github.com/openaustralia/buildstep/commit/1033f663bff788978e144fae3bbbda058b059b3f fixed the problem