search

planningalerts-scrapers / issues

Only for keeping track of all issues related to scraping
0 stars 0 forks source link

Yarra City Council #783

Open mlandauer opened 1 year ago

mlandauer commented 1 year ago

This issue has been automatically created by PlanningAlerts. Only close this issue once the authority is working again on PlanningAlerts.

mlandauer commented 1 year ago

Looks like they've moved over to using tech one

malone-c commented 10 months ago

Hi @mlandauer, I'm keen to help out with this. I currently can't access the tech one multi-scraper. Would it be possible to get onboarded? Otherwise, is there another way I can contribute? I have written and MVP scraper with Selenium to check feasibility, but I figure it's better for the project if similar scrapers are integrated. Cheers!

mlandauer commented 5 months ago

Hi @mlandauer, I'm keen to help out with this. I currently can't access the tech one multi-scraper. Would it be possible to get onboarded? Otherwise, is there another way I can contribute? I have written and MVP scraper with Selenium to check feasibility, but I figure it's better for the project if similar scrapers are integrated. Cheers!

Hi @malone-c. Unfortunately we've had to close source a few of the "multi" scrapers because an increasing number of commercial users were just copying the scrapers rather than using and our API. If commercial users are not paying for the API we're unable to make this service free for everyone else. We're just a tiny handful of people doing this in a charity.

The downside for us is that we can't take outside contributions for those scrapers which is awful and I hate. I wish there was another way.

It's a long way of saying sorry. The upside I guess is that I'm working on a fix now. So hopefully things will start working again soon for this authority.

However, if you did feel like contributing to one of the majority of open source scrapers your help would be very much appreciated. Thanks!

mlandauer commented 5 months ago

Ugh. The scraper works locally but fails on morph. After a bit of poking around I figured out it's sitting behind cloudflare. I'm getting so bored of this. I can't be bothered.

katska commented 5 months ago

oh no @mlandauer

katska commented 5 months ago

@mlandauer do you think this means they're misidentifying our scrapers as bad? https://www.cloudflare.com/learning/bots/what-is-data-scraping/

I'd be happy to get in touch with them if you'd offer some technical description about how we're getting the data to help make our case here. I can speak to the non-malicious nature of what and why we're scraping if you can speak to the how. They may be able to advise what we can say to those councils using their services on how to be more discriminate in order to allow our for public purpose scraping.

Worth a go?

katska commented 2 months ago

Missive conversation: https://mail.missiveapp.com/#inbox/conversations/5058e772-df9e-4f9c-9559-5435afc6d46b

katska commented 2 months ago

@malone-c As you may have seen, our problem here is that your council (assuming Yarra City is your local council?) are blocking our ability to access their planning applications programmatically. I've written to them (July 5). If you are local to Yarra City, you might consider getting in touch with the council planners or a local councillor and let them know that you value this service and why it's important to you and/or people in your area. It means more to them to answer to local people I think, than an organisation they don't have a responsibility to service. What do you think?

katska commented 2 months ago

Missive conversation: https://mail.missiveapp.com/#inbox/conversations/c9f5b4e7-1782-4503-8102-0d2e3a5a967f

katska commented 2 months ago

@malone-c Council planners responded with an acknowledgement of my email today.