search

planningalerts-scrapers / issues

Only for keeping track of all issues related to scraping
0 stars 0 forks source link

Boroondara City Council #944

Closed mlandauer closed 2 months ago

mlandauer commented 4 months ago

This issue has been automatically created by PlanningAlerts. Only close this issue once the authority is working again on PlanningAlerts.

katska commented 4 months ago

from: https://morph.io/planningalerts-scrapers/multiple_icon boroondara: ERROR: execution expired

boroondara: ERROR: execution expired /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:937:in initialize' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:937:inopen' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:937:in block in connect' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/timeout.rb:103:intimeout' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:935:in connect' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:920:indo_start' /app/vendor/ruby-2.5.1/lib/ruby/2.5.0/net/http.rb:915:in start' /app/vendor/bundle/ruby/2.5.0/gems/net-http-persistent-4.0.1/lib/net/http/persistent.rb:655:instart' /app/vendor/bundle/ruby/2.5.0/gems/net-http-persistent-4.0.1/lib/net/http/persistent.rb:595:in connection_for' /app/vendor/bundle/ruby/2.5.0/gems/net-http-persistent-4.0.1/lib/net/http/persistent.rb:885:inrequest' /app/vendor/bundle/ruby/2.5.0/gems/mechanize-2.7.7/lib/mechanize/http/agent.rb:280:in fetch' /app/vendor/bundle/ruby/2.5.0/gems/mechanize-2.7.7/lib/mechanize.rb:464:inget' /app/lib/icon_scraper.rb:40:in scrape_with_params' /app/lib/icon_scraper.rb:17:inscrape' scraper.rb:14:in block in scrape' scraper.rb:10:ineach' scraper.rb:10:in scrape' scraper.rb:31:in

'

katska commented 4 months ago

Clicking through to accept the terms at https://eservices.boroondara.vic.gov.au/EPlanning/Common/Common/terms.aspx (URL listed in the scraper) is very slow to load or perhaps stuck? yet going to other pages seems fine - clicking on the menu to see recent applications for example https://eservices.boroondara.vic.gov.au/EPlanning/Pages/XC.Track/SearchApplication.aspx?d=thisweek&k=LodgementDate&t=PlnPermit,PlnAppeals,PlnPostPer,PlanPermGr,PlanAmend,PlanAppeal is just fine!

katska commented 3 months ago

@mlandauer anything we can do in the code to allow for very slow to load website?

mlandauer commented 2 months ago

Okay, that's strange. Scraper works locally through a US VPN and running connecting directly. The error it's showing on morph looks a lot like what happens when something's blocked. I'm going to try going via the australian vpn anyway and see if that fixes it. If it does it means they're probably specifically blocking morph.io which would be strange IMHO.

katska commented 2 months ago

woohoo! @mlandauer 🙏 looking good! 🎉