search

openaustralia / planningalerts

Find out and have your say about what's being built and knocked down in your area.
https://www.planningalerts.org.au
Other
95 stars 51 forks source link

ScraperWiki scrapers can take ages to run (PA-460) #473

Closed oajirabot closed 9 years ago

oajirabot commented 10 years ago

Currently in the proposed code to run ScraperWiki scrapers locally (https://github.com/openaustralia/planningalerts-parsers/pull/7) every time you hit a scraper it runs a scrape. This can take an incredibly long time as SW scrapers are designed differently to our local scrapers (e.g. they may fetch every single application, over multiple index pages).

It would be nice to devise a way so that they don't take an incredibly long time to run.

Issue details imported from Jira:

Detail Value
Assignee Unassigned
Reporter Henare Degan
Created 2013-09-03 17:38:08
Updated 2013-09-09 21:33:57

Imported Jira Comments

Henare Degan - 2013-09-03 17:46:19

The first step is to see how big an issue this is by doing a test run and seeing how long it takes. It could be a really serious issue if it takes too long or it could be less important if it doesn't drag things out too long.

Henare Degan - 2013-09-03 19:52:00

OK I'm running a test scrape now.

Henare Degan - 2013-09-03 23:45:51

Bugger, it didn't get far because we can only geocode 2,500 addresses a day with our test key.

BTW over the last 9 days, 4 days before and 5 days after the 1.9.3 upgrade, the scrape (and email) time has varied between 2:05 and 3:52 so I guess aiming for under 4 hours is good.

deploy@kedumba:/srv/www/test.planningalerts.org.au/app/current$ RAILS_ENV=production time bundle exec rake planningalerts:applications:scrape
Faraday: you may want to install system_timer for reliable timeouts
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/faraday-0.8.7/lib/faraday/request/multipart.rb:4: warning: already initialized constant DEFAULT_BOUNDARY
DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/1/4/rails-3-2-0-rc2-has-been-released. (called from /srv/www/test.planningalerts.org.au/app/releases/20130828133621/Rakefile:7)
Scraping 80 authorities
Error Validation failed: Description can't be blank while trying to save application A003702171 for Brisbane City Council, QLD. So, skipping
Error Validation failed: Description can't be blank while trying to save application A003701569 for Brisbane City Council, QLD. So, skipping
Error Validation failed: Description can't be blank while trying to save application A003698397 for Brisbane City Council, QLD. So, skipping
2 new applications found for Brisbane City Council, QLD with date from 2013-08-24 to 2013-09-03
3 applications errored for Brisbane City Council, QLD with date from 2013-08-24 to 2013-09-03
Took 47 s to collect applications from Brisbane City Council, QLD
22 new applications found for Boroondara City Council, VIC with date from 2013-08-24 to 2013-09-03
Took 104 s to collect applications from Boroondara City Council, VIC
194 new applications found for Logan City Council, QLD with date from 2013-08-24 to 2013-09-03
Took 419 s to collect applications from Logan City Council, QLD
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/act?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/act?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/act?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/act?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/act?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/act?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/act?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/act?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/act?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/act?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/act?year=2013&month=8&day=24. So, skipping
0 new applications found for ACT Planning & Land Authority, ACT with date from 2013-08-24 to 2013-09-03
Took 11 s to collect applications from ACT Planning & Land Authority, ACT
20 new applications found for Wollongong City Council, NSW with date from 2013-08-24 to 2013-09-03
Took 133 s to collect applications from Wollongong City Council, NSW
1 new applications found for Department of Planning and Local Government, SA with date from 2013-08-24 to 2013-09-03
Took 6 s to collect applications from Department of Planning and Local Government, SA
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kogarah?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kogarah?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kogarah?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kogarah?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kogarah?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kogarah?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kogarah?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kogarah?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kogarah?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kogarah?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kogarah?year=2013&month=8&day=24. So, skipping
0 new applications found for Kogarah City Council, NSW with date from 2013-08-24 to 2013-09-03
Took 2 s to collect applications from Kogarah City Council, NSW
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kuringgai?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kuringgai?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kuringgai?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kuringgai?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kuringgai?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kuringgai?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kuringgai?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kuringgai?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kuringgai?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kuringgai?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kuringgai?year=2013&month=8&day=24. So, skipping
0 new applications found for Ku-ring-gai Council, NSW with date from 2013-08-24 to 2013-09-03
Took 3 s to collect applications from Ku-ring-gai Council, NSW
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/albury?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/albury?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/albury?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/albury?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/albury?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/albury?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/albury?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/albury?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/albury?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/albury?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/albury?year=2013&month=8&day=24. So, skipping
0 new applications found for Albury City Council, NSW with date from 2013-08-24 to 2013-09-03
Took 241 s to collect applications from Albury City Council, NSW
72 new applications found for Warringah Council, NSW with date from 2013-08-24 to 2013-09-03
Took 70 s to collect applications from Warringah Council, NSW
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/wagga_wagga?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/wagga_wagga?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/wagga_wagga?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/wagga_wagga?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/wagga_wagga?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/wagga_wagga?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/wagga_wagga?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/wagga_wagga?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/wagga_wagga?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/wagga_wagga?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/wagga_wagga?year=2013&month=8&day=24. So, skipping
0 new applications found for City of Wagga Wagga, NSW with date from 2013-08-24 to 2013-09-03
Took 235 s to collect applications from City of Wagga Wagga, NSW
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/griffith?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/griffith?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/griffith?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/griffith?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/griffith?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/griffith?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/griffith?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/griffith?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/griffith?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/griffith?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/griffith?year=2013&month=8&day=24. So, skipping
0 new applications found for Griffith City Council, NSW with date from 2013-08-24 to 2013-09-03
Took 134 s to collect applications from Griffith City Council, NSW
28 new applications found for Wyong Shire Council, NSW with date from 2013-08-24 to 2013-09-03
Took 34 s to collect applications from Wyong Shire Council, NSW
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/hornsby?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/hornsby?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/hornsby?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/hornsby?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/hornsby?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/hornsby?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/hornsby?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/hornsby?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/hornsby?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/hornsby?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/hornsby?year=2013&month=8&day=24. So, skipping
0 new applications found for Hornsby Shire Council, NSW with date from 2013-08-24 to 2013-09-03
Took 2 s to collect applications from Hornsby Shire Council, NSW
11 new applications found for Fraser Coast Regional Council, QLD with date from 2013-08-24 to 2013-09-03
Took 38 s to collect applications from Fraser Coast Regional Council, QLD
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/ipswich?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/ipswich?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/ipswich?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/ipswich?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/ipswich?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/ipswich?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/ipswich?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/ipswich?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/ipswich?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/ipswich?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/ipswich?year=2013&month=8&day=24. So, skipping
0 new applications found for City of Ipswich, QLD with date from 2013-08-24 to 2013-09-03
Took 2 s to collect applications from City of Ipswich, QLD
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/lockyer_valley?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/lockyer_valley?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/lockyer_valley?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/lockyer_valley?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/lockyer_valley?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/lockyer_valley?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/lockyer_valley?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/lockyer_valley?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/lockyer_valley?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/lockyer_valley?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/lockyer_valley?year=2013&month=8&day=24. So, skipping
0 new applications found for Lockyer Valley Regional Council, QLD with date from 2013-08-24 to 2013-09-03
Took 236 s to collect applications from Lockyer Valley Regional Council, QLD
63 new applications found for Redland City Council, QLD with date from 2013-08-24 to 2013-09-03
Took 216 s to collect applications from Redland City Council, QLD
15 new applications found for Moreland City Council, VIC with date from 2013-08-24 to 2013-09-03
Took 42 s to collect applications from Moreland City Council, VIC
0 new applications found for Bankstown City Council, NSW with date from 2013-08-24 to 2013-09-03
Took 0 s to collect applications from Bankstown City Council, NSW
6 new applications found for Bellingen Shire Council, NSW with date from 2013-08-24 to 2013-09-03
Took 25 s to collect applications from Bellingen Shire Council, NSW
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/blacktown?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/blacktown?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/blacktown?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/blacktown?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/blacktown?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/blacktown?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/blacktown?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/blacktown?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/blacktown?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/blacktown?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/blacktown?year=2013&month=8&day=24. So, skipping
0 new applications found for Blacktown City Council, NSW with date from 2013-08-24 to 2013-09-03
Took 3 s to collect applications from Blacktown City Council, NSW
29 new applications found for Blue Mountains City Council, NSW with date from 2013-08-24 to 2013-09-03
Took 9 s to collect applications from Blue Mountains City Council, NSW
31 new applications found for Bundaberg Regional Council, QLD with date from 2013-08-24 to 2013-09-03
Took 8 s to collect applications from Bundaberg Regional Council, QLD
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/charters_towers?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/charters_towers?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/charters_towers?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/charters_towers?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/charters_towers?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/charters_towers?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/charters_towers?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/charters_towers?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/charters_towers?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/charters_towers?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/charters_towers?year=2013&month=8&day=24. So, skipping
0 new applications found for Charters Towers Regional Council, QLD with date from 2013-08-24 to 2013-09-03
Took 2 s to collect applications from Charters Towers Regional Council, QLD
Error Geokit::TooManyQueriesError while trying to save application X/590/2013 for City of Ballarat, VIC. So, skipping
16 new applications found for City of Ballarat, VIC with date from 2013-08-24 to 2013-09-03
1 applications errored for City of Ballarat, VIC with date from 2013-08-24 to 2013-09-03
Took 24 s to collect applications from City of Ballarat, VIC
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/burnside?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/burnside?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/burnside?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/burnside?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/burnside?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/burnside?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/burnside?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/burnside?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/burnside?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/burnside?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/burnside?year=2013&month=8&day=24. So, skipping
0 new applications found for City of Burnside, SA with date from 2013-08-24 to 2013-09-03
Took 16 s to collect applications from City of Burnside, SA
70 new applications found for City of Cockburn, WA with date from 2013-08-24 to 2013-09-03
Took 51 s to collect applications from City of Cockburn, WA
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/geelong_city?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/geelong_city?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/geelong_city?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/geelong_city?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/geelong_city?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/geelong_city?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/geelong_city?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/geelong_city?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/geelong_city?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/geelong_city?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/geelong_city?year=2013&month=8&day=24. So, skipping
0 new applications found for City of Greater Geelong, VIC with date from 2013-08-24 to 2013-09-03
Took 4 s to collect applications from City of Greater Geelong, VIC
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kingston?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kingston?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kingston?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kingston?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kingston?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kingston?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kingston?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kingston?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kingston?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kingston?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/kingston?year=2013&month=8&day=24. So, skipping
0 new applications found for City of Kingston, VIC with date from 2013-08-24 to 2013-09-03
Took 4 s to collect applications from City of Kingston, VIC
Error execution expired while getting data from url http://test.scrapers.planningalerts.org.au/marion?year=2013&month=9&day=3. So, skipping
0 new applications found for City of Marion, SA with date from 2013-08-24 to 2013-09-03
Took 87 s to collect applications from City of Marion, SA
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/melbourne_city?year=2013&month=9&day=3. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/melbourne_city?year=2013&month=9&day=2. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/melbourne_city?year=2013&month=9&day=1. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/melbourne_city?year=2013&month=8&day=31. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/melbourne_city?year=2013&month=8&day=30. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/melbourne_city?year=2013&month=8&day=29. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/melbourne_city?year=2013&month=8&day=28. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/melbourne_city?year=2013&month=8&day=27. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/melbourne_city?year=2013&month=8&day=26. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/melbourne_city?year=2013&month=8&day=25. So, skipping
Error 500 Internal Server Error while getting data from url http://test.scrapers.planningalerts.org.au/melbourne_city?year=2013&month=8&day=24. So, skipping
0 new applications found for Melbourne City Council, VIC with date from 2013-08-24 to 2013-09-03
Took 213 s to collect applications from Melbourne City Council, VIC
Error Geokit::TooManyQueriesError while trying to save application PLP/2013/590 for City of Ryde, NSW. So, skipping
59 new applications found for City of Ryde, NSW with date from 2013-08-24 to 2013-09-03
1 applications errored for City of Ryde, NSW with date from 2013-08-24 to 2013-09-03
Took 24 s to collect applications from City of Ryde, NSW
Error Geokit::TooManyQueriesError while trying to save application X/590/2013 for City of Stonnington, VIC. So, skipping
rake aborted!
Geokit::TooManyQueriesError
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/bundler/gems/geokit-a541ba216b80/lib/geokit/services/google3.rb:99:in `json2GeoLoc'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/bundler/gems/geokit-a541ba216b80/lib/geokit/services/google3.rb:67:in `do_geocode'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/bundler/gems/geokit-a541ba216b80/lib/geokit/geocoders.rb:96:in `geocode'
/srv/www/test.planningalerts.org.au/app/releases/20130828133621/app/models/location.rb:18:in `geocode'
/srv/www/test.planningalerts.org.au/app/releases/20130828133621/app/models/application.rb:170:in `geocode'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:429:in `_run__914400612__save__4__callbacks'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `send'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `__run_callback'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `send'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `run_callbacks'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/callbacks.rb:264:in `create_or_update'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/persistence.rb:84:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/validations.rb:50:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:259:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:313:in `with_transaction_returning_status'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:208:in `transaction'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:259:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:258:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/associations/has_many_association.rb:16:in `insert_record'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/autosave_association.rb:348:in `save_collection_association'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/autosave_association.rb:337:in `each'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/autosave_association.rb:337:in `save_collection_association'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/autosave_association.rb:191:in `add_autosave_association_callbacks'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/autosave_association.rb:161:in `instance_eval'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/autosave_association.rb:161:in `autosave_associated_records_for_applications'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `_run__989560290__update__4__callbacks'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `send'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `__run_callback'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:385:in `_run_update_callbacks'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `send'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `run_callbacks'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/callbacks.rb:272:in `update'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/persistence.rb:348:in `create_or_update'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/callbacks.rb:264:in `create_or_update'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:414:in `_run__989560290__save__4__callbacks'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `send'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `__run_callback'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `send'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `run_callbacks'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/callbacks.rb:264:in `create_or_update'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/persistence.rb:84:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/validations.rb:50:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:259:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:313:in `with_transaction_returning_status'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:208:in `transaction'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:259:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/transactions.rb:258:in `save'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/persistence.rb:181:in `update_attribute'
/srv/www/test.planningalerts.org.au/app/releases/20130828133621/app/models/authority.rb:15:in `add'
/srv/www/test.planningalerts.org.au/app/releases/20130828133621/app/models/authority.rb:142:in `collect_applications_date_range'
/srv/www/test.planningalerts.org.au/app/releases/20130828133621/app/models/authority.rb:132:in `each'
/srv/www/test.planningalerts.org.au/app/releases/20130828133621/app/models/authority.rb:132:in `collect_applications_date_range'
/srv/www/test.planningalerts.org.au/app/releases/20130828133621/app/models/authority.rb:114:in `collect_applications'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/core_ext/benchmark.rb:5:in `ms'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activesupport-3.2.13/lib/active_support/core_ext/benchmark.rb:5:in `ms'
/srv/www/test.planningalerts.org.au/app/releases/20130828133621/app/models/authority.rb:113:in `collect_applications'
/srv/www/test.planningalerts.org.au/app/releases/20130828133621/app/models/application.rb:57:in `collect_applications'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/relation/delegation.rb:6:in `each'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/relation/delegation.rb:6:in `__send__'
/srv/www/test.planningalerts.org.au/app/shared/bundle/ruby/1.8/gems/activerecord-3.2.13/lib/active_record/relation/delegation.rb:6:in `each'
/srv/www/test.planningalerts.org.au/app/releases/20130828133621/app/models/application.rb:57:in `collect_applications'
/srv/www/test.planningalerts.org.au/app/releases/20130828133621/lib/tasks/planningalerts.rake:16
Tasks: TOP => planningalerts:applications:scrape
(See full trace by running task with --trace)
Command exited with non-zero status 1
79.75user 10.03system 41:39.89elapsed 3%CPU (0avgtext+0avgdata 423840maxresident)k
24inputs+3352outputs (1major+79104minor)pagefaults 0swaps
deploy@kedumba:/srv/www/test.planningalerts.org.au/app/current$

Henare Degan - 2013-09-06 00:42:46

After going through every scraper I've found 3 scrapers that can run for a very long time, most other scrapers seem pretty well behaved (see notes in https://github.com/openaustralia/planningalerts-parsers/commit/02534ff0ce2ac9554b6172bf5e9f7684ec86a086).

For these 3 scrapers they can hold up the scrape from the app for 11 minutes, as The Hills one did in my tests. We also don't get any data from them because the script's just busy doing the scrape.

henare commented 9 years ago

Not an issue on morph.io