search

planningalerts-scrapers / issues

Only for keeping track of all issues related to scraping
0 stars 0 forks source link

Townsville: migrate custom scraper to icon scraper #57

Open mlandauer opened 4 years ago

mlandauer commented 4 years ago

Currently we can't do that because the xml feed on the site doesn't work. Sigh.

(Or I guess we could add support for scraping via non-xml to the icon scraper. Sigh)

Here's an example of an xml feed not working http://eplanning.townsville.qld.gov.au/Pages/XC.Track/SearchApplication.aspx?d=thismonth&k=LodgementDate&t=PDMCUCode,PDMCUimp,PDOpWorks,PDReconfig,QMCU,QRAL,QOPW,QDBW,QPOS,QEXC,QSPS,QCAR,PDSAMCUse,PDSARecon,PDSAOpWks&o=xml

Similar problem to #56

mlandauer commented 3 years ago

xml feed (above) is still broken

mlandauer commented 2 years ago

Oh it actually looks like some html scraping was added to the icon scraper. Let's give it a try

mlandauer commented 2 years ago

Unfortunately the html scraping code that was added doesn't generalise well to slightly different layouts and there's very little in the way of markup to make it generalise. So, either we need the xml output to get fixed or we need to make another special slightly different version of the hardcoded html scraper for the townsville case.

mlandauer commented 2 years ago

@katska any chance that could get in touch with Townsville council and see if they will fix the xml output? If you could manage to get them to fix it would make things SO much easier. 🙏🏼

mlandauer commented 1 year ago

xml feed still looks broken

katska commented 1 week ago

@mlandauer when you say XML broken can you say more? Planning Alerts is finding DAs! https://www.planningalerts.org.au/authorities/townsville Does something still seem broken? Would love to close this if not.