Request for United Kingdom - Walsall Council support

[xavierxxiii]

https://cag.walsall.gov.uk/BinCollections/

Sample postcode: WS10 8HS

[dt215git]

Looks simple enough, have you had a go at it yourself?

[xavierxxiii]

really new to this so no idea where to start but if you think it's simple, I will see what I can do. Any pointers will be appreciated

[dt215git]

There are plenty of online/youtube on web scraping, so they'll give some general ideas, but most people will open up the web developer tools window in their browser and watch the requests/responses being sent to get a feel for what's going on as you interact with the page. No harm in having a play with that and seeing what you notice - sometimes you'll see an underlying api being used, sometimes you'll just see it loading html. Where you go from there depends on what you see.

In terms of the Walsall site here are a few things that jump out:

• Putting in the post code you gave results in a list of properties within that postcode. What do you notice about the urls if you hover over those addresses? That gives you an indication there might be an alternative way of getting collection schedules for individual addresses.
• If you click on the address, you get another page with links to the individual collection schedules. Is there any pattern to those urls? That gives you an indication of how you might access the individual collection schedules.
• If you click on one of the collections, you get a page with a table containing days and dates. Those dates are what you're after, so is there a way to scrape those? A tutorial on scraping table data can help with that one.

There's no harm in looking at the .md and .py files in the repo to see how other UK councils have been dealt with, you'll see how others have approached things and there's a decent amount of copy, paste & adapt going on.

It may be simpler to have a play in a stand alone script in the first instance. If you can get that extracting the collection dates then it can be tweaked to work within this add on.

[xavierxxiii]

Super helpful. Thank you!