Buckinghamshire Council Failing - Element not found - URL/page layout change?

adamcarter81 commented 2 months ago

Name of Council

Buckinghamshire / Wycombe

Issue Information

Buckinghamshire / Wycombe council has started failing. A quick look and it looks like the site layout has changed, and I get the following error in HA:

Unexpected error fetching Wycombe data
Unexpected error fetching High Wycombe data
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/helpers/update_coordinator.py", line 354, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/homeassistant/.homeassistant/custom_components/uk_bin_collection/sensor.py", line 150, in _async_update_data
    data = await self.hass.async_add_executor_job(self.ukbcd.run)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 101, in run
    return self.client_code(
           ^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 121, in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 61, in template_method
    bin_data_dict = self.get_and_parse_data(this_url, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 84, in get_and_parse_data
    bin_data_dict = self.parse_data("", url=address_url, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/councils/BuckinghamshireCouncil.py", line 50, in parse_data
    inputElement_pc = driver.find_element(
                      ^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 748, in find_element
    return self.execute(Command.FIND_ELEMENT, {"using": by, "value": value})["value"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 354, in execute
    self.error_handler.check_response(response)
  File "/srv/homeassistant/lib/python3.12/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="COPYOFECHOCOLLECTIONDATES_ADDRESSSELECTION_ADDRESSSELECTIONPOSTCODE"]"}
  (Session info: chrome-headless-shell=127.0.6533.99); For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors#no-such-element-exception

Checking the site: https://www.buckinghamshire.gov.uk/waste-and-recycling/find-out-when-its-your-bin-collection/ using postcode HP13 7BA (any address after) and the element isn't found , it appears the site has undergone an update

Verification

[X] I searched for similar issues at https://github.com/robbrad/UKBinCollectionData/issues?q=is:issue and found no duplicates
[X] I have checked my address/postcode/UPRN works on the council's website
[X] I have provided a detailed explanation of the issue as well as steps to replicate the issue
[X] I understand that this project is run by volunteer contributors therefore completion of this issue cannot be guaranteed

robbrad commented 2 months ago

Yes looks like it

preator67 commented 1 month ago

Can confirm: the layout has changed (I wrote the original implementation for the old one).

They look to have changed the system used to provide the data. I did some preliminary testing, but I still can't find a way to eliminate the Selenium element: options are now i) parsing it out of the page, ii) a pdf file, or iii) a .ics file, but I still can't see how to get at them without going through the dropdowns.

This is on my radar, but I don't have time to get to it at the moment. I'll circle back at some stage, if nobody can fix it beforehand.

robbrad / UKBinCollectionData