Open dp247 opened 1 year ago
Added a uid to the input.json for Bolton but the council is still not working - there seams to be some blocking happening - maybe a user agent update would help
Bolton is fixed according to https://robbrad.github.io/UKBinCollectionData/3.9/463/#suites/98d3104e051c652961429bf95fa0b5d6/e916aa0c1a17e4/
Chilterns is working according to https://robbrad.github.io/UKBinCollectionData/3.9/466/#suites/98d3104e051c652961429bf95fa0b5d6/331174cc580460a7/
Some of the councils look like they are working but just don't have schema files. While adding blank schema files would help add parity it would silently hide the non-working councils
Note: it could be a json is output but the scrape is bad in which case it looks like it's working or has scraped without error
@robbrad on checking the Allure reports after #296 merged I noticed it looks like only the 3.9 one has been updating since around February as the executor of the 3.8 report we're seeing was https://github.com/robbrad/UKBinCollectionData/actions/runs/4272808681
CheshireEast Works 60% of the time - may follow up on a different topic to improve reliability
CroydonCouncil works but is sporadic
East Riding working but not via integration tests
Fenland works via Custom Comp. in HA but not on int tests
More Test info after running int tests locally
================================== short test summary info ===================================
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[CheshireEastCouncil-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, '...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[HuntingdonDistrictCouncil-None-None] - AssertionError: True
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[KingstonUponThamesCouncil-None-None] - ValueError: time data 'Friday, 5 January\n (this
collection has been adjusted...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[BexleyCouncil-http://selenium:4444-local] - selenium.common.exceptions.TimeoutException: Message:
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[GlasgowCityCouncil-None-None] - AssertionError: True
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MalvernHillsDC-None-None] - ValueError: time data 'Please refer to the christmas collection calendar.' does not match...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[LondonBoroughHounslow-None-None] - AssertionError: True
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[EastLindseyDistrictCouncil-http://selenium:4444-local] - IndexError: list index out of range
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MertonCouncil-None-None] - AttributeError: 'NoneType' object has no attribute 'find_all'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[HaltonBoroughCouncil-http://selenium:4444-local] - selenium.common.exceptions.ElementClickInterceptedException: Message: element click interc...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[EastRidingCouncil-http://selenium:4444-local] - selenium.common.exceptions.StaleElementReferenceException: Message: stale element referenc...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MidSussexDistrictCouncil-None-None] - AttributeError: 'NoneType' object has no attribute 'find_all_next'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MiltonKeynesCityCouncil-None-None] - AttributeError: 'str' object has no attribute 'text'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthumberlandCouncil-http://selenium:4444-local] - AttributeError: 'NoneType' object has no attribute 'replace'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthWestLeicestershire-http://selenium:4444-local] - ValueError: time data 'Tue 2nd Jan' does not match format '%a %dth %b'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[FenlandDistrictCouncil-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, '...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SouthOxfordshireCouncil-None-None] - ValueError: Error parsing bin data: time data 'Your usual collection day is different this...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SouthTynesideCouncil-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, '...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SwaleBoroughCouncil-None-None] - requests.exceptions.SSLError: HTTPSConnectionPool(host='swale.gov.uk', port=443): Max retr...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[TorbayCouncil-None-None] - ValueError: No bin data found for provided UPRN.
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[ValeofGlamorganCouncil-None-None] - json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[WarwickDistrictCouncil-None-None] - AttributeError: 'NoneType' object has no attribute 'next_element'FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[WelhatCouncil-None-None] - AttributeError: 'str' object has no attribute 'text'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[StHelensBC-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, '...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NeathPortTalbotCouncil-http://selenium:4444-local] - selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to loc...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[WestSuffolkCouncil-http://selenium:4444-local] - selenium.common.exceptions.TimeoutException: Message:
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthEastDerbyshireDistrictCouncil-http://selenium:4444-local] - selenium.common.exceptions.TimeoutException: Message:
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthEastLincs-None-None] - AttributeError: 'str' object has no attribute 'text'
=================== 28 failed, 94 passed, 16 warnings in 509.42s (0:08:29) ===================
Latest Stats
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[KingstonUponThamesCouncil-None-None] - AssertionError: True
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[FenlandDistrictCouncil-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthEastLincs-None-None] - AttributeError: 'str' object has no attribute 'text'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MidSussexDistrictCouncil-None-None] - AttributeError: 'NoneType' object has no attribute 'find_all_next'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[LondonBoroughHounslow-None-None] - AssertionError: True
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MiltonKeynesCityCouncil-None-None] - AttributeError: 'str' object has no attribute 'text'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MertonCouncil-None-None] - AttributeError: 'NoneType' object has no attribute 'find_all'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[LiverpoolCityCouncil-None-None] - ValueError: time data 'Tomorrow' does not match format '%A, %d %B'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SouthNorfolkCouncil-None-None] - AssertionError: True
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SouthTynesideCouncil-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MalvernHillsDC-None-None] - ValueError: time data 'Please refer to the christmas collection calendar.' does not match format '%A %d/%m/%Y'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NeathPortTalbotCouncil-http://selenium:4444-local] - selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="ContentPlaceHolderDefau...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[StHelensBC-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[CheshireEastCouncil-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[WarwickDistrictCouncil-None-None] - AttributeError: 'NoneType' object has no attribute 'next_element'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[WelhatCouncil-None-None] - AttributeError: 'str' object has no attribute 'text'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[ValeofGlamorganCouncil-None-None] - json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthWestLeicestershire-http://selenium:4444-local] - ValueError: time data 'Tue 23rd Jan' does not match format '%a %dth %b'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[HuntingdonDistrictCouncil-None-None] - AssertionError: True
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[HaltonBoroughCouncil-http://selenium:4444-local] - selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element is not clickable at point (110, 683)
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SwaleBoroughCouncil-None-None] - requests.exceptions.SSLError: HTTPSConnectionPool(host='swale.gov.uk', port=443): Max retries exceeded with url: /bins-littering-and-the-environment/bins/collection-day...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[WestSuffolkCouncil-http://selenium:4444-local] - selenium.common.exceptions.TimeoutException: Message:
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthEastDerbyshireDistrictCouncil-http://selenium:4444-local] - selenium.common.exceptions.TimeoutException: Message:
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthumberlandCouncil-http://selenium:4444-local] - AttributeError: 'NoneType' object has no attribute 'replace'
========================================================== 24 failed, 100 passed, 16 warnings in 91.57s (0:01:31) ===========================================================```
Latest Stats
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[KingstonUponThamesCouncil-None-None] - AssertionError: True FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[FenlandDistrictCouncil-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthEastLincs-None-None] - AttributeError: 'str' object has no attribute 'text' FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MidSussexDistrictCouncil-None-None] - AttributeError: 'NoneType' object has no attribute 'find_all_next' FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[LondonBoroughHounslow-None-None] - AssertionError: True FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MiltonKeynesCityCouncil-None-None] - AttributeError: 'str' object has no attribute 'text' FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MertonCouncil-None-None] - AttributeError: 'NoneType' object has no attribute 'find_all' FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[LiverpoolCityCouncil-None-None] - ValueError: time data 'Tomorrow' does not match format '%A, %d %B' FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SouthNorfolkCouncil-None-None] - AssertionError: True FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SouthTynesideCouncil-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MalvernHillsDC-None-None] - ValueError: time data 'Please refer to the christmas collection calendar.' does not match format '%A %d/%m/%Y' FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NeathPortTalbotCouncil-http://selenium:4444-local] - selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="ContentPlaceHolderDefau... FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[StHelensBC-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[CheshireEastCouncil-None-None] - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[WarwickDistrictCouncil-None-None] - AttributeError: 'NoneType' object has no attribute 'next_element' FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[WelhatCouncil-None-None] - AttributeError: 'str' object has no attribute 'text' FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[ValeofGlamorganCouncil-None-None] - json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthWestLeicestershire-http://selenium:4444-local] - ValueError: time data 'Tue 23rd Jan' does not match format '%a %dth %b' FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[HuntingdonDistrictCouncil-None-None] - AssertionError: True FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[HaltonBoroughCouncil-http://selenium:4444-local] - selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element is not clickable at point (110, 683) FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SwaleBoroughCouncil-None-None] - requests.exceptions.SSLError: HTTPSConnectionPool(host='swale.gov.uk', port=443): Max retries exceeded with url: /bins-littering-and-the-environment/bins/collection-day... FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[WestSuffolkCouncil-http://selenium:4444-local] - selenium.common.exceptions.TimeoutException: Message: FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthEastDerbyshireDistrictCouncil-http://selenium:4444-local] - selenium.common.exceptions.TimeoutException: Message: FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthumberlandCouncil-http://selenium:4444-local] - AttributeError: 'NoneType' object has no attribute 'replace' ========================================================== 24 failed, 100 passed, 16 warnings in 91.57s (0:01:31) ===========================================================```
Getting this via email was scary
Updating Failing Councils list
======================================================================== short test summary info ========================================================================
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[BracknellForestCouncil] - SystemExit: 1
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[CheshireEastCouncil] - requests.exceptions.SSLError: HTTPSConnectionPool(host='online.cheshireeast.gov.uk', port=443): Max retries exceeded with url: /MyCollectionDay/SearchByAjax/GetBart...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[CroydonCouncil] - ValueError: Code 5: Failed to get bin data. Too many requests. Please wait a few minutes before trying again.
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[BradfordMDC] - IndexError: string index out of range
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[GedlingBoroughCouncil] - KeyError: 'Garden%20Waste%20J-2024.pdf'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[GlasgowCityCouncil] - requests.exceptions.SSLError: HTTPSConnectionPool(host='www.glasgow.gov.uk', port=443): Max retries exceeded with url: /forms/refuseandrecyclingcalendar/PrintCalend...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[ManchesterCityCouncil] - jsonschema.exceptions.ValidationError: [] should be non-empty
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MertonCouncil] - AttributeError: 'NoneType' object has no attribute 'find_all'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MiltonKeynesCityCouncil] - AttributeError: 'NoneType' object has no attribute 'find_all'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NorthTynesideCouncil] - AttributeError: 'NoneType' object has no attribute 'select'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[EastRidingCouncil] - selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <div class="accordion closed meta-none " id="when-and-wher...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[BarnetCouncil] - selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element is not clickable at point (69, 976)
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[EastLindseyDistrictCouncil] - selenium.common.exceptions.TimeoutException: Message:
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[GatesheadCouncil] - selenium.common.exceptions.TimeoutException: Message:
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SouthOxfordshireCouncil] - jsonschema.exceptions.ValidationError: [] should be non-empty
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SouthTynesideCouncil] - requests.exceptions.SSLError: HTTPSConnectionPool(host='www.southtyneside.gov.uk', port=443): Max retries exceeded with url: /apiserver/ajaxlibrary/ (Caused by SSLE...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[StHelensBC] - requests.exceptions.SSLError: HTTPSConnectionPool(host='www.sthelens.gov.uk', port=443): Max retries exceeded with url: /article/3473/Check-your-collection-dates (C...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[StroudDistrictCouncil] - jsonschema.exceptions.ValidationError: [] should be non-empty
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[KirkleesCouncil] - selenium.common.exceptions.TimeoutException: Message:
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[MidSussexDistrictCouncil] - AttributeError: 'NoneType' object has no attribute 'find_all_next'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[NeathPortTalbotCouncil] - selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="ContentPlaceHolderD...
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[ValeofGlamorganCouncil] - json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[TorridgeDistrictCouncil] - jsonschema.exceptions.ValidationError: [] should be non-empty
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[ValeofWhiteHorseCouncil] - jsonschema.exceptions.ValidationError: Additional properties are not allowed ('colour' was unexpected)
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[RushcliffeBoroughCouncil] - jsonschema.exceptions.ValidationError: [] should be non-empty
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[RushmoorCouncil] - json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[SwaleBoroughCouncil] - AttributeError: 'NoneType' object has no attribute 'find_all'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[KnowsleyMBCouncil] - ValueError: time data '' does not match format '%A %d/%m/%Y'
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[ThreeRiversDistrictCouncil] - selenium.common.exceptions.TimeoutException: Message:
FAILED uk_bin_collection/tests/step_defs/test_validate_council.py::test_scenario_outline[WestSuffolkCouncil] - jsonschema.exceptions.ValidationError: Additional properties are not allowed ('colour' was unexpected)
======================================================== 30 failed, 137 passed, 17 warnings in 440.81s (0:07:20) ========================================================
Tasks