Ashford Borough Council

ChopsKingsland commented 11 months ago

Name of Council

Example Address/Postcode

TN23 7SP (Any of the numbered addresses, not businesses)

Additional Information

https://secure.ashford.gov.uk/WasteCollections/CollectionDayLookup/ this website allows you to lookup postcode, then it asks you for address

Verification

[X] I searched for similar issues at https://github.com/robbrad/UKBinCollectionData/issues?q=is:issue and found no duplicates
[X] I have provided a tested working address/postcode/UPRN with bin collections available
[X] I understand that this project is run by volunteer contributors and completion depends on numerous factors - even with a request, we cannot guarantee if/when your council will get a script

OliverCullimore commented 4 months ago

I can't seem to resolve the following SSL protocol error that's happening trying to make a get request to this URL: requests.exceptions.SSLError: HTTPSConnectionPool(host='secure.ashford.gov.uk', port=443): Max retries exceeded with url: /waste/collectiondaylookup/ (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1000)')))

I've tried the usual with no change:

requests.packages.urllib3.disable_warnings()

Testing it with SSL Labs reports it to only support TLS 1.2: https://www.ssllabs.com/ssltest/analyze.html?d=secure.ashford.gov.uk

From this, I attempted to force the request to use TLS 1.2, but the same error persists:

class CustomHttpAdapter(requests.adapters.HTTPAdapter):
    """Transport adapter" that allows us to use custom ssl_context."""

    def __init__(self, ssl_context=None, **kwargs):
        self.ssl_context = ssl_context
        super().__init__(**kwargs)

    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = urllib3.poolmanager.PoolManager(
            num_pools=connections,
            maxsize=maxsize,
            block=block,
            ssl_context=self.ssl_context,
        )

class CouncilClass(AbstractGetBinDataClass):

    def parse_data(self, page: str, **kwargs) -> dict:
        # Start a new session to walk through the form
        s = requests.Session()
        ssl_context = ssl.create_default_context()
        ssl_context.minimum_version = ssl.TLSVersion.TLSv1
        ssl_context.maximum_version = ssl.TLSVersion.TLSv1_2
        ssl_context.options = ssl.PROTOCOL_TLS & ssl.OP_NO_TLSv1_3
        s.mount("https://", CustomHttpAdapter(ssl_context))
        requests.packages.urllib3.disable_warnings()

        # Get our initial session running
        response = s.get("https://secure.ashford.gov.uk/waste/collectiondaylookup/")

Full script with the above attempt in it: https://github.com/OliverCullimore/UKBinCollectionData/tree/389-ashford-borough

@dp247 @robbrad any ideas here or should we resort to Selenium?

robbrad commented 3 months ago

@OliverCullimore resort to Selenium - use branch https://github.com/robbrad/UKBinCollectionData/tree/council_pack_3

robbrad / UKBinCollectionData