robbrad / UKBinCollectionData

UK Council Bin Collection Data Parser Outputting Bin Data as a JSON
MIT License
114 stars 75 forks source link

Data only refreshing on service reload or HA restart #767

Open raldred opened 1 month ago

raldred commented 1 month ago

My understanding is that the HouseholdBinCoordinator should run every 12 hours to grab the latest data.

Mine isn't.

It only seems to run on the initial load of the integration. Either by restarting HA or by reloading the service from the device screen.

Any ideas? I can't see anything in the logs.

dp247 commented 1 month ago

Please post the logs

raldred commented 1 month ago

So turns out that it was just timing out due to some images that timed out on the target council website. I have just blocked the offending URLs on my selenium runner host. Very strange that it seemed to work after a reload. Sorry for wasting your time.

raldred commented 3 days ago

I'm reopening this as there's definitely something not right... There isn't a huge amount of logs generated when enabling debug for this component but it seems like it just doesn't attempt to fetch updated data unless the integration is reloaded.

home-assistant_uk_bin_collection.log

11th Aug @ 13:02 ``` 2024-08-11 13:02:19.791 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor Bottles and cans (Blue Bin) 2024-08-11 13:02:19.792 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-08-13 2024-08-11 13:02:19.792 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home Bottles and cans (Blue Bin) 2024-08-11 13:02:19.792 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: 2 2024-08-11 13:02:19.792 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: Next Week: Tuesday 2024-08-11 13:02:19.796 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor Garden waste (Grey Bin) 2024-08-11 13:02:19.797 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-08-13 2024-08-11 13:02:19.797 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home Garden waste (Grey Bin) 2024-08-11 13:02:19.797 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: 2 2024-08-11 13:02:19.797 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: Next Week: Tuesday 2024-08-11 13:02:19.799 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor Paper and card (Brown Bin) 2024-08-11 13:02:19.800 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-08-13 2024-08-11 13:02:19.800 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home Paper and card (Brown Bin) 2024-08-11 13:02:19.800 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: 2 2024-08-11 13:02:19.801 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: Next Week: Tuesday 2024-08-11 13:02:19.803 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor General waste (Green Bin) 2024-08-11 13:02:19.803 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-08-20 2024-08-11 13:02:19.803 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home General waste (Green Bin) 2024-08-11 13:02:19.804 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: 9 2024-08-11 13:02:19.804 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: Future: 2024-08-20 ```

Multiple similar blocks of logs where nothing changes apart from the _days attribute

15th Aug @ 12:02: ``` 2024-08-15 12:02:20.199 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor Bottles and cans (Blue Bin) 2024-08-15 12:02:20.200 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-08-13 2024-08-15 12:02:20.200 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home Bottles and cans (Blue Bin) 2024-08-15 12:02:20.201 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: -2 2024-08-15 12:02:20.201 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: This Week: Tuesday 2024-08-15 12:02:20.274 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor Garden waste (Grey Bin) 2024-08-15 12:02:20.275 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-08-13 2024-08-15 12:02:20.275 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home Garden waste (Grey Bin) 2024-08-15 12:02:20.276 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: -2 2024-08-15 12:02:20.276 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: This Week: Tuesday 2024-08-15 12:02:20.279 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor Paper and card (Brown Bin) 2024-08-15 12:02:20.280 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-08-13 2024-08-15 12:02:20.280 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home Paper and card (Brown Bin) 2024-08-15 12:02:20.280 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: -2 2024-08-15 12:02:20.281 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: This Week: Tuesday 2024-08-15 12:02:20.292 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor General waste (Green Bin) 2024-08-15 12:02:20.293 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-08-20 2024-08-15 12:02:20.293 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home General waste (Green Bin) 2024-08-15 12:02:20.293 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: 5 2024-08-15 12:02:20.294 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: Next Week: Tuesday ```

Only after a reload of the service on the integrations screen does it actually go and pull the latest data.

After reload today 15th Aug @ 12:32 ``` 2024-08-15 12:33:20.422 INFO (MainThread) [custom_components.uk_bin_collection] [UKBinCollection] Data Supplied: {'uprn': 'REDACTED', 'postcode': 'REDACTED', 'web_driver': 'http://192.168.86.15:4444', 'headless': True, 'local_browser': False, 'skip_get_url': True, 'url': 'https://myaccount.chorley.gov.uk/wastecollections.aspx', 'name': 'Home', 'council': 'ChorleyCouncil'} 2024-08-15 12:33:20.422 INFO (MainThread) [custom_components.uk_bin_collection] [UKBinCollection] Setting up UK Bin Collection Data for council: ChorleyCouncil 2024-08-15 12:33:20.422 INFO (MainThread) [custom_components.uk_bin_collection] [UKBinCollection] Config entry data: {'uprn': 'REDACTED', 'postcode': 'REDACTED', 'web_driver': 'http://192.168.86.15:4444', 'headless': True, 'local_browser': False, 'skip_get_url': True, 'url': 'https://myaccount.chorley.gov.uk/wastecollections.aspx', 'name': 'Home', 'council': 'ChorleyCouncil'} 2024-08-15 12:33:20.424 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Setting up UK Bin Collection Data platform. 2024-08-15 12:33:20.424 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Supplied: {'uprn': 'REDACTED', 'postcode': 'REDACTED', 'web_driver': 'http://192.168.86.15:4444', 'headless': True, 'local_browser': False, 'skip_get_url': True, 'url': 'https://myaccount.chorley.gov.uk/wastecollections.aspx', 'name': 'Home', 'council': 'ChorleyCouncil'} 2024-08-15 12:33:20.425 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] UKBinCollectionApp args: ['ChorleyCouncil', 'https://myaccount.chorley.gov.uk/wastecollections.aspx', '--uprn=REDACTED', '--postcode=REDACTED', '--web_driver=http://192.168.86.15:4444', '--skip_get_url'] 2024-08-15 12:33:20.429 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Args set 2024-08-15 12:33:20.429 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] UKBinCollectionApp Init 2024-08-15 12:33:20.429 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] UKBinCollectionApp Init Refresh 2024-08-15 12:33:20.430 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] UKBinCollectionApp Updating 2024-08-15 12:33:20.430 INFO (MainThread) [custom_components.uk_bin_collection] [UKBinCollection] Successfully set up UK Bin Collection Data for council: ChorleyCouncil 2024-08-15 12:33:28.522 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] UKBinCollectionApp: { "bins": [ { "type": "General waste (Green Bin)", "collectionDate": "20/08/2024" }, { "type": "Bottles and cans (Blue Bin)", "collectionDate": "27/08/2024" }, { "type": "Garden waste (Grey Bin)", "collectionDate": "27/08/2024" }, { "type": "General waste (Green Bin)", "collectionDate": "03/09/2024" }, { "type": "Bottles and cans (Blue Bin)", "collectionDate": "10/09/2024" }, { "type": "Garden waste (Grey Bin)", "collectionDate": "10/09/2024" }, { "type": "Paper and card (Brown Bin)", "collectionDate": "10/09/2024" }, { "type": "General waste (Green Bin)", "collectionDate": "17/09/2024" }, { "type": "Bottles and cans (Blue Bin)", "collectionDate": "24/09/2024" }, { "type": "Garden waste (Grey Bin)", "collectionDate": "24/09/2024" }, { "type": "General waste (Green Bin)", "collectionDate": "01/10/2024" }, { "type": "Bottles and cans (Blue Bin)", "collectionDate": "08/10/2024" }, { "type": "Garden waste (Grey Bin)", "collectionDate": "08/10/2024" }, { "type": "Paper and card (Brown Bin)", "collectionDate": "08/10/2024" } ] } 2024-08-15 12:33:28.523 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Supplied: {'bins': [{'type': 'General waste (Green Bin)', 'collectionDate': '20/08/2024'}, {'type': 'Bottles and cans (Blue Bin)', 'collectionDate': '27/08/2024'}, {'type': 'Garden waste (Grey Bin)', 'collectionDate': '27/08/2024'}, {'type': 'General waste (Green Bin)', 'collectionDate': '03/09/2024'}, {'type': 'Bottles and cans (Blue Bin)', 'collectionDate': '10/09/2024'}, {'type': 'Garden waste (Grey Bin)', 'collectionDate': '10/09/2024'}, {'type': 'Paper and card (Brown Bin)', 'collectionDate': '10/09/2024'}, {'type': 'General waste (Green Bin)', 'collectionDate': '17/09/2024'}, {'type': 'Bottles and cans (Blue Bin)', 'collectionDate': '24/09/2024'}, {'type': 'Garden waste (Grey Bin)', 'collectionDate': '24/09/2024'}, {'type': 'General waste (Green Bin)', 'collectionDate': '01/10/2024'}, {'type': 'Bottles and cans (Blue Bin)', 'collectionDate': '08/10/2024'}, {'type': 'Garden waste (Grey Bin)', 'collectionDate': '08/10/2024'}, {'type': 'Paper and card (Brown Bin)', 'collectionDate': '08/10/2024'}]} and type of data is 2024-08-15 12:33:28.524 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Next Collection Dates: {'General waste (Green Bin)': '20/08/2024', 'Bottles and cans (Blue Bin)': '27/08/2024', 'Garden waste (Grey Bin)': '27/08/2024', 'Paper and card (Brown Bin)': '10/09/2024'} 2024-08-15 12:33:28.524 DEBUG (MainThread) [custom_components.uk_bin_collection.sensor] Finished fetching Home data in 8.095 seconds (success: True) 2024-08-15 12:33:28.524 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] UKBinCollectionApp Init Refresh complete 2024-08-15 12:33:28.524 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor General waste (Green Bin) 2024-08-15 12:33:28.525 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-08-20 2024-08-15 12:33:28.525 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home General waste (Green Bin) 2024-08-15 12:33:28.525 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: 5 2024-08-15 12:33:28.525 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: Next Week: Tuesday 2024-08-15 12:33:28.525 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor Bottles and cans (Blue Bin) 2024-08-15 12:33:28.526 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-08-27 2024-08-15 12:33:28.526 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home Bottles and cans (Blue Bin) 2024-08-15 12:33:28.526 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: 12 2024-08-15 12:33:28.526 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: Future: 2024-08-27 2024-08-15 12:33:28.526 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor Garden waste (Grey Bin) 2024-08-15 12:33:28.527 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-08-27 2024-08-15 12:33:28.527 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home Garden waste (Grey Bin) 2024-08-15 12:33:28.527 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: 12 2024-08-15 12:33:28.527 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: Future: 2024-08-27 2024-08-15 12:33:28.527 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor Paper and card (Brown Bin) 2024-08-15 12:33:28.527 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-09-10 2024-08-15 12:33:28.528 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home Paper and card (Brown Bin) 2024-08-15 12:33:28.530 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: 26 2024-08-15 12:33:28.530 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: Future: 2024-09-10 ```
robbrad commented 2 days ago

@raldred - if you look in your custom_components folder can you find this line

https://github.com/robbrad/UKBinCollectionData/blob/b588502d1d594a5b4c26ffa73157714baf871f0e/custom_components/uk_bin_collection/sensor.py#L166

does it say 15m ?

raldred commented 2 days ago

@robbrad It does

async def async_added_to_hass(self):
        """When entity is added to hass."""
        # Schedule periodic updates
        self._unsub_update_interval = async_track_time_interval(
            self.hass, self._update_state_based_on_time, timedelta(minutes=15)
        )

I thought the call outs to the councils would be handled by the HouseholdBinCoordinator? Which seems like it's scheduled to run every 12h

def __init__(self, hass, ukbcd, name):
        """Initialize my coordinator."""
        super().__init__(
            hass,
            _LOGGER,
            name="UK Bin Collection Data",
            update_interval=timedelta(hours=12),
        )
        _LOGGER.info(f"{LOG_PREFIX} UKBinCollectionApp Init")
        self.ukbcd = ukbcd
        self.name = name
robbrad commented 2 days ago

Sorry yes the 15m is for the senors update the scrape is from HouseholdBinCoordinator as you say

But what your observing is its not every 12 hours?

raldred commented 2 days ago

@robbrad not seeing it update at all unless I reload the integration or restart home assistant, then it updates the one time then never again until I do the same.

In the log file i included above covers 5 days I think, and there's only logging of updating today when i reloaded the integration, the same dates still showing days after collection.

robbrad commented 2 days ago

Thanks @raldred im away this weekend but it looks like I need to restructure the sensor. Hopefully get to this early next week

raldred commented 2 days ago

No rush at all @robbrad, appreciate the quick back and forth. I was trying to figure it out myself but honestly from my limited experience building custom components, the HouseholdBinCoordinator looks to be pretty standard.