search

J-Lindvig / Fuelprices_DK

Scraping of 5 types of fuel :fuelpump: from 8 different fuelcompanies in Denmark :denmark:.
GNU General Public License v3.0
20 stars 11 forks source link

F24 and Q8 down/slow. Parse error #10

Closed mm98 closed 2 years ago

mm98 commented 2 years ago

Hi,

Thanks for this awesome HA extension!

It looks like F24 and Q8 API endpoints are very slow (randomly) at the moment ("" returned from API's). Which results in all fuel prices will be marked as Unavailable.

image

This causes multiple timeouts/errors in HA logs:

2022-11-03 08:23:19.212 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform fuelprices_dk is taking over 10 seconds.
2022-11-03 08:24:05.509 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: sensor.fuelprices_dk
2022-11-03 08:24:09.214 ERROR (MainThread) [homeassistant.components.sensor] Setup of platform fuelprices_dk is taking longer than 60 seconds. Startup will proceed without waiting any longer.
2022-11-03 08:24:55.064 ERROR (MainThread) [custom_components.fuelprices_dk.sensor] Unexpected error fetching sensor data: list index out of range
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 205, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 164, in _async_update_data
    return await self.update_method()
  File "/config/custom_components/fuelprices_dk/sensor.py", line 30, in async_update_data
    await hass.async_add_executor_job(fuelPrices.refresh)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_api.py", line 151, in refresh
    company.refreshPrices()
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_api.py", line 200, in refreshPrices
    self._products = getattr(self._parser, self._key)(self._url, self._products)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_parsers.py", line 149, in f24
    return self._f24_q8(url, products)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_parsers.py", line 182, in _f24_q8
    json_product = r.json()["Products"][productDict["Index"]]
IndexError: list index out of range

Would it maybe be an option to add the timeout argument to the request.get() and validate, that you get any content back in the response. The status is 200 - but no content.

https://github.com/J-Lindvig/Fuelprices_DK/blob/e224e01856848566f3bc2fbf0603c2f638636e63/custom_components/fuelprices_dk/fuelprices_dk_parsers.py#L192-L197

I guess r.text is valid, but it is empty.

https://github.com/J-Lindvig/Fuelprices_DK/blob/e224e01856848566f3bc2fbf0603c2f638636e63/custom_components/fuelprices_dk/fuelprices_dk_parsers.py#L199-L201

mediaNen commented 2 years ago

Exact same issue for me, since updating to the newest core edition of hass

J-Lindvig commented 2 years ago

It seems like Q8 and F24 is having some issues on their sites with prices. image image

Since Fuelprices rely on working websites, it will cause an error.

J-Lindvig commented 2 years ago

Hi,

Thanks for this awesome HA extension!

It looks like F24 and Q8 API endpoints are very slow (randomly) at the moment ("" returned from API's). Which results in all fuel prices will be marked as Unavailable.

image

This causes multiple timeouts/errors in HA logs:

2022-11-03 08:23:19.212 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform fuelprices_dk is taking over 10 seconds.
2022-11-03 08:24:05.509 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: sensor.fuelprices_dk
2022-11-03 08:24:09.214 ERROR (MainThread) [homeassistant.components.sensor] Setup of platform fuelprices_dk is taking longer than 60 seconds. Startup will proceed without waiting any longer.
2022-11-03 08:24:55.064 ERROR (MainThread) [custom_components.fuelprices_dk.sensor] Unexpected error fetching sensor data: list index out of range
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 205, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 164, in _async_update_data
    return await self.update_method()
  File "/config/custom_components/fuelprices_dk/sensor.py", line 30, in async_update_data
    await hass.async_add_executor_job(fuelPrices.refresh)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_api.py", line 151, in refresh
    company.refreshPrices()
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_api.py", line 200, in refreshPrices
    self._products = getattr(self._parser, self._key)(self._url, self._products)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_parsers.py", line 149, in f24
    return self._f24_q8(url, products)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_parsers.py", line 182, in _f24_q8
    json_product = r.json()["Products"][productDict["Index"]]
IndexError: list index out of range

Would it maybe be an option to add the timeout argument to the request.get() and validate, that you get any content back in the response. The status is 200 - but no content.

https://github.com/J-Lindvig/Fuelprices_DK/blob/e224e01856848566f3bc2fbf0603c2f638636e63/custom_components/fuelprices_dk/fuelprices_dk_parsers.py#L192-L197

I guess r.text is valid, but it is empty.

https://github.com/J-Lindvig/Fuelprices_DK/blob/e224e01856848566f3bc2fbf0603c2f638636e63/custom_components/fuelprices_dk/fuelprices_dk_parsers.py#L199-L201

I am adding the timeout, as I should have done in the first place.... ;-)

The missing data from f24/q8 ("") is because their prices i served by an API and not a webpage.