Error fetching co2signal data

Twanislas commented 3 years ago

The problem

Every hour, I'm getting an Error fetching co2signal data in my logs. However, data seems to populate, except once an hour when the call fails.

What is version of Home Assistant Core has the issue?

core-2021.8.0

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

co2signal

Link to integration documentation on our website

https://www.home-assistant.io/integrations/co2signal/

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2021-08-04 16:18:49 ERROR (SyncWorker_7) [homeassistant.components.co2signal] Unexpected exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/co2signal/__init__.py", line 119, in get_data
    data = CO2Signal.get_latest(
  File "/usr/local/lib/python3.9/site-packages/CO2Signal/co2signal.py", line 34, in get_latest
    latest_data = (requests
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
--- snip ---
2021-08-04 17:18:49 ERROR (SyncWorker_1) [homeassistant.components.co2signal] Unexpected exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/co2signal/__init__.py", line 119, in get_data
    data = CO2Signal.get_latest(
  File "/usr/local/lib/python3.9/site-packages/CO2Signal/co2signal.py", line 34, in get_latest
    latest_data = (requests
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2021-08-04 17:18:49 ERROR (MainThread) [homeassistant.components.co2signal] Error fetching co2signal data:

Additional information

Since this is configured from the new energy component, I'm not sure how to gather more information to help, sorry.

probot-home-assistant[bot] commented 3 years ago

co2signal documentation co2signal source _{^{(message by IssueLinks)}}

chemelli74 commented 3 years ago

I think they had a disservice as I had the same error at the same point in time. Now everything works fine again.

Simone

Twanislas commented 3 years ago

Might be indeed, I'll monitor and if I don't have other errors I'll just go ahead and close this tomorrow.

Thanks :)

andriej commented 3 years ago

Errors persist in log, 10 times already since configuration (2 hours):

Logger: homeassistant.components.co2signal
Source: components/co2signal/__init__.py:119
Integration: CO2 Signal (documentation, issues)
First occurred: 20:29:49 (10 occurrences)
Last logged: 21:51:00

Unexpected exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/co2signal/__init__.py", line 119, in get_data
    data = CO2Signal.get_latest(
  File "/usr/local/lib/python3.9/site-packages/CO2Signal/co2signal.py", line 28, in get_latest
    latest_data = (requests
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

kquinsland commented 3 years ago

I just got my API key and tried to set up the co2 sensor myself. I am getting Unexpected error on the config flow dialogue. I get this error when the Use home location option is selected.

When i take a look at the logs, I see this as the culprit:

file "/usr/src/homeassistant/homeassistant/components/co2signal/sensor.py", line 127, in state
return round(self.coordinator.data["data"][self._description.key], 2) # type: ignore[misc]
TypeError: type NoneType doesn't define __round__ method

This is a distinct/different stack trace from the one @andriej posted. I am more than happy to open a new issue if it turns out that the two stack traces are not related.

EDIT I tried again with manually supplying the lat/lon and got a different error:

2021-08-04 20:16:55 ERROR (SyncWorker_2) [homeassistant.components.co2signal] Unexpected exception
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/co2signal/__init__.py", line 119, in get_data
data = CO2Signal.get_latest(
File "/usr/local/lib/python3.9/site-packages/CO2Signal/co2signal.py", line 34, in get_latest
latest_data = (requests
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Which seemed suspicious.... so i tried do do things manually:

❯ curl -H 'auth-token: 9<...>d' "https://api.co2signal.com/v1/latest?lon=35.....&lat=-120....."
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.15.3</center>
</body>
</html>

So they may be having some issues w/ their API endpoint

pdcemulator commented 3 years ago

Same here, new setup of the integration -> unexpected error

vampywiz17 commented 3 years ago

Same problem, i get 502 error, if i try to collect data.

Altycoder commented 3 years ago

Same problem for me too using 2021.8.1

Twanislas commented 3 years ago

Same here :

Logger: homeassistant.components.co2signal
Source: components/co2signal/__init__.py:119
Integration: CO2 Signal (documentation, issues)
First occurred: 4 août 2021, 16:03:49 (42 occurrences)
Last logged: 9:36:17

Unexpected exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/co2signal/__init__.py", line 119, in get_data
    data = CO2Signal.get_latest(
  File "/usr/local/lib/python3.9/site-packages/CO2Signal/co2signal.py", line 34, in get_latest
    latest_data = (requests
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I just sent them an email as they might not be aware, pointing to this issue.

Wait and see :)

AlexandreLogist commented 3 years ago

For me it's back online since a couple of minutes 👍

Twanislas commented 3 years ago

For me it's transient :

pdcemulator commented 3 years ago

Working now on my installation

AbeltjeNL commented 3 years ago

Same problem here.

Kongkille commented 3 years ago

Hi! Product Manager at electricityMap (and CO2 signal) here.

We're currently experiencing a large influx of CO2signal beta-users, which has interfered with the activation of new tokens and the performance of the API. We're currently investigating this issue - I'll make an update on this issue when we know more!

Kongkille commented 3 years ago

Hello again,

We believe we have a fix now.

It seems that the root issue was the memory, so we've tried to increase it quite a bit, and currently, the checks are looking promising. Let's hope this resolves it!

matteos1 commented 3 years ago

so is not possible require new token?

Kongkille commented 3 years ago

The token you requested may be delayed, so perhaps if you wait a few hours it may pop up. Otherwise, you can request a new token. If you're having issues you can also reach out to us at hello@tmrow.com :)

chilicheech commented 3 years ago

i just got a new token and i'm still getting the Unexpected error with Use home location

AbeltjeNL commented 3 years ago

Im getting these errors when creating the addon:

Error adding entities for domain sensor with platform co2signal
Error while setting up co2signal platform for sensor

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 382, in async_add_entities await asyncio.gather(*tasks) File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 587, in _async_add_entity await entity.add_to_platform_finish() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 711, in add_to_platform_finish self.async_write_ha_state() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 464, in async_write_ha_state self._async_write_ha_state() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 498, in _async_write_ha_state state = self._stringify_state() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 470, in _stringify_state state = self.state File "/usr/src/homeassistant/homeassistant/components/co2signal/sensor.py", line 127, in state return round(self.coordinator.data["data"][self._description.key], 2) # type: ignore[misc] TypeError: type NoneType doesn't define round method

Tried all three config methods, rebooted HA but no dice.

Edit:just registered with another e-mailadres and applied for a new API key. Everything is working fine now and the above errors didn't show.

kquinsland commented 3 years ago

Thanks for confirming, @Kongkille . I suspected that the HA launch would be the 'hug of death'.

I am still not able to add the integration successfully, though. I'll try again in a few days.

It looks like two things need to happen:

the co2 signal API needs some fortification; loads of good SREs on the market right now :)
this integration needs a lot more work on failure handling. At a minimum, "unknown error" messages should not propagate to the front end if the error is known; the API didn't return 200/OK and valid JSON. Ideally, some basic failure / exponential back off logic would also be a very good idea. Show the sensor as 'unavailable' in the frontend and keep checking periodically to see if the API is back up or not.

Twanislas commented 3 years ago

Last error for the component fired on 2021-08-05 12:38:17 CET for me. Looks like the adjustments from @Kongkille's team resolved the issue (at least for my case). Thanks !

Thanks everyone for the constructive feedback !

Let's keep this open another day, and close if there's no more issues :)

pimw1 commented 3 years ago

I had the same issue, but it has been resolved for me.

tankdeer commented 3 years ago

I too am receiving the Unexpected Error issue when trying to setup the integration. Tried two different API keys. One generated yesterday, and one today

kquinsland commented 3 years ago

@tankdeer

Can you look closer at the logs? there are two issues at play:

the co2 signal API is getting swamped; this manifests as gaps in the graph data or possibly a JSONDecodeError
An error with using the lat/lon already saved in HA, this manifests as TypeError: type NoneType doesn't define round method or similar in the logs

If you're getting the latter of the two, join the club. You should be able to specify a country/region code (i used US) to at least get something up and running. At the very least, this will confirm that your API key(s) work

tankdeer commented 3 years ago

@kquinsland

Good call. It's the latter issue. I was able to do the same as you, by using the ever-so-vague country code. It created the entities, but with the same error for the undefined round method

2021-08-06 20:22:17 ERROR (MainThread) [homeassistant.components.sensor] Error adding entities for domain sensor with platform co2signal
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 382, in async_add_entities
    await asyncio.gather(*tasks)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 587, in _async_add_entity
    await entity.add_to_platform_finish()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 711, in add_to_platform_finish
    self.async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 464, in async_write_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 498, in _async_write_ha_state
    state = self._stringify_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 470, in _stringify_state
    state = self.state
  File "/usr/src/homeassistant/homeassistant/components/co2signal/sensor.py", line 127, in state
    return round(self.coordinator.data["data"][self._description.key], 2)  # type: ignore[misc]
TypeError: type NoneType doesn't define __round__ method
2021-08-06 20:22:17 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up co2signal platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 257, in _async_setup_platform
    await asyncio.gather(*pending)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 382, in async_add_entities
    await asyncio.gather(*tasks)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 587, in _async_add_entity
    await entity.add_to_platform_finish()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 711, in add_to_platform_finish
    self.async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 464, in async_write_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 498, in _async_write_ha_state
    state = self._stringify_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 470, in _stringify_state
    state = self.state
  File "/usr/src/homeassistant/homeassistant/components/co2signal/sensor.py", line 127, in state
    return round(self.coordinator.data["data"][self._description.key], 2)  # type: ignore[misc]
TypeError: type NoneType doesn't define __round__ method

Since the US is not exactly small, it'll be nice when we can be a little more specific on our locations

chemelli74 commented 3 years ago

Since the US is not exactly small, it'll be nice when we can be a little more specific on our locations

Here the list of available zones: http://api.electricitymap.org/v3/zones

Simone

chilicheech commented 3 years ago

That did the trick for me. Instead of using my home location i used a country code and it worked

On Fri, Aug 6, 2021, 23:27 Simone Chemelli @.***> wrote:

Since the US is not exactly small, it'll be nice when we can be a little more specific on our locations

Here the list of available zones: http://api.electricitymap.org/v3/zones

Simone

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/53982#issuecomment-894613176, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFS346YLQPHPVBMFRJATZTT3TG6HANCNFSM5BRUHUIA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

tankdeer commented 3 years ago

Since the US is not exactly small, it'll be nice when we can be a little more specific on our locations

Here the list of available zones: http://api.electricitymap.org/v3/zones

Simone

Using anything more specific than country code (US), doesn't work for me. I tried US-OR, US-NW-PACW, home location, & coordinates, and only country code was accepted. However, I still have the round method error in my log file, and the entities are unavailable

AbeltjeNL commented 3 years ago

Since the US is not exactly small, it'll be nice when we can be a little more specific on our locations

Here the list of available zones: http://api.electricitymap.org/v3/zones

Simone

Using anything more specific than country code (US), doesn't work for me. I tried US-OR, US-NW-PACW, home location, & coordinates, and only country code was accepted. However, I still have the round method error in my log file, and the entities are unavailable

The round error and entities unavailable are fixed on my side by applying for an api key on a different e-mailadres.

Planning on waiting for a week or two to apply for a new api key on my main e-mailadres.

tankdeer commented 3 years ago

Since the US is not exactly small, it'll be nice when we can be a little more specific on our locations

Here the list of available zones: http://api.electricitymap.org/v3/zones

Simone

Using anything more specific than country code (US), doesn't work for me. I tried US-OR, US-NW-PACW, home location, & coordinates, and only country code was accepted. However, I still have the round method error in my log file, and the entities are unavailable

The round error and entities unavailable are fixed on my side by applying for an api key on a different e-mailadres.

Planning on waiting for a week or two to apply for a new api key on my main e-mailadres.

Unfortunately I tried that as well and it did not work either.

I am curious if this is supported for all locations? Looking at the ElectricityMap, most of the US is shown as grey. Does that mean the API won't work for people living in those areas, because the data is unavailable?

Kongkille commented 3 years ago

Hi all,

Thanks for highlighting some of these issues. We are aware that they are issues with the CO2 Signal API especially in terms of error handling and the overall UX of the service. I'm seeing a lot of issues due to using lat/lon and not being able to find the right country codes. Several users have also reached out to us regarding this, and have unfortunately spent a long time on setup even though we do not have data for the zone yet.

I'm wondering whether we should open a new issue around potentially adding functionality into home assistant to test whether there is actually real-time data for the zone the user is trying to get data from? That way we'll be able to potentially save a lot of people from troubleshooting. I'd be happy to help however I can.

In the short term, for the people in this thread, and I am not quite sure how this integration is set up, I think the easiest solution is to find the relevant country codes by looking at our live map on electricitymap.org in case the lat/lon is having issues.

Specifically for the US, as I see a lot of comments about that. We do have data for all of the US, but most of the data is delayed by up to 24 hours which does indeed mean that the CO2 Signal API won't work for most of the grids colored grey. Unfortunately, there's not much for us to do except hope that the data providers will allow for real-time data in the future.

The team is really excited about your interest in the API. We're a fairly small team and are not able to dedicate as much time as we want to on this, but hopefully, we can integrate some of the feedback we've received to make it a bit more stable and easier to use.

github-actions[bot] commented 2 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / core