[Data Issue]: CO2signal data doesn't match with Electricity Map

[eutampieri]

When did this happen?

"2022-11-17T19:00:00Z"

What zones are affected?

"IT-CNO"

What is the problem?

Fetching https://api.co2signal.com/v1/latest?countryCode=IT-NO gives me 517 g/kWh while the app says this

[eutampieri]

Explore-data-2022-11-21 15_28_50.csv Here's the data recorded since the issue started

[madsnedergaard]

Hi, the app is showing data a few hours delayed to ensure we have a wider coverage on the map. Therefore the values might not necessarily match at a given time :)

[eutampieri]

@madsnedergaard actually, that’s not the issue I’m reporting. Even with that taken into account, for a given time the app displays a lose value than the co2signal API.

[VIKTORVAV99]

Could you check again if the issue is still there? There were some calculation issues that was fixed today that might have played a part in this as well.

[eutampieri]

If I'm not doing anything wrong, these are the API data points (I save in a DB the first value the API gives me so, if the value changes for a time range, I save only the first). The app says 417 g/kWh at 20:00 in IT-CNO.

[VIKTORVAV99]

I'm going to reopen this for now until we get a better understanding of why there is a disparity between the co2signal data and the app (because it does seem to be one).

[eutampieri]

If you need more help on my side, please ping me so I don’t loose your message. For now, thank you.

[madsnedergaard]

This is strange, I'm wondering if it could be a timezone related thing? We also cache data on the app, so could be that you're not seeing the latest data.

When querying right now at 10.06 am in GMT (11.06 in DK time), I get similar numbers for the app, CO2Signal and commercial API:

CO2Signal API:

Commercial API:

App:

[eutampieri]

By chance, do you detect timezones based on IP address geolocation?

[VIKTORVAV99]

@madsnedergaard could you check SE-SE4 on co2signal vs the commercial api? I'm seeing discrepancies between the co2signal api and the state files that I can't attribute to time zones miss matches.

[eutampieri]

@madsnedergaard actually, my issue is with IT-NO, not IT-CNO.

[eutampieri]

And here’s the data straight from HomeAssistant, whose debug says

"data": {
    "config_entry": {
      "entry_id": "64b95bd7136cae470bc50d1b15985f66",
      "version": 1,
      "domain": "co2signal",
      "title": "CO2 Signal",
      "data": {
        "api_key": "**REDACTED**"
      },
      "options": {},
      "pref_disable_new_entities": false,
      "pref_disable_polling": false,
      "source": "user",
      "unique_id": null,
      "disabled_by": null
    },
    "data": {
      "_disclaimer": "This data is the exclusive property of Electricity Maps and/or related parties. If you're in doubt about your rights to use this data, please contact api@co2signal.com",
      "status": "ok",
      "countryCode": "IT-NO",
      "data": {
        "datetime": "2022-11-23T12:00:00.000Z",
        "carbonIntensity": 481,
        "fossilFuelPercentage": 74.12
      },
      "units": {
        "carbonIntensity": "gCO2eq/kWh"
      }
    }
  }

[madsnedergaard]

@madsnedergaard could you check SE-SE4 on co2signal vs the commercial api? I'm seeing discrepancies between the co2signal api and the state files that I can't attribute to time zones miss matches.

Same value of 32g for 2022-11-23T13:00:00.000Z on both APIs :) Map says 41g for 2022-11-23 at 12:00 PM right now, so in an hour or two it should also say 32g.

And here’s the data straight from HomeAssistant, whose debug says

Thanks for sharing the detailed info. At what time does the script/scraper run? I cannot reproduce it, so could be a specific time where new data vs old data vs cache makes things go weird 🤔

[eutampieri]

`Luis seem timestamps to me :)

{
    "id": 29,
    "type": "result",
    "success": true,
    "result": {
        "sensor.co2_signal_co2_intensity": [
            {
                "s": "493",
                "a": {},
                "lu": 1669110544.58958
            },
            {
                "s": "483",
                "lu": 1669111525.244081
            },
            {
                "s": "482",
                "lu": 1669113554.952491
            },
            {
                "s": "458",
                "lu": 1669114454.401386
            },
            {
                "s": "465",
                "lu": 1669115613.064229
            },
            {
                "s": "420",
                "lu": 1669118313.224761
            },
            {
                "s": "483",
                "lu": 1669119213.493568
            },
            {
                "s": "433",
                "lu": 1669122022.728489
            },
            {
                "s": "476",
                "lu": 1669123280.322045
            },
            {
                "s": "433",
                "lu": 1669125968.417084
            },
            {
                "s": "478",
                "lu": 1669126674.488427
            },
            {
                "s": "443",
                "lu": 1669128741.43075
            },
            {
                "s": "475",
                "lu": 1669129641.570992
            },
            {
                "s": "471",
                "lu": 1669133456.94182
            },
            {
                "s": "446",
                "lu": 1669136161.773303
            },
            {
                "s": "492",
                "lu": 1669137061.279364
            },
            {
                "s": "436",
                "lu": 1669140214.046025
            },
            {
                "s": "497",
                "lu": 1669140606.868205
            },
            {
                "s": "503",
                "lu": 1669144206.54828
            },
            {
                "s": "476",
                "lu": 1669146906.452961
            },
            {
                "s": "510",
                "lu": 1669147806.595256
            },
            {
                "s": "516",
                "lu": 1669151239.073149
            },
            {
                "s": "531",
                "lu": 1669155530.335818
            },
            {
                "s": "539",
                "lu": 1669159130.384837
            },
            {
                "s": "448",
                "lu": 1669161830.373331
            },
            {
                "s": "547",
                "lu": 1669162730.847146
            },
            {
                "s": "558",
                "lu": 1669165430.353982
            },
            {
                "s": "555",
                "lu": 1669169030.449055
            },
            {
                "s": "556",
                "lu": 1669172630.277572
            },
            {
                "s": "562",
                "lu": 1669176230.510809
            },
            {
                "s": "546",
                "lu": 1669179830.352901
            },
            {
                "s": "509",
                "lu": 1669183430.675507
            },
            {
                "s": "478",
                "lu": 1669187030.254212
            },
            {
                "s": "510",
                "lu": 1669190668.24344
            },
            {
                "s": "522",
                "lu": 1669194269.236576
            },
            {
                "s": "486",
                "lu": 1669197682.357048
            },
            {
                "s": "464",
                "lu": 1669201284.002452
            },
            {
                "s": "420",
                "lu": 1669204812.253574
            },
            {
                "s": "481",
                "lu": 1669205712.309731
            },
            {
                "s": "446",
                "lu": 1669209312.44347
            },
            {
                "s": "438",
                "lu": 1669212012.692511
            }
        ]
    }
}

[madsnedergaard]

And this was all from IT-NO and not mixed with IT-CNO, right? Here's the data we have since midnight, so unsure why you're getting different values...

            "2022-11-23T00:00:00.000Z": 399,
            "2022-11-23T01:00:00.000Z": 403,
            "2022-11-23T02:00:00.000Z": 403,
            "2022-11-23T03:00:00.000Z": 394,
            "2022-11-23T04:00:00.000Z": 401,
            "2022-11-23T05:00:00.000Z": 415,
            "2022-11-23T06:00:00.000Z": 420,
            "2022-11-23T07:00:00.000Z": 385,
            "2022-11-23T08:00:00.000Z": 373,
            "2022-11-23T09:00:00.000Z": 373,
            "2022-11-23T10:00:00.000Z": 369,
            "2022-11-23T11:00:00.000Z": 372,
            "2022-11-23T12:00:00.000Z": 369,
            "2022-11-23T13:00:00.000Z": 438,
            "2022-11-23T14:00:00.000Z": 440

[eutampieri]

Here's a test I've just done (it matches your results, but then there must be something wrong in HA):

``` eugeniotampieri@Air-di-Eugenio arera_worker % curl -vvv -6 -H "auth-token:" "https://api.co2signal.com/v1/latest?countryCode=IT-NO" | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2606:4700:3030::6815:3cc7:443... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to api.co2signal.com (2606:4700:3030::6815:3cc7) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem * CApath: none * (304) (OUT), TLS handshake, Client hello (1): } [322 bytes data] * (304) (IN), TLS handshake, Server hello (2): { [122 bytes data] * (304) (IN), TLS handshake, Unknown (8): { [19 bytes data] * (304) (IN), TLS handshake, Certificate (11): { [2335 bytes data] * (304) (IN), TLS handshake, CERT verify (15): { [78 bytes data] * (304) (IN), TLS handshake, Finished (20): { [52 bytes data] * (304) (OUT), TLS handshake, Finished (20): } [52 bytes data] * SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com * start date: Jun 14 00:00:00 2022 GMT * expire date: Jun 14 23:59:59 2023 GMT * subjectAltName: host "api.co2signal.com" matched cert's "*.co2signal.com" * issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3 * SSL certificate verify ok. * Using HTTP2, server supports multiplexing * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7f94df810a00) > GET /v1/latest?countryCode=IT-NO HTTP/2 > Host: api.co2signal.com > user-agent: curl/7.79.1 > accept: */* > auth-token: > * Connection state changed (MAX_CONCURRENT_STREAMS == 256)! < HTTP/2 200 < date: Wed, 23 Nov 2022 14:18:52 GMT < content-type: application/json; charset=utf-8 < content-length: 360 < x-ratelimit-remaining-hour: 54 < x-ratelimit-remaining-day: 1320 < x-ratelimit-limit-day: 1440 < ratelimit-reset: 2468 < ratelimit-remaining: 54 < ratelimit-limit: 60 < x-ratelimit-limit-hour: 60 < x-powered-by: Express < access-control-allow-origin: * < access-control-allow-headers: Origin, X-Requested-With, Content-Type, Accept, electricitymap-token < vary: Accept-Encoding < x-kong-upstream-latency: 19 < x-kong-proxy-latency: 4 < via: kong/2.5.0 < cf-cache-status: DYNAMIC < report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=ql0R6WJ5uILKsY8HQ0%2FNkivE%2Fa3NiNHJCcjT32gu2WKcSAFddNOgYnEapZ1BQXl94JlhqZ5vH6XdJJ%2Fjej65jwWh5ijA%2BWU39AjzWcLb2ma5sSoHXXUHMy0kKTt2k1WVhBtsO%2B27K2cQ6qboq0u1%2Fg%3D%3D"}],"group":"cf-nel","max_age":604800} < nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} < server: cloudflare < cf-ray: 76ea895e0b02d6b6-CDG < alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400 < { [360 bytes data] 100 360 100 360 0 0 969 0 --:--:-- --:--:-- --:--:-- 1000 * Connection #0 to host api.co2signal.com left intact { "_disclaimer": "This data is the exclusive property of Electricity Maps and/or related parties. If you're in doubt about your rights to use this data, please contact api@co2signal.com", "status": "ok", "countryCode": "IT-NO", "data": { "datetime": "2022-11-23T14:00:00.000Z", "carbonIntensity": 440, "fossilFuelPercentage": 66.22 }, "units": { "carbonIntensity": "gCO2eq/kWh" } } ``` ``` date && curl -vvv -4 -H "auth-token: " "https://api.co2signal.com/v1/latest?countryCode=IT-NO" | jq Mer 23 Nov 2022 15:22:27 CET % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.67.201.21:443... * Connected to api.co2signal.com (172.67.201.21) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem * CApath: none * (304) (OUT), TLS handshake, Client hello (1): } [322 bytes data] * (304) (IN), TLS handshake, Server hello (2): { [122 bytes data] * (304) (IN), TLS handshake, Unknown (8): { [19 bytes data] * (304) (IN), TLS handshake, Certificate (11): { [2335 bytes data] * (304) (IN), TLS handshake, CERT verify (15): { [79 bytes data] * (304) (IN), TLS handshake, Finished (20): { [52 bytes data] * (304) (OUT), TLS handshake, Finished (20): } [52 bytes data] * SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com * start date: Jun 14 00:00:00 2022 GMT * expire date: Jun 14 23:59:59 2023 GMT * subjectAltName: host "api.co2signal.com" matched cert's "*.co2signal.com" * issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3 * SSL certificate verify ok. * Using HTTP2, server supports multiplexing * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7faa0580d800) > GET /v1/latest?countryCode=IT-NO HTTP/2 > Host: api.co2signal.com > user-agent: curl/7.79.1 > accept: */* > auth-token: > * Connection state changed (MAX_CONCURRENT_STREAMS == 256)! < HTTP/2 200 < date: Wed, 23 Nov 2022 14:22:28 GMT < content-type: application/json; charset=utf-8 < content-length: 360 < x-ratelimit-remaining-hour: 51 < x-ratelimit-remaining-day: 1317 < x-ratelimit-limit-day: 1440 < ratelimit-reset: 2252 < ratelimit-remaining: 51 < ratelimit-limit: 60 < x-ratelimit-limit-hour: 60 < x-powered-by: Express < access-control-allow-origin: * < access-control-allow-headers: Origin, X-Requested-With, Content-Type, Accept, electricitymap-token < vary: Accept-Encoding < x-kong-upstream-latency: 20 < x-kong-proxy-latency: 5 < via: kong/2.5.0 < cf-cache-status: DYNAMIC < report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=XBNMMfqdriB2sqeW3kLEpVrfJX1KUNIOahZPcYuxIJ7ToRLE2k2l0MC9M0auq7PbDz9JiZxERFoam5TCk6iro2j2ILmop%2B71f5EF5KJ731ir1LtZizPjMtKi11eO1m1w8JekoQ%3D%3D"}],"group":"cf-nel","max_age":604800} < nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} < server: cloudflare < cf-ray: 76ea8ea10f88ba9a-MXP < alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400 < { [360 bytes data] 100 360 100 360 0 0 1345 0 --:--:-- --:--:-- --:--:-- 1422 * Connection #0 to host api.co2signal.com left intact { "_disclaimer": "This data is the exclusive property of Electricity Maps and/or related parties. If you're in doubt about your rights to use this data, please contact api@co2signal.com", "status": "ok", "countryCode": "IT-NO", "data": { "datetime": "2022-11-23T14:00:00.000Z", "carbonIntensity": 440, "fossilFuelPercentage": 66.22 }, "units": { "carbonIntensity": "gCO2eq/kWh" } } ```

[VIKTORVAV99]

I've been meaning to take a look at the CO2signal ha integration since it has some outdated links and information so I'll take a peek at this issue as well when I do that. But I can't give you a time frame for when I have the time unfortunately.

[VIKTORVAV99]

Just adding this here: I've been trying to make sense of it but none of the values match the App. So there is definitely something weird going on in either the HA integration or in the https://github.com/danielsjf/CO2Signal repo that the integration use.

On a side note, should the "Data provided by CO2Signal" say "Data provided by Electricity Maps" instead?

EDIT: It looks like it matches the production mode on the app pretty well:

Could it be that the HA intergration is using production data instead of consumption data?

[jarek]

Could it be that the HA intergration is using production data instead of consumption data?

Reading the source code, HA integration seems to use latest_data['data']['carbonIntensity'] from https://api.co2signal.com/v1/latest, and https://docs.co2signal.com/#introduction says this returns a response like:

{
    "countryCode": "FR",
    "data": {
        "carbonIntensity": 92.97078744790771,
        "datetime": "2017-02-09T08:30:00.000Z",
        "fossilFuelPercentage": 12.028887656434616
    },
    "status": "ok",
    "units": {
        "carbonIntensity": "gCO2eq/kWh"
    }
}

So the questions are:

• what kind of data does https://api.co2signal.com/v1/latest?countryCode return in the carbonIntensity key (specifically the v1 version)?
• • is the carbonIntensity data in the API production or consumption?
• does the response and/or the data differ between v1 and v3 of co2signal API?
• should the docs for v3 co2signal API be publicly visible?

[VIKTORVAV99]

Could it be that the HA intergration is using production data instead of consumption data?

Reading the source code, HA integration seems to use latest_data['data']['carbonIntensity'] from https://api.co2signal.com/v1/latest, and https://docs.co2signal.com/#introduction says this returns a response like:

{
    "countryCode": "FR",
    "data": {
        "carbonIntensity": 92.97078744790771,
        "datetime": "2017-02-09T08:30:00.000Z",
        "fossilFuelPercentage": 12.028887656434616
    },
    "status": "ok",
    "units": {
        "carbonIntensity": "gCO2eq/kWh"
    }
}

So the questions are:

• what kind of data does https://api.co2signal.com/v1/latest?countryCode return in the carbonIntensity key (specifically the v1 version)?
• • is the carbonIntensity data in the API production or consumption?
• does the response and/or the data differ between v1 and v3 of co2signal API?
• should the docs for v3 co2signal API be publicly visible?

I don't even think there is a v3 of CO2Signal, but I have started to dig into it as I now have access to all the different components in this whole chain. (CO2Signal, CO2Signal PyPi package, and then HA).

[tvwerkhoven]

I just noticed the same issue (which didn't exist 1.5 years ago when I set up my cron jobs). The co2signal API returns a different value than shown on https://app.electricitymaps.com/ for at least zones BE, DE, NL. Timing cannot be an issue as I've collected a time range of data from co2signal.

This is data I collected for my local plotting:

time                 board location quantity        room source    value
----                 ----- -------- --------        ---- ------    -----
2023-06-17T05:00:00Z       BE       carbonIntensity      co2signal 108
2023-06-17T05:00:00Z       NL       carbonIntensity      co2signal 448
2023-06-17T05:00:00Z       DE       carbonIntensity      co2signal 463
2023-06-17T04:00:00Z       BE       carbonIntensity      co2signal 111
2023-06-17T04:00:00Z       NL       carbonIntensity      co2signal 359
2023-06-17T04:00:00Z       DE       carbonIntensity      co2signal 540
2023-06-17T03:00:00Z       BE       carbonIntensity      co2signal 122
2023-06-17T03:00:00Z       NL       carbonIntensity      co2signal 508
2023-06-17T03:00:00Z       DE       carbonIntensity      co2signal 590

and this is what app.electricitmaps.com shows (7:00 is 5:00Z):

How can this be? Is a different/legacy algorithm/averaging being used perhaps? As mentioned before, I checked this when I set up my cron job and then there was no delta.

Update: doing some chi-by-eye the data correlates but is quite different for the rest (taking DE zone as example):

1. It seems there's a timing offset. What timezone is shown on app.electricitymaps.com? If I assume it's local time (CET), I notice a ~2-hour offset in the chart (see below), so does this mean the website is showing UTC?
2. The values are different, co2signal in general shows higher values

[VIKTORVAV99]

We have identified a few issues but the big ones basically boil down to the following two:

1. Some of the issues is related to how the data flows differ between the app and homeassistant. For the app we retroactively change data if the source data changes (it can be updates or if it's delayed) (something that is surprisingly common) while it seems like homeassistant can't update previously recoded data automatically, at least the current integration can't.

2. Another this is that the app is on a 2 hour delay to make sure we get as much data as possible before showing it in the app (this has nothing to do with timezones) as for example ENTSO-E has a publishing deadline of 2 hours for many bidding areas/countries and exchanges, if we don't get the data for this in time it drastically changes the carbon intensity in many cases.

We are working on some improvements that should help with this but nothing to share about these yet.

[tvwerkhoven]

Thanks for your reply @VIKTORVAV99

Note that in my case 1. is not applicable: this is data directly from the CO2signal REST API (and then stored locally).

While 2 is applicable and it seems timezone info is missing on app.electricitymaps.com, this does not fully explain it because also with the shift there is no identical data between CO2signal or app.electricitymaps.com. It might be that historical data is altered in the future, but also the newest data point in both sources differ.

Am I missing something? It seems there must be another source of the delta?

[madsnedergaard]

Closing this as we merged the CO2Signal API into our main API and we haven't heard of this problem since :)

[eutampieri]

Tak 🤗