search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
72.67k stars 30.42k forks source link

error sensor bad data #60556

Closed Lordfas closed 2 years ago

Lordfas commented 2 years ago

The problem

Logger: homeassistant.components.pvpc_hourly_pricing.sensor Source: components/pvpc_hourly_pricing/sensor.py:136 Integration: Spain electricity hourly pricing (PVPC) (documentation, issues) First occurred: 21:22:10 (2 occurrences) Last logged: 21:26:06

sensor.precio_luz: repeated bad data update, mark component as unavailable source

What version of Home Assistant Core has the issue?

core-2021.11.5

What was the last working version of Home Assistant Core?

core-2021.11.5

What type of installation are you running?

Home Assistant Core

Integration causing the issue

esios

Link to integration documentation on our website

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

ad-ha commented 2 years ago

I am finding the same issues. The PVPC has become unavailable.

My log shows the following:

Logger: aiopvpc.pvpc_data
Source: /usr/local/lib/python3.9/site-packages/aiopvpc/pvpc_data.py:231
First occurred: 7:09:26 PM (12 occurrences)
Last logged: 11:39:00 PM

Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '860', 'X-Iinfo': '6-18602997-0 0CNN RT(1638218339535 9) q(0 -1 -1 1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1396_1885724=o8iGaACy+k2RLpQe3pZfE2M6pWEAAAAAWlqKbJ+gbdejF3g5h4njzw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '862', 'X-Iinfo': '13-87936528-0 0CNN RT(1638220139538 10) q(0 -1 -1 0) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1396_1885724=Bu1qb1HNzFwyhaYe3pZfE2tBpWEAAAAAxWu7KSQ4MF8ZsdIxasTD7A==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '862', 'X-Iinfo': '12-77704484-0 0CNN RT(1638221939533 12) q(0 -1 -1 2) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1396_1885724=/GNLbgJv0j12obge3pZfE3NIpWEAAAAAVNiv2o243jqlz6Ggp8VmCg==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '862', 'X-Iinfo': '12-78019039-0 0CNN RT(1638223739534 13) q(0 -1 -1 0) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1396_1885724=0+FbXmDkPwp+qske3pZfE3tPpWEAAAAAMnnAkf0TAyzMjFBDA4bTCw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '861', 'X-Iinfo': '14-98378988-0 0CNN RT(1638225539532 9) q(0 -1 -1 1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1396_1885724=QRqTU0BhkH6NVdce3pZfE4NWpWEAAAAA4CL3P2stUmXc+XTbwqhBFA==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>

and 

Logger: homeassistant.components.pvpc_hourly_pricing.sensor
Source: components/pvpc_hourly_pricing/sensor.py:136
Integration: Spain electricity hourly pricing (PVPC) (documentation, issues)
First occurred: 7:10:27 PM (1 occurrences)
Last logged: 7:10:27 PM

sensor.pvpc: repeated bad data update, mark component as unavailable source
probot-home-assistant[bot] commented 2 years ago

pvpc_hourly_pricing documentation pvpc_hourly_pricing source (message by IssueLinks)

probot-home-assistant[bot] commented 2 years ago

Hey there @azogue, mind taking a look at this issue as it has been labeled with an integration (pvpc_hourly_pricing) you are listed as a code owner for? Thanks! (message by CodeOwnersMention)

lazynoda commented 2 years ago

Same problem here. I've noticed that accessing directly to https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-30 via curl works.

System info version | core-2021.11.5 -- | -- installation_type | Home Assistant Container dev | false hassio | false docker | true user | root virtualenv | false python_version | 3.9.7 os_name | Linux os_version | 5.10.63-v7l+ arch | armv7l timezone | Europe/Madrid
azogue commented 2 years ago

It's also happening for me (and probably for all)

Looks like the ESIOS server is banning us again 😭

It returns a 403 status error for a request at a PUBLIC url 😤, and it's just banning us based on the content of the User-Agent header 😢, that's why the URL works with curl, or with a browser

I'll try to make a quickfix to try to avoid the banning, again 😭, but it looks like the webadmins at ESIOS are kind of angry at us 🙈

vz-john commented 2 years ago

Can you add an option to add an API Token from REE? https://www.esios.ree.es/es/pagina/api

jjprz commented 2 years ago

Can you add an option to add an API Token from REE? https://www.esios.ree.es/es/pagina/api

It could be interesting

martinezpenya commented 2 years ago

Same problem here!

davefx commented 2 years ago

Alternately, maybe you could add an option to set the used User Agent, so they cannot easily identify us!

fernandodpr commented 2 years ago

I've just received my personal token for the REE API. They also sent the following message asking for a responsible use of the resource.

Este token se emite para uso personal del solicitante.

· Si va a dar acceso público a los proyectos desarrollados por Vd. (aplicaciones, páginas web, scripts…) que utilizan datos recuperados de la API de e·sios, debe llevarse la información que devuelve la API a un servidor de su propiedad, de tal forma que esas aplicaciones, páginas o scripts accedan a los datos almacenados en su servidor y no a los sistemas de REE.

· Uso responsable de la API: no realice peticiones masivas, redundantes o innecesarias, tal como información ya descargada con anterioridad o información que se sabe que no se ha modificado, así como peticiones a indicadores inexistentes

Would be great to have the possibility in the integration to insert the API token and send all petitions authenticated.

lazynoda commented 2 years ago

If "esios" API is banning us, we could also use REData API: https://www.ree.es/es/apidatos. Example for today: https://apidatos.ree.es/en/datos/mercados/precios-mercados-tiempo-real?start_date=2021-12-01T00:00&end_date=2021-12-01T23:59&time_trunc=hour

From JSON response, take one of the objects inside "include" (maybe filtering by id == "1001"?) and that data is in €/MWh. I don't see any rate limit restriction or anything else.

The REData API provides a simple REST service to allow third parties to access the backend data used in the REData application. By using this API, you will be able to retrieve data from the REData widgets and use them for your own purposes.

lazynoda commented 2 years ago

But totally agree on using "esios" personal API keys so we don't run on the same problem again. How much time it took @fernandodpr for them to provide a key? I guess we need to write it in HA documentation.

MrMarble commented 2 years ago

But totally agree on using "esios" personal API keys so we don't run on the same problem again. How much time it took @fernandodpr for them to provide a key? I guess we need to write it in HA documentation.

I asked for a token last night, and they replied today, exactly 14h apart

fernandodpr commented 2 years ago

But totally agree on using "esios" personal API keys so we don't run on the same problem again. How much time it took @fernandodpr for them to provide a key? I guess we need to write it in HA documentation.

Mostly the same as @MrMarble, I asked for it yesterday at 17h and received an email today at 11 h. The method it's a little rudimentary, but it seems to work. I don't have a lot of spare time to test the new API, but as soon as I can I'll try It.

martinezpenya commented 2 years ago

I've requested it today, once I receive where can I test it? I'm new in this but I want to help, if I can. I've been looking around the code and haven't seen where is the url and de token that the component is using now.

azogue commented 2 years ago

Quickfix for those in a hurry: https://github.com/azogue/aiopvpc/pull/35#issuecomment-984023527