🤟 Fix user-agent banning by ESIOS server

[azogue]

closes #34

related to https://github.com/home-assistant/core/issues/60556

Quick-Fix motivated by the last change in the ESIOS server (on 2021-11-30 😱), which is now apparently banning HomeAssistant requests, filtering us out because of the 'User-Agent' headers data 😤, as the server is returning a 403 status code error for a PUBLIC url 🤷.

So, until future moves to authenticated endpoints, or alternative data sources, let's continue playing 🐈 and 🐁 with the ESIOS webadmins 🤣🤣

[Serious note for those webadmins]: I'm just a volunteer, and I'm an email away for contact, so please, if you are reading this, give me a call before filtering us all out 🤝🥺

Changes:

• :zap: Substantially decrease the number of API requests to ESIOS, avoiding unnecessary calls to refresh data for the same electricity prices. Before, when used from the pvpc_hourly_pricing HA Core integration, the ESIOS API was called 2 times/hour from 0h to 20h, and 4 times/hour in the evening, from 20h to 0h, retrieving today + tomorrow prices. This makes a total of ~56 requests/day, which is not a lot 😅, but it seems the aggregated total for the HA user base (🔥 >30k requests/day just counting users pushing HA analytics) is being some kind of a problem for ESIOS, as it looks like they're trying to bane us 🥺😭 Now, the API handler avoids calls to retrieve already available prices, cutting down the number of requests to just 1-2 requests/day 🤩

• :bug: Set standard User-Agent header info, to try to avoid server-side banning 🙈, and rotate it if banning is detected, using common User-Agent browser identifiers.

• :recycle: Minor code refactor to prepare for future library changes, in order to move to authenticated API endpoints in future versions.

[azogue]

🚀 For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

🚀 pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration 🧪, try this one instead 😜: esios.zip

[deltazerorsan]

🚀 For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

🚀 pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration 🧪, try this one instead 😜: esios.zip

The custom pvpc is running, thanks.

I'm also trying esios, and I got a token today, but I can't see the Esios in the integration list... (yes the folder is inside custom_components)

HA over raspi, core-2021.11.5.

Any directions appreciated!

[martinezpenya]

rocket For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

rocket pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration test_tube, try this one instead stuck_out_tongue_winking_eye: esios.zip

Yes! it works again with custom pvpc.

Tomorrow I'll probably get the esios token and test the other one. Thank you!.

[jjprz]

🚀 For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

🚀 pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration 🧪, try this one instead 😜: esios.zip

Testing...

Logger: custom_components.esios.esios_data.api Source: custom_components/esios/esios_data/api.py:242 Integration: ESIOS API Data (documentation, issues) First occurred: 00:00:44 (1 occurrences) Last logged: 00:00:44

2021-12-01 23:00:00+00:00 not found in {} -> list index out of range

[bvis]

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration 🧪, try this one instead 😜: esios.zip

I have now an ESIOS token, do you have instructions about its configuration with this esios.zip version?

[amibumping]

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration test_tube, try this one instead stuck_out_tongue_winking_eye: esios.zip

I have now an ESIOS token, do you have instructions about its configuration with this esios.zip version?

Go to integrations, add new one, look up for esios api, and then enter your token and save. I didn't test it myself yet.

EDIT: working

[jeancsamin]

Applying the quick fix using token ends up in REE forbidding the API call again. Not sure how the code works, but if it triggers many updates while the price curves don't change, i.e. we are pulling the same data over and over again, they end up banning the access via token as well.

[fernandodpr]

I'm now using the new ESIOS integration. It works!! 😃🥳. Congratulations for your work and will inform you in case of any issues. Could you provide the GitHub link of the new integration?

[deltazerorsan]

I'm now using the new ESIOS integration. It works!! 😃🥳. Congratulations for your work and will inform you in case of any issues. Could you provide the GitHub link of the new integration?

Hi @fernandodpr,

I'm trying to install ESIOS as a custom component but the integration do not appear on the list, Is there any special procedure I'm missing?

Thanks!

[fernandodpr]

Hi @deltazerorsan, I followed the instructions provided by @azogue :

1. Downloaded the zip file " esios.zip" and extracted in the folder config/custom_components. This should create a folder called esios.
2. Reboot your HA instance.
3. Go to Configuration -> Integrations-> Press the '+ ADD INTEGRATION' button -> Search for "ESIOS API Data"
4. Add the integration, by filling the required inputs. API TOKEN, location and a few more checkboxes.
5. All done! Search for your sensor, mine It's called sensor.esios_pvpc

[jumavi]

Hola, La integración con el token personal funciona perfecto. He desintalado la versión oficial y obviamente reiniciado pero me siguen apareciendo dos errores. ¿que puede ser?

Logger: aiopvpc.pvpc_data Source: /usr/local/lib/python3.9/site-packages/aiopvpc/pvpc_data.py:231 First occurred: 1 de diciembre de 2021 21:57:38 (96 occurrences) Last logged: 16:27:19

Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-03-28' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '859', 'X-Iinfo': '7-36927098-0 2NNN RT(1638458838950 41) q(0 -1 -1 1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'visid_incap_1885724=KidT3bTWTI2TGT+AnDKTHtblqGEAAAAAQUIPAAAAAABu0gwW9yWPEzn0GXj7gbnN; expires=Thu, 01 Dec 2022 17:05:24 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None', 'Set-Cookie': 'incap_ses_510_1885724=EKq/CL/EZERT51dhzOITB9blqGEAAAAAouTHo/Y/3dMijGy+Jb4gqg==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')> Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-12-01' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '860', 'X-Iinfo': '6-22198842-0 2NNN RT(1638458839404 33) q(0 -1 -1 -1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'visid_incap_1885724=m3ogOnXBSIijujykXcxQLdflqGEAAAAAQUIPAAAAAADF1cT7IWYCtULlP3xTCLj5; expires=Thu, 01 Dec 2022 17:05:24 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None', 'Set-Cookie': 'incap_ses_510_1885724=7SSdJDKJmWgr6FdhzOITB9flqGEAAAAA0ZxM5jMuqdXVRK1Rw9yxvw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')> Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '860', 'X-Iinfo': '4-73046775-0 2NNN RT(1638458839401 42) q(0 -1 -1 -1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'visid_incap_1885724=20eopxSuSEeSU7o8Ou/Y0dflqGEAAAAAQUIPAAAAAACScp5Jrd363bO15qK7STBU; expires=Thu, 01 Dec 2022 17:04:13 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None', 'Set-Cookie': 'incap_ses_510_1885724=kjx2QI2gXikx6FdhzOITB9flqGEAAAAAMxelCM76E5mJNSfGwzFBZw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')> Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-30' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '860', 'X-Iinfo': '4-73046778-0 2NNN RT(1638458839404 48) q(0 -1 -1 -1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'visid_incap_1885724=lurvS6o6QGq2hXiMd8GQH9flqGEAAAAAQUIPAAAAAACqK3BpygLpPOEmLOchKK6N; expires=Thu, 01 Dec 2022 17:04:13 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None', 'Set-Cookie': 'incap_ses_510_1885724=sVE9LNpD1Ws16FdhzOITB9flqGEAAAAAXbDb0j6H+CmpxtEaJiNVsw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')> Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-12-02' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '861', 'X-Iinfo': '10-86545175-0 2cNN RT(1638458839406 66) q(0 -1 -1 -1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'visid_incap_1885724=EXkD5c+USy+k138g+QWOB9flqGEAAAAAQUIPAAAAAADMaBYOj1gZR3PctkN7uz5E; expires=Thu, 01 Dec 2022 17:04:13 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None', 'Set-Cookie': 'incap_ses_510_1885724=+gt6Q9ixgxpB6FdhzOITB9flqGEAAAAAycP95M+uQqkHT8QQaMnomg==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>

Logger: aiopvpc.pvpc_data Source: /usr/local/lib/python3.9/site-packages/aiopvpc/pvpc_data.py:425 First occurred: 1 de diciembre de 2021 21:57:38 (38 occurrences) Last logged: 16:27:19

BAD Download of PVPC prices from 2021-11-29 09:00:00 to 2021-12-02 13:27:15.397859 in 0.13 sec BAD Download of PVPC prices from 2021-11-29 09:00:00 to 2021-12-02 14:27:15.400131 in 0.13 sec BAD Download of PVPC prices from 2021-11-29 09:00:00 to 2021-12-02 15:27:15.401872 in 0.26 sec BAD Download of PVPC prices from 2021-03-28 01:00:00 to 2021-03-28 03:00:00 in 0.17 sec BAD Download of PVPC prices from 2021-11-29 09:00:00 to 2021-12-02 16:27:15.403588 in 0.14 sec

[martinezpenya]

Hi @deltazerorsan, I followed the instructions provided by @azogue :

1. Downloaded the zip file " esios.zip" and extracted in the folder config/custom_components. This should create a folder called **esios**.

2. Reboot your HA instance.

3. Go to Configuration -> Integrations-> Press the '+ ADD INTEGRATION' button -> Search for "_ESIOS API Data_"

4. Add the integration, by filling the required inputs. [API TOKEN](https://api.esios.ree.es/), location and a few more checkboxes.

5. All done! Search for your sensor, mine It's called **sensor.esios_pvpc**

I've followed this steps with my personal token and is working!. Thank you!

[deltazerorsan]

SOLVED, IT WAS A PROBLEM IN HA WITH ANY MODULE

ESIOS WORKING PROPERLY:

Hi @deltazerorsan, I followed the instructions provided by @azogue :

1. Downloaded the zip file " esios.zip" and extracted in the folder config/custom_components. This should create a folder called esios.
2. Reboot your HA instance.
3. Go to Configuration -> Integrations-> Press the '+ ADD INTEGRATION' button -> Search for "ESIOS API Data"
4. Add the integration, by filling the required inputs. API TOKEN, location and a few more checkboxes.
5. All done! Search for your sensor, mine It's called sensor.esios_pvpc

Yeah! My problem is between step 2 and 3... no ESIOS API Data... Are you using a development version of HA?

Thanks

[fernandodpr]

.... ESIOS API Data... Are you using a development version of HA?

No, I'm using standard version of HA. Could you look into /config/info. To check if HA It's detecting the integration?

[deltazerorsan]

.... ESIOS API Data... Are you using a development version of HA?

No, I'm using standard version of HA. Could you look into /config/info. To check if HA It's detecting the integration?

Nope, it's not there... I have other three custom components, including pvpc, working correctly... mmm, so estrange...

I got the zip from here: https://github.com/azogue/aiopvpc/files/7531844/esios.zip

[etatus]

Is there any official repository for the API version of pvpc_hourly_pricing integration (ESIOS API connector)

The one referenced in the integration (esios.zip) doesn't seem to work: https://github.com/azogue/ha-esios

[josecarre]

🚀 For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

🚀 pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration 🧪, try this one instead 😜: esios.zip

gracias :), llevaba varios días usando velas por no saber el precio! xD. Ya vuelve a funcionar con el aiopvpc 2.3.0 gracias :), estoy a la espera de recibir el token de REE, cuando lo tenga me paso a la nueva/esios. Se seguirían usando los mismos sensores en HA? por mantener el histórico de precios, o son sensores nuevos? thanks!

[fernandodpr]

@josecarre el sensor nuevo por defecto tiene otra id. Lo que puedes hacer es modificar la id una vez instalado y colocar la misma que tenías anteriormente (recuerda eliminar la otra integración). De esa forma en los gráficos tendrás histórico, en grafana también en caso de que lo uses y tampoco tendrás que modificar tus automatizaciones.

[mrtmsgh]

rocket For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your config/custom_components folder, so your HomeAssistant instance with the PVPC price sensor could work again after a simple restart (using python3.9 and HA Core >= 2021.11.0!)

rocket pvpc_hourly_pricing.zip

Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration test_tube, try this one instead stuck_out_tongue_winking_eye: esios.zip

I've tried the two options and both are working OK for me. Thanks!

[maarlo]

I have tried esios.zip and works properly. Thanks!

[samolo75]

I tried also esios.zip with the personal token and it works. But it only shows the price on the day, not for the next day. Am I alone?

[deltazerorsan]

You need to wait until 20:00 ... 20:30 and then you have next day. Untill 00:00.

It works fine for me.

El mar., 21 dic. 2021 12:31, samolo75 @.***> escribió:

I tried also esios.zip with the personal token and it works. But it only shows the price on the day, not for the next day. Am I alone?

— Reply to this email directly, view it on GitHub https://github.com/azogue/aiopvpc/pull/35#issuecomment-998704468, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM33FISVUM6M7DDNMHDGW2DUSBQQ5ANCNFSM5JFLYPSQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[samolo75]

You need to wait until 20:00 ... 20:30 and then you have next day. Untill 00:00. It works fine for me. El mar., 21 dic. 2021 12:31, samolo75 @.> escribió: … I tried also esios.zip with the personal token and it works. But it only shows the price on the day, not for the next day. Am I alone? — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM33FISVUM6M7DDNMHDGW2DUSBQQ5ANCNFSM5JFLYPSQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.>

I am pretty sure I did check late on the day, for a few days and I never got the next day prices. I will keep checking. Thanks

[deltazerorsan]

Yeah it's a little trickier cause the attributes for next day are present only between the time REE offers them and 00:00.

Right now 22:50 CET I can see the "next day" attributes, like:

min_price (next day): 0.42475 min_price_at (next day): 4

but later they disappear and you get an Unknown wherever you are using it... But tomorrow they reappear whit the values of the next day, and so on.

[samolo75]

Yeah it's a little trickier cause the attributes for next day are present only between the time REE offers them and 00:00.

Right now 22:50 CET I can see the "next day" attributes, like:

min_price (next day): 0.42475 min_price_at (next day): 4

but later they disappear and you get an Unknown wherever you are using it... But tomorrow they reappear whit the values of the next day, and so on.

Ok. Sorry. It actually works fine when I checked yesterday. I guess I didn't do it right before.