sensor.pvpc: repeated bad data update, mark component as unavailable source

[etatus]

Home Assistant shows the following error in the pvpc_hourly_pricing integration:

Logger: aiopvpc.pvpc_data Source: /usr/local/lib/python3.9/site-packages/aiopvpc/pvpc_data.py:231 ... Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: ...

It seems the same happened before: https://github.com/azogue/aiopvpc/issues/31

[adrigrillo]

I am facing the same error.

2021-11-29 14:39:31 ERROR (MainThread) [aiopvpc.pvpc_data] Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers:  <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '863', 'X-Iinfo': '13-79778917-0 2cNN RT(1638193170742 21) q(0 -1 -1 -1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000')>

As it is an forbidden error, I have also tried to access perform the petition from the browser from the same network and I can download the json file without any problem. I think that is going to be a temporal, maybe related with a cookie?

[r-jordan]

Same error in homeassistant log file:

Logger: aiopvpc.pvpc_data
Source: /usr/local/lib/python3.9/site-packages/aiopvpc/pvpc_data.py:231
First occurred: 14:14:59 (20 occurrences)
Last logged: 16:43:00

Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '862', 'X-Iinfo': '14-97234813-0 2cNN RT(1638198479888 49) q(0 -1 -1 0) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1397_1885724=T+grIQDBI1FS6a7DXCRjE8/spGEAAAAAS0m3mp3EYpPIkOEQ1q3b+w==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '862', 'X-Iinfo': '14-97480235-0 2cNN RT(1638199862119 55) q(0 -1 -1 6) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1397_1885724=ZlXBGZxI9VYXq8DDXCRjEzbypGEAAAAAHgdkVabP+EeNm3+QfF4oOw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '863', 'X-Iinfo': '11-68345129-0 2cNN RT(1638199882224 22) q(0 -1 -1 -1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '863', 'X-Iinfo': '12-78150422-0 2cNN RT(1638199922279 20) q(0 -1 -1 -1) r(1 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000')>
Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '858', 'X-Iinfo': '2-2590203-0 0CNN RT(1638200580020 35) q(0 -1 -1 0) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'incap_ses_1397_1885724=G3cEB/4nw0zpCcrDXCRjEwT1pGEAAAAAMMLm0qNXleg/Q3gA9ybFjw==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')>

However the data is retrieved without problem from a curl command:

$ curl -v 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29'
*   Trying 45.60.63.41...
* TCP_NODELAY set
* Connected to api.esios.ree.es (45.60.63.41) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=imperva.com
*  start date: Nov 25 12:12:16 2021 GMT
*  expire date: May 27 03:06:52 2022 GMT
*  subjectAltName: host "api.esios.ree.es" matched cert's "*.esios.ree.es"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Atlas R3 DV TLS CA H2 2021
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x55d36c904610)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /archives/70/download_json?locale=es&date=2021-11-29 HTTP/2
> Host: api.esios.ree.es
> User-Agent: curl/7.58.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 200 
< etag: "39975b14"
< content-type: application/json
< content-length: 9506
< server: MSX-2
< strict-transport-security: max-age=63072000;
< x-frame-options: SAMEORIGIN
< status: 200 OK
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< content-disposition: download; filename="PVPC_CURV_DD"
< content-transfer-encoding: binary
< x-request-id: 6c98b526-a780-4994-82d2-2710af90fec1
< x-runtime: 0.028711
< referrer-policy: no-referrer
< cache-control: max-age=240, public
< expires: Mon, 29 Nov 2021 15:58:56 GMT
< date: Mon, 29 Nov 2021 15:54:56 GMT
< set-cookie: visid_incap_1885724=gDG5DV9nQ7+FsvCd9r7ugtD3pGEAAAAAQUIPAAAAAAADJSZuuBv48p1hFnQeSyGk; expires=Tue, 29 Nov 2022 10:36:50 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None
< set-cookie: incap_ses_482_1885724=yXKkEWSmoxnEVPBi+WiwBtD3pGEAAAAAIuP3XDJh9CmVMM41anvrig==; path=/; Domain=.esios.ree.es; Secure; SameSite=None
< x-cdn: Imperva
< x-iinfo: 0-56647988-56647099 2CNN RT(1638201296397 0) q(0 0 0 0) r(0 0)
< 
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{"PVPC":[{"Dia":"29/11/2021","Hora":"00-01", ...

[adrigrillo]

I have been doing more testing and everything leads to think that REE operator has forbidden all the petitions using the current user agent aioPVPC Python library. Concretely, any user agent that contains the word aioPVPC returns the error.

This can be easily tested using the following command and changing the user agent:

curl --location --request GET 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' \
--header 'User-Agent: aioPVPC'

[deltazerorsan]

I have been doing more testing and everything leads to think that REE operator has forbidden all the petitions using the current user agent aioPVPC Python library. Concretely, any user agent that contains the word aioPVPC returns the error.

This can be easily tested using the following command and changing the user agent:

curl --location --request GET 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-11-29' \
--header 'User-Agent: aioPVPC'

... you can leave empty the User-Agent: or put any garbage and the result is correct.

This seems to be done on purpose by REE...

[etatus]

Here is the line where User-Agent is set in the code: https://github.com/azogue/aiopvpc/blob/30836536a32b99f83dbe083a736bf457f1b66983/aiopvpc/pvpc_data.py#L35

It should be changed to other text not containing aioPVPC (e.g. simply 'Python library')

[r-jordan]

I don't know why REE has banned the aioPVPC text in the User-Agent but they may probably ban the new one with the same reason. Anyone knows how to ask REE about that?

One reason could be that the User-Agent does not contain the correct syntax which is explained here:

https://developer.mozilla.org/es/docs/Web/HTTP/Headers/User-Agent

What do you think?

[sermayoral]

I suppose the REE API is getting lots of requests with 'aioPVPC' text in the User-Agent, so they have decided to ban these requests in order to decrease the requests number they get.

[liebefuroscar]

A solution could be to use the personal token ?? I have one. I could try ...

EDIT: https://community.home-assistant.io/t/esios-ree-integration-spain-national-network-of-electricity/155115/137

WORKING!!!

[adrigrillo]

Seems that asking for a personal key and using it for the request is the way to go. We can remove the user agent or fake a browser as a temporal solution, but still, if they still feel that we are overloading the API, they will close permanently without identification.

A solution could be to use the personal token ??

~Is there information about how to get it? Thanks beforehand.~

Never mind, it's here https://www.esios.ree.es/es/pagina/api.

[liebefuroscar]

I read that there were about 26,000 requests a day ... Just send an email to consultasios@ree.es saying: I would like to request from REE a unique and personal token to make calls to their API.

In my case, in 10 minutes I received my token

[r-jordan]

Hi guys, I made a temporal solution.

First, install the custom deps development addon from this repository:

https://github.com/home-assistant/addons-development

Then start it with this config:

pypi:
  - aiopvpc==2.2.4
apk: []

Wait until the addon stops. Then go to the directory /config/deps/lib/python3.9/site-packages/aiopvpc and edit the pvpc_data.py with a different User-Agent string. Finally, restart HA and you got it.

A permanent solution most probably will require a personal token ...

Hope it helps.

[joseant]

Another temporal solution:

Copy this folder to custom_components https://github.com/home-assistant/core/tree/dev/homeassistant/components/pvpc_hourly_pricing

and add this to the file sensor.py

import aiopvpc

aiopvpc.pvpc_data._REQUEST_HEADERS = {
    "User-Agent": "Python library",
    "Accept": "application/json",
}

restart home assistant

[azogue]

Another temporal solution:

Best temporal solution --> https://github.com/azogue/aiopvpc/pull/35#issuecomment-984023527