Price list for energy injected on the grid when solar installations

albetaCOM commented 3 years ago

Hi,

Is it possible to add the hourly price for the energy injected on the grid aswell? (https://www.esios.ree.es/es/analisis/1739?vis=1&start_date=16-01-2021T00%3A00&end_date=16-01-2021T23%3A00&compare_start_date=15-01-2021T00%3A00&groupby=hour&compare_indicators=1013,1014,1015)

This would be awesome

azogue commented 3 years ago

Hi @albetaCOM,

that data is available in https://api.esios.ree.es/indicators/1739, but needs an access token from ESIOS (HTTP Token: Access denied. for the public)

This library was intended to support a Home Assistant Integration, by accessing only public files :(

If you know where to access that information without scraping the Analysis webpage (like this link: https://api.esios.ree.es/archives/70/download, used for the PVPC prices), I could try to implement this feature :)

Same goes for the prices of the new tariffs to come in the next month... I would adapt the library if I found how to read the new values🥺)

BTW, if you have an ESIOS token, you can access the prices with:

curl "https://api.esios.ree.es/indicators/1739" \
     -H 'Authorization: Token token=<your-token-here>'

Output for today is:

{
  "indicator": {
    "name": "Precio de la energía excedentaria del autoconsumo para el mecanismo de compensación simplificada (PVPC)",
    "short_name": "Precio de la energía excedentaria",
    "id": 1739,
    "composited": false,
    "step_type": "linear",
    "disaggregated": false,
    "magnitud": [
      {
        "name": "Precio",
        "id": 23
      }
    ],
    "tiempo": [
      {
        "name": "Hora",
        "id": 4
      }
    ],
    "geos": [
      {
        "geo_id": 3,
        "geo_name": "España"
      }
    ],
    "values_updated_at": "2021-05-15T20:19:45.000+02:00",
    "values": [
      {
        "value": 42.04,
        "datetime": "2021-05-16T00:00:00.000+02:00",
        "datetime_utc": "2021-05-15T22:00:00Z",
        "tz_time": "2021-05-15T22:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 38.38,
        "datetime": "2021-05-16T01:00:00.000+02:00",
        "datetime_utc": "2021-05-15T23:00:00Z",
        "tz_time": "2021-05-15T23:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 36.42,
        "datetime": "2021-05-16T02:00:00.000+02:00",
        "datetime_utc": "2021-05-16T00:00:00Z",
        "tz_time": "2021-05-16T00:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 21.91,
        "datetime": "2021-05-16T03:00:00.000+02:00",
        "datetime_utc": "2021-05-16T01:00:00Z",
        "tz_time": "2021-05-16T01:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 21.73,
        "datetime": "2021-05-16T04:00:00.000+02:00",
        "datetime_utc": "2021-05-16T02:00:00Z",
        "tz_time": "2021-05-16T02:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 21.62,
        "datetime": "2021-05-16T05:00:00.000+02:00",
        "datetime_utc": "2021-05-16T03:00:00Z",
        "tz_time": "2021-05-16T03:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 21.91,
        "datetime": "2021-05-16T06:00:00.000+02:00",
        "datetime_utc": "2021-05-16T04:00:00Z",
        "tz_time": "2021-05-16T04:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 25.07,
        "datetime": "2021-05-16T07:00:00.000+02:00",
        "datetime_utc": "2021-05-16T05:00:00Z",
        "tz_time": "2021-05-16T05:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 31.11,
        "datetime": "2021-05-16T08:00:00.000+02:00",
        "datetime_utc": "2021-05-16T06:00:00Z",
        "tz_time": "2021-05-16T06:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 21.91,
        "datetime": "2021-05-16T09:00:00.000+02:00",
        "datetime_utc": "2021-05-16T07:00:00Z",
        "tz_time": "2021-05-16T07:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 18.78,
        "datetime": "2021-05-16T10:00:00.000+02:00",
        "datetime_utc": "2021-05-16T08:00:00Z",
        "tz_time": "2021-05-16T08:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 10.84,
        "datetime": "2021-05-16T11:00:00.000+02:00",
        "datetime_utc": "2021-05-16T09:00:00Z",
        "tz_time": "2021-05-16T09:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 9.83,
        "datetime": "2021-05-16T12:00:00.000+02:00",
        "datetime_utc": "2021-05-16T10:00:00Z",
        "tz_time": "2021-05-16T10:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 7.33,
        "datetime": "2021-05-16T13:00:00.000+02:00",
        "datetime_utc": "2021-05-16T11:00:00Z",
        "tz_time": "2021-05-16T11:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 1.78,
        "datetime": "2021-05-16T14:00:00.000+02:00",
        "datetime_utc": "2021-05-16T12:00:00Z",
        "tz_time": "2021-05-16T12:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 0.81,
        "datetime": "2021-05-16T15:00:00.000+02:00",
        "datetime_utc": "2021-05-16T13:00:00Z",
        "tz_time": "2021-05-16T13:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 1.78,
        "datetime": "2021-05-16T16:00:00.000+02:00",
        "datetime_utc": "2021-05-16T14:00:00Z",
        "tz_time": "2021-05-16T14:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 1.78,
        "datetime": "2021-05-16T17:00:00.000+02:00",
        "datetime_utc": "2021-05-16T15:00:00Z",
        "tz_time": "2021-05-16T15:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 4.86,
        "datetime": "2021-05-16T18:00:00.000+02:00",
        "datetime_utc": "2021-05-16T16:00:00Z",
        "tz_time": "2021-05-16T16:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 35.77,
        "datetime": "2021-05-16T19:00:00.000+02:00",
        "datetime_utc": "2021-05-16T17:00:00Z",
        "tz_time": "2021-05-16T17:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 53.24,
        "datetime": "2021-05-16T20:00:00.000+02:00",
        "datetime_utc": "2021-05-16T18:00:00Z",
        "tz_time": "2021-05-16T18:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 70.82,
        "datetime": "2021-05-16T21:00:00.000+02:00",
        "datetime_utc": "2021-05-16T19:00:00Z",
        "tz_time": "2021-05-16T19:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 70.34,
        "datetime": "2021-05-16T22:00:00.000+02:00",
        "datetime_utc": "2021-05-16T20:00:00Z",
        "tz_time": "2021-05-16T20:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      },
      {
        "value": 64.88,
        "datetime": "2021-05-16T23:00:00.000+02:00",
        "datetime_utc": "2021-05-16T21:00:00Z",
        "tz_time": "2021-05-16T21:00:00.000Z",
        "geo_id": 3,
        "geo_name": "España"
      }
    ]
  }
}

jlopez77 commented 3 years ago

I know is not a super clean way to do it but...

You can use the "default" token: token="request_your_personal_token_sending_email_to_consultasios@ree.es"

And it will work. I've used it until I've got my own token while developing my own solution (thousands of requests as I'm a bit slow and I need to check every 10 seconds).

azogue commented 3 years ago

curl "https://api.esios.ree.es/indicators/1739" -H 'Authorization: Token token="request_your_personal_token_sending_email_to_consultasios@ree.es"'

it works 🤣😂🙈🎪

is that documented anywhere??

(IMHO, that kind of GET endpoints should not require authorization, same as the one with the PVPC prices, but that token make me 🤪)

jlopez77 commented 3 years ago

Na, of course it's not documented. I've been watching the webpage doing it's requests with this 'universal' token.

I completely agree with you that it should not require any kind of token.

azogue commented 3 years ago

I've been watching the webpage doing it's requests with this 'universal' token.

Para poder utilizar esta API deberán solicitar un token personal enviando un correo a consultasios@ree.es, ya que el token público que usa la propia página web cambia cada cierto tiempo.

🤣🤣🤡

jlopez77 commented 3 years ago

oh, what a shame.

I don't really believe they change the Token, but better safe than sorry.

azogue commented 3 years ago

If their own frontend does that, maybe it's not that bad idea to implement it here :)

The custom token could be optional (in the config flow in HA), same as these injection prices, and default to this special one if injection prices are enabled without token...

jlopez77 commented 3 years ago

Yep, that sounds fantastic. By configuration users can add their own token but if it's not provided, you can default to the universal one

azogue commented 3 years ago

Now I only need the time to do it 😅

jlopez77 commented 3 years ago

use your superpowers as always.

Last time you said "it will take time" was with the tariff change on JUN1 and it took 40 minutes.

jcsogo commented 3 years ago

This is a much more important feature with the new Energy panel in HA, as the calculations there should be matching the price for the energy returned to the grid. Do you plan to have some time to work on this in the near term? I can try to provide something but my coding skills are rusty - so I don't mind to spend time on this, but I won't if you're planning to do it soon enough.

Thanks for the library and the HA integration.

EddieKnight1 commented 3 years ago

A job well done deserves also being paid for it. Open Source doesn't mean "free for all" some times. If you manage to get some time to develop the necessary items for token request and energy injected, I'm up for sending you a "gift" as a reward. You say what you need and I'll provide it (if it doesn't go out of my budget).

azogue commented 2 years ago

Hi there 👋 @jcsogo, @jlopez77 @albetaCOM @EddieKnight1 @paniagualopez @danimart1991 @Albruaje

At last, I had some time to focus a little on this problem, and I did a proof of concept with a new integration using the ESIOS auth token to retrieve not only the PVPC, but also the grid injection prices and more indicators, like CO2 intensity.

I don't know yet if I'm going to split the actual library (this aiopvpc) or the HA Core integration (pvpc_hourly_pricing), or try to adapt it to use the new code if a token is provided in the config flow.

And I also don't know when I will have another time to continue working on this, maybe a week, maybe a month :(

But in the meantime, you could try it right now (and provide some feedback 😜), so I'm leaving it here for you 👍

esios.zip

Proceed as always and unzip the file in config/custom_components to create a new esios folder with the new integration (no need to remove/disable the current one). Then restart HA and add it by looking for 'Esios'.

This one requires a valid esios token, won't work other way.

-> Some screenshots for configuration:

-> How it looks:

azogue commented 2 years ago

BTW, @danimart1991, in this one, when the next-day prices are available, the ratios and other attributes are duplicated, fixing https://github.com/azogue/aiopvpc/issues/25, but also requires some changes for the lovelace card 😅

You'll see the new behavior this afternoon 🙈

danimart1991 commented 2 years ago

BTW, @danimart1991, in this one, when the next-day prices are available, the ratios and other attributes are duplicated, fixing #25, but also requires some changes for the lovelace card 😅

You'll see the new behavior this afternoon 🙈

I wait then for the update to avoid breaking changes with common users. Thanks for all your work 👍🏻

azogue commented 2 years ago

~~ but also requires some changes for the lovelace card~~

It looks like it's working perfectly fine 🤩

EddieKnight1 commented 2 years ago

Hola Eugenio,Soy Eddie Knight y lo prometido es deuda. Dime si esta dirección es válida para enviarte un vale de Amazon o si prefieres otro método.

Edit: Vale, veo que al responder el correo se envía aqui. Busco otro método :D

EddieKnight1 commented 2 years ago

Hi,

After downloading esiso zip and unpacking con custom_components, followed by adding the integration, I get a message "Configuration flow could not be loaded". The log reports this:

Logger: homeassistant.config_entries Source: config_entries.py:711 First occurred: 20:10:00 (1 occurrences) Last logged: 20:10:00

Error occurred loading configuration flow for integration esios: No module named 'zoneinfo'

I'm using python 3.8.12 at the moment, unable to upgrade yet to python 3.9.

azogue commented 2 years ago

@EddieKnight1,

Error occurred loading configuration flow for integration esios: No module named 'zoneinfo' I'm using python 3.8.12 at the moment, unable to upgrade yet to python 3.9.

That's the reason, zoneinfo is an internal module from python>=3.9

If you cannot upgrade, then install backports.zoneinfo 😜

If you are using a custom python environment to run HA, just run pip install backports.zoneinfo on it.

But, if you're using some dockerized HomeAssistant setup (different from the one in HA OS), and accessing it is complicated, I imagine it could also work by setting that dependency in the manifest.json file of the custom component, setting "requirements": ["backports.zoneinfo"],

BTW, Python3.8 is going to be deprecated soon in HA, hurry up with that upgrade, or move to HA OS 😊

BTW(2), the esios custom component requires a quite modern HA version to work with (2021.10/11 at least); it won't work other way :(

EddieKnight1 commented 2 years ago

@EddieKnight1,

Error occurred loading configuration flow for integration esios: No module named 'zoneinfo' I'm using python 3.8.12 at the moment, unable to upgrade yet to python 3.9.

That's the reason, zoneinfo is an internal module from python>=3.9

If you cannot upgrade, then install backports.zoneinfo 😜

If you are using a custom python environment to run HA, just run pip install backports.zoneinfo on it.

But, if you're using some dockerized HomeAssistant setup (different from the one in HA OS), and accessing it is complicated, I imagine it could also work by setting that dependency in the manifest.json file of the custom component, setting "requirements": ["backports.zoneinfo"],

BTW, Python3.8 is going to be deprecated soon in HA, hurry up with that upgrade, or move to HA OS 😊

BTW(2), the esios custom component requires a quite modern HA version to work with (2021.10/11 at least); it won't work other way :(

Thanks!

I've tried several ways but finally followed the plugin developer in TrueNAS for my Jail update. Python 3.9 installed and component installed. Waiting now for the personal API key.

azogue commented 2 years ago

@EddieKnight1 ,

I've tried several ways but finally followed the plugin developer in TrueNAS for my Jail update. Python 3.9 installed and component installed. Waiting now for the personal API key.

Sorry about the mess and lost time, it was entirely my fault 🙈🙏

I made an error in the code 🐛, making it impossible to run the component under python3.8.

And the saddest thing about that is that I discovered it by chance, just because yesterday I literally copied some of the new code over the old one to quick-fix an issue in the current HA integration, about sensor attributes using 48h prices in the evening... And that caused a failed pipeline in the PR in HA when running tests with python 3.8 😱

I'll try to re-publish the esios.zip as soon as possible, so it can work with py38, if some other user falls in the same scenario...

Python 3.9 installed and component installed.

Best thing you could do 👍, and again, sorry about the miss-guidance 🥺

Waiting now for the personal API key.

BTW, is that taking too long? I remember that request was relatively quick, like < 3h. Is it slower now?

EddieKnight1 commented 2 years ago

@EddieKnight1 ,

I've tried several ways but finally followed the plugin developer in TrueNAS for my Jail update. Python 3.9 installed and component installed. Waiting now for the personal API key.

Sorry about the mess and lost time, it was entirely my fault 🙈🙏

I made an error in the code 🐛, making it impossible to run the component under python3.8.

And the saddest thing about that is that I discovered it by chance, just because yesterday I literally copied some of the new code over the old one to quick-fix an issue in the current HA integration, about sensor attributes using 48h prices in the evening... And that caused a failed pipeline in the PR in HA when running tests with python 3.8 😱

I'll try to re-publish the esios.zip as soon as possible, so it can work with py38, if some other user falls in the same scenario...

Python 3.9 installed and component installed.

Best thing you could do 👍, and again, sorry about the miss-guidance 🥺

Waiting now for the personal API key.

BTW, is that taking too long? I remember that request was relatively quick, like < 3h. Is it slower now?

No worries about the issues. Its part of the test/UAT. Good thing is we've found the issue and got a workaround and solution.

Regarding the key, I sent the request out of working hours so probably will get delayed til Monday. I'll bring feedback next week.

EddieKnight1 commented 2 years ago

@EddieKnight1 ,

I've tried several ways but finally followed the plugin developer in TrueNAS for my Jail update. Python 3.9 installed and component installed. Waiting now for the personal API key.

Sorry about the mess and lost time, it was entirely my fault 🙈🙏 I made an error in the code 🐛, making it impossible to run the component under python3.8. And the saddest thing about that is that I discovered it by chance, just because yesterday I literally copied some of the new code over the old one to quick-fix an issue in the current HA integration, about sensor attributes using 48h prices in the evening... And that caused a failed pipeline in the PR in HA when running tests with python 3.8 😱 I'll try to re-publish the esios.zip as soon as possible, so it can work with py38, if some other user falls in the same scenario...

Python 3.9 installed and component installed.

Best thing you could do 👍, and again, sorry about the miss-guidance 🥺

Waiting now for the personal API key.

BTW, is that taking too long? I remember that request was relatively quick, like < 3h. Is it slower now?

No worries about the issues. Its part of the test/UAT. Good thing is we've found the issue and got a workaround and solution.

Regarding the key, I sent the request out of working hours so probably will get delayed til Monday. I'll bring feedback next week.

API key received and integration configured. The Energy setting in HA is also capturing the price/hour for the surplus now.

What a good job @azogue !

jlopez77 commented 2 years ago

I'm a bit late.

It works perfect.

ldsmovil commented 2 years ago

Perfect. Could be added to next core update?

jmmateos commented 2 years ago

How long does it take to send the token?

foglietti commented 2 years ago

How long does it take to send the token?

I have requested the token yesterday at 20:07 and received it 30 minutes ago. But I have read that someone received it in 10 minutes. I suppose it depends on the time of the day they receive the e-mail.

cyberdie commented 2 years ago

Tested and working as expected. Thanks!

enredador commented 2 years ago

Hello. I have been using the esios.zip custom_component with a personal token. It works fantastic! However, since 2021.12. the log reports some things upon setting up: `2021-12-21 18:59:31 WARNING (MainThread) [homeassistant.helpers.frame] Detected code that uses str for device registry entry_type. This is deprecated and will stop wor king in Home Assistant 2022.3, it should be updated to use DeviceEntryType instead. Please report this issue. Stack (most recent call last): File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/src/homeassistant/homeassistant/main.py", line 331, in sys.exit(main()) File "/usr/src/homeassistant/homeassistant/main.py", line 318, in main exit_code = runner.run(runtime_conf) File "/usr/src/homeassistant/homeassistant/runner.py", line 121, in run return loop.run_until_complete(setup_and_run_hass(runtime_config)) File "/usr/local/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete self.run_forever() File "/usr/local/lib/python3.9/asyncio/base_events.py", line 596, in run_forever self._run_once() File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once handle._run() File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run self._context.run(self._callback, self._args) File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 500, in _async_add_entity device = device_registry.async_get_or_create(*processed_dev_info) # type: ignore[arg-type] File "/usr/src/homeassistant/homeassistant/helpers/device_registry.py", line 360, in async_get_or_create report( # type: ignore[unreachable] File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 74, in report _LOGGER.warning(msg, stack_info=True) Stack (most recent call last): File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/src/homeassistant/homeassistant/main.py", line 331, in sys.exit(main()) File "/usr/src/homeassistant/homeassistant/main.py", line 318, in main exit_code = runner.run(runtime_conf) File "/usr/src/homeassistant/homeassistant/runner.py", line 121, in run return loop.run_until_complete(setup_and_run_hass(runtime_config)) File "/usr/local/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete self.run_forever() File "/usr/local/lib/python3.9/asyncio/base_events.py", line 596, in run_forever self._run_once() File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once handle._run() File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run self._context.run(self._callback, self._args) File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 500, in _async_add_entity device = device_registry.async_get_or_create(**processed_dev_info) # type: ignore[arg-type] File "/usr/src/homeassistant/homeassistant/helpers/device_registry.py", line 360, in async_get_or_create report( # type: ignore[unreachable] File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 74, in report _LOGGER.warning(msg, stack_info=True) 2021-12-21 18:59:31 WARNING (MainThread) [custom_components.esios.sensor] Setup of ESIOS PVPC (sensor.esios_pvpc) finished`

This is recorded twice, I suppose once for each sensor (pvpc and grid_injection). I'm reluctant to it but, should I return to standard integration? I tried to a couple of weeks ago, but auth errors were still happening.

enredador commented 2 years ago

I managed to solve it, changing two lines of sensor.py:

Adding: from homeassistant.helpers.device_registry import DeviceEntryType And in __init__ changing entry_type="service", to entry_type=DeviceEntryType.SERVICE,

So I can keep using this "token and injection price" version until this version is integrated in the core integration.