Use Bundesnetzagentur SMARD API instead of EPEX Spot Web Scraper

[TobiasBraeutigam]

Hi, thank you for your great component.

I just want to suggest, that you do not need the EPEX Spot Web Scraper anymore. Instead you could use the SMARD Plattform published by the Bundesnetzagentur. There you get all the data from official sources for free and via API.

SMARD is the information platform of the Federal Network Agency about the German electricity market. All users can access visually and tabularly prepared data at www.smard.de. The API documented here provides access to the electricity market data of the Federal Network Agency.

https://www.smard.de/home/marktdaten?marketDataAttributes=%7B%22resolution%22:%22hour%22,%22from%22:1684792800000,%22to%22:1685051999999,%22moduleIds%22:%5B8004169%5D,%22selectedCategory%22:8,%22activeChart%22:true,%22style%22:%22color%22,%22categoriesModuleOrder%22:%7B%7D,%22region%22:%22DE%22%7D

https://github.com/bundesAPI/smard-api

https://smard.api.bund.dev/

I hope it helps.

Best regards, Tobias

[mampfes]

Thank you, this is really intersting information. I wasn't aware of that service. I will try to integrate it.

[TobiasBraeutigam]

Hi, here is an working example for loading the data from smard:

import datetime
import requests
import pandas as pd

# Mögliche Filter:
# Stromerzeugung
# 1223 - Braunkohle
# 1224 - Kernenergie
# 1225 - Wind Offshore
# 1226 - Wasserkraft
# 1227 - Sonstige Konventionelle
# 1228 - Sonstige Erneuerbare
# 4066 - Biomasse
# 4067 - Wind Onshore
# 4068 - Photovoltaik
# 4069 - Steinkohle
# 4070 - Pumpspeicher
# 4071 - Erdgas

# Stromverbrauch
# 410 - Gesamt (Netzlast)
# 4359 - Residuallast
# 4387 - Pumpspeicher

# Marktpreis
# 4169 - Deutschland/Luxemburg
# 5078 - Anrainer DE/LU
# 4996 - Belgien
# 4997 - Norwegen 2
# 4170 - Österreich
# 252 - Dänemark 1
# 253 - Dänemark 2
# 254 - Frankreich
# 255 - Italien (Nord)
# 256 - Niederlande
# 257 - Polen
# 258 - Polen
# 259 - Schweiz
# 260 - Slowenien
# 261 - Tschechien
# 262 - Ungarn

# Prognostizierte Erzeugung
# 3791 - Offshore
# 123 - Onshore
# 125 - Photovoltaik
# 715 - Sonstige
# 5097 - Wind und Photovoltaik
# 122 - Gesamt

smard_filter = 4169

# Land / Regelzone / Marktgebiet:
# DE - Land: Deutschland
# AT - Land: Österreich
# LU - Land: Luxemburg
# DE-LU - Marktgebiet: DE/LU (ab 01.10.2018)
# DE-AT-LU - Marktgebiet: DE/AT/LU (bis 30.09.2018)
# 50Hertz - Regelzone (DE): 50Hertz
# Amprion- Regelzone (DE): Amprion
# TenneT - Regelzone (DE): TenneT
# TransnetBW - Regelzone (DE): TransnetBW
# APG - Regelzone (AT): APG
# Creos - Regelzone (LU): Creos
smard_region = "DE-LU"

# Auflösung der Daten:
# hour - Stündlich
# quarterhour - Viertelstündlich
# day - Täglich
# week - Wöchentlich
# month - Monatlich
# year - Jährlich
smard_resolution = "hour"

# Anfrage der SMARD Daten
def get_smard_response(url):    
    response = requests.get(url)
    if response.status_code == 200:
        smard_response = response.json()
    else:
        print(f"Fehler. Statuscode: {response.status_code}")               
    return smard_response

# Wandelt ein Unix-Timestamp (in Millisekunden) in ein Date Objekt
def timestamp_to_date(timestamp):    
    date = pd.to_datetime(timestamp, unit='ms')
    return date

# Wandelt ein Date Objekt in einen Unix-Timestamp (in Millisekunden)
def date_to_timestamp(date):    
    timestamp = date.value // 10**9
    timestamp *= 1000
    return timestamp

url_prefix = "https://www.smard.de/app/chart_data/"

# Hole verfügbare Timestamps
url = url_prefix + f"{smard_filter}/{smard_region}/index_{smard_resolution}.json"
smard_response = get_smard_response(url)

smard_start_ts = smard_response['timestamps'][-1]
smard_start_ts
date = timestamp_to_date(smard_start_ts)
date
date_to_timestamp(date)

# Hole Zeitreihendaten ab Timestamp
url = url_prefix + f"{smard_filter}/{smard_region}/{smard_filter}_{smard_region}_{smard_resolution}_{smard_start_ts}.json"
smard_response = get_smard_response(url)

# Aufbereitung der Zeitreihen
data_df = pd.DataFrame(smard_response["series"], columns=["date", "value"])
data_df.date = pd.to_datetime(data_df.date, unit='ms')
data_df.set_index(keys="date", drop=True, inplace=True)

data_df = data_df.tz_localize(tz="UTC")
data_df = data_df.tz_convert('Europe/Berlin')

data_df
data_df.plot()

data_df.dropna().plot()```

Result:
![image](https://github.com/mampfes/ha_epex_spot/assets/89102799/6047471b-4fcb-41d5-9f6a-10fde36b9b0a)

[mampfes]

Thanks, this is really helpful!

[mampfes]

Hm, I am struggling with the returned data...

For the following attributes

        smard_filter = 4169
        smard_region = "DE-LU"
        smard_resolution = "hour"

I only get market prices for today. The price values for tomorrow are all null in the json response. Do you see the same values?

Same result if I visualize the data via the web UI: --> Today is 08.06.2023, so there are no values for tomorrow.

[TobiasBraeutigam]

Yes, I have the same issue. I mailed SMARD. The days before ist was still working. Other sources like ENTSO-E do provide the missing data.

Gesendet von Outlook für iOShttps://aka.ms/o0ukef

---

Von: Steffen Zimmermann @.> Gesendet: Thursday, June 8, 2023 8:27:32 PM An: mampfes/ha_epex_spot @.> Cc: TobiasBraeutigam @.>; Author @.> Betreff: Re: [mampfes/ha_epex_spot] Use Bundesnetzagentur SMARD API instead of EPEX Spot Web Scraper (Issue #29)

Hm, I am struggling with the returned data...

For the following attributes

    smard_filter = 4169
    smard_region = "DE-LU"
    smard_resolution = "hour"

I only get market prices for today. The price values for tomorrow are all null in the json response. Do you see the same values?

— Reply to this email directly, view it on GitHubhttps://github.com/mampfes/ha_epex_spot/issues/29#issuecomment-1583137290, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AVHZTT4HIOGKCBRRSTYDHKDXKIKRJANCNFSM6AAAAAAYNZ2DI4. You are receiving this because you authored the thread.Message ID: @.***>

[TobiasBraeutigam]

The Data is available again:

[ProfDrYoMan]

Would be great to see this official source being integrated.

[mampfes]

fixed by #32