bundesAPI / smard-api

https://smard.api.bund.dev
54 stars 8 forks source link

Include information about download endpoint #12

Open niwla23 opened 1 year ago

niwla23 commented 1 year ago

There is another way to fetch data from the API as CSV. This is probably more useful for some usecases. The endpoint URL is https://www.smard.de/nip-download-manager/nip/download/market-data . It takes only Post requests with this body:

{
    "request_form": [
        {
            "format": "CSV",
            "moduleIds": [
                2000122,
                2005097,
                2000715,
                2000125,
                2003791,
                2000123
            ],
            "region": "DE",
            "timestamp_from": 1663452000000,
            "timestamp_to": 16634916000000,
            "type": "discrete",
            "language": "de"
        }
    ]
}

You can get the moduleIds from the "Daten herunterladen" Page on smard.de . For some weird reason only the moduleIds used by the download form there work together. For example,. you can't combine consumption and production data into one request.

Maybe you want to include this endpoint in the docs?

I have also built a currently incomplete (supports only production, forecast and consumption) go client for it: https://github.com/niwla23/smard-go

EDIT: these are the id sets that work together:

productionModuleIds := []int{1001224, 1004066, 1004067, 1004068, 1001223, 1004069, 1004071, 1004070, 1001226, 1001228, 1001227, 1001225}
productionForecastIds := []int{2000122, 2005097, 2000715, 2000125, 2003791, 2000123}
consumptionModuleIds := []int{5000410, 5004387, 5004359}
ThomasRegier commented 1 year ago

I'm usiung this strategy as well. Some more useful stuff which I realized quick and dirty. As well as the moduleIds for Trade (import / Export) and Market (prices / kwh).

def get_time_in_mil(date_end = dt.datetime.now(),days = 8):
    """Gets time_start and time_end in milli seconds for downloads from the page: 'https://www.smard.de/nip-download-manager/nip/download/market-data'. Download will be for data from today backwards for the number of days specified in 'days' """
    days_dif = (date_end - dt.datetime(2022, 8, 1)).days
    # time in milli seconds. 1 / 1000 seconds.
    a = 1659304800000 # 1.8.22 day start
    b = 1659391199999 # 1.8.22 day end
    day_length = 60**2*24*1000 # day_length = (b-a+1)

    time_start = a + (days_dif - days + 1) * day_length
    time_end = a + days_dif * day_length + day_length - 1
    return time_start, time_end

time_start, time_end = get_time_in_mil(date_end = dt.datetime.now(),days = 3653)

url = 'https://www.smard.de/nip-download-manager/nip/download/market-data'

moduleIds_market = [8004169,8004170,8000251,8005078,8000252,8000253,8000254,8000255,8000256,8000257,8000258,8000259,8000260,8000261,8000262,8004996,8004997]

moduleIds_trade = [22004629,22004722,22004724,22004998,22004712,22004404,22004409,22004545,22004546,22004548,22004550,22004551,22004552,22004405,22004547,22004403,22004406,22004407,22004408,22004410,22004412,22004549,22004553]

moduleIds=moduleIds_trade
payload = {"request_form":[{"format":"CSV","moduleIds":moduleIds,"region":"DE","timestamp_from":time_start,"timestamp_to":time_end,"type":"discrete","language":"de"}]}

y = requests.post(url, json = payload_market)
with open(f'energy_market.csv', 'wb') as f:
    f.write(y.content)