Creating Data Pipeline

[Neodymidius]

This issue refer to the completion of a data pipeline for the project.

Electric energy

• For the electricity production data set, a Python Script will be used to get the data. That is because only a post request can get the data from their website.
• Initial idea: Pull csv with Python script and then use JayVee for the further pipeline.

Weather data

• For the weather data, multiple data sources in form of text files will be pulled from the dwd servers.
• For good reference, multiple weather station will be taken and then stored.
• We will see, if the text file can be somehow used for jayvee or if python will firstly manipulate the text first into a csv file.

Database

• Goal is a SQLite database created with jayvee
• It shall be stored that we have for each weather station columns for temperature and other data (humidity e.g.)
• For electric energy production there is also one column. It is the hourly total energy production of all producers

[Neodymidius]

Regarding weather stations

• List of all stations around Germany are listed below:

  List

• Stations_id von_datum bis_datum Stationshoehe geoBreite geoLaenge Stationsname Bundesland
• 04931 19880101 20240521 371 48.6883 9.2235 Stuttgart-Echterdingen Baden-Württemberg
• 03668 19510101 20240521 314 49.5030 11.0549 Nürnberg Bayern
• 03379 19970701 20240521 515 48.1632 11.5429 München-Stadt Bayern
• 01420 19810101 20240521 100 50.0259 8.5213 Frankfurt/Main Hessen
• 01078 19760301 20240521 37 51.2960 6.7686 Düsseldorf Nordrhein-Westfalen
• 02014 19490101 20240521 55 52.4644 9.6779 Hannover Niedersachsen
• 01228 20020101 20240521 0 54.1651 6.3460 UFS TW Ems Hamburg
• 00433 19510101 20240521 48 52.4676 13.4020 Berlin-Tempelhof Berlin

[Neodymidius]

Regarding data collection

• For Energy Production, I had to request with a post request to smard.de the csv file.
• For weather data: I get zip files with a GET request for each station. There is then meta data that is not interesting for me plus the actual temperature data. Stored in a txt file, but very easy to convert to csv, because a semicolon is already separating every value

Regarding pipeline.sh

It will execute a hand full of programs: Python scripts for extracting the data, Jayvee files for adding constraints. Because there is a lot of data collected for all weather stations, I had to split up the pipelines because of heap memory limitations.