search

yannickrud / made-project

Repository of Yannick Rudolf for the Methods of Advanced Data Engineering course at FAU SS24
https://oss.cs.fau.de/teaching/specific/made/
Creative Commons Attribution 4.0 International
0 stars 0 forks source link

Extract the data #2

Open yannickrud opened 4 months ago

yannickrud commented 4 months ago

Extract the data from the datasources from issue #1

yannickrud commented 4 months ago

Data from https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/monthly/weather_phenomena/historical/ is extracted to sql table in pipeline.py

It is data about weather phenomenons on a montly basis in germany

yannickrud commented 3 months ago

it is not monthly data but the anual data for weather phenomena

yannickrud commented 3 months ago

The meta data is extracted from txt file. Probably markdown table but with weird spacing

The data of the stations is within a zip file. That contains meta data which is split in more files than in the meta data file described above but does not contain more data. The actual data about weather phenomena is in a text file that contains comma separated values