As STA I would like to integrate the new municipal weather forecasts of the Province of Bolzano, so to use them for the new infotainment system on the local trains

rcavaliere commented 9 months ago

[x] Organization of sharing of JSON files with weather service of the Province of Bolzano (@rcavaliere and @dulvui)
[x] Preparation of specification document (@rcavaliere)
[x] Development of Data Collector

Example of JSON file: SMOS_MCPL-WX_EXP_SIAG.zip

Specification proposal for data integration: 231019_SpecificheIntegrazione_NOI_v1.1.pdf

rcavaliere commented 9 months ago

@dulvui the weather service has confirmed me that they could make an automatic upload somewhere in a system controlled by us (e.g. FTP or similar). What do you suggest here, what could be imagine to do? Once integrated in the Open Data Hub DB, theoretically we would not need the JSON file anymore...

rcavaliere commented 8 months ago

@dulvui I have finalized the specification document, as usual. You can find in the issue summary and also here. Can you also provide me a feedback for the automatic upload of the JSON files, see previous comment? Thanks!

231019_SpecificheIntegrazione_NOI_v1.1.pdf

dulvui commented 8 months ago

@dulvui Awesome! I would suggest to use a simple S3 bucket, where they can post the JSON file with a simple POST request. I will create the S3 bucket and send you the details via email.

dulvui commented 7 months ago

@rcavaliere The data collector is now running in testing and here you can see the data:

WeatherForecast Stations https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/WeatherForecast?where=sactive.eq.true Data https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/WeatherForecast/*/latest?where=sactive.eq.true

WeatherForecastService Stations https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/WeatherForecastService?where=sactive.eq.true Data https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/WeatherForecastService/*/latest?where=sactive.eq.true

rcavaliere commented 7 months ago

@dulvui I have tested this new Data Collector. In general, as usual, the Data Collector works fine. I have however following findings:

[x] WeatherForecast and WeatherForecastService: just few data type available, we expect here more. Did we miss something here?
[x] WeatherForecast and WeatherForecastService: origin not set
[x] WeatherForecast: data type available as string not available in measurementstring
[x] Data types are not multilingual (no entries in type_metadata table)

dulvui commented 7 months ago

@rcavaliere Great! Oh I forgot about the string values. I'll add them now and fix the other issues too.

dulvui commented 7 months ago

@rcavaliere All issues should be fixed now.

The only datatypes with no translations are forecast-precipitation-max and forecast-precipitation-min of WeatherForecastService, because there is no translations given for them, but I could add some translations manually.

Here the links to the datatypes:

WeatherForecast DataTypes https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/WeatherForecast/*/latest?limit=-1&where=sactive.eq.true&select=tname,tmetadata

WeatherForecastService DataTypes https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/WeatherForecastService/*/latest?limit=-1&where=sactive.eq.true&select=tname,tmetadata

rcavaliere commented 7 months ago

@dulvui looks great now! I am not sure, but I have still the feeling that we still do not have all the data. In particular, for certain types, we are waiting for measurements with different periods, isn't it? At present I see measurements referring to a certain data type and one period only associated. Let's do not manual translation, just include the available translations provided by the API, so fine here as you implemented!

dulvui commented 7 months ago

@rcavaliere You are right, there is a problem with the periods. I will clean the database and do a new import, because I think this periods are sent by an older version of the data collector.

The problem could be also related to the fact, that we cannot store values that have a timestamp in the past of the latest timestamp of a dataset. So for example if the current forecast on the day 01.01.2023, makes forecasts until 05.01.2023, the latest timestamp for would be on 05.01.2023. So the next day 02.01.2023 only the forecasts from 05.01.2023 until 06.05.2023 would be saved. So the new values for the days between 02.01.2023 to 05.01.2023 would get lost.

This is a general problem we currently always have with forecasts. I will think about an solution and discuss it with you and Clements the next week.

niettadesantis commented 7 months ago

Hi @rcavaliere , I would like to keep updated about this topic, as this data could be interesting also for the tourism domain. It would be great if the data could be exposed with the same format as the weather endpoint is already doing: https://tourism.opendatahub.bz.it/swagger/index.html#/Weather

rcavaliere commented 7 months ago

@niettadesantis what do you mean with "format"? The format is the same ma the data structure is different, since we are speaking here mainly about numeric forecasts, not qualitative forecasts (texts). Moreover the forecasts are specific for each municipality, so the data structure is for sure different. Above you can find the end-points that will be made available. I don't see particular issue, if you want to consume this data you have to consider this API, but keep in mind that you should understand how you may want to present this additional, different information

niettadesantis commented 7 months ago

Hi @rcavaliere thanks for your comment: actually I meant "structure". ;-) @dulvui mentioned during the meeting today, that there are not only numeric values but also texts (just like "cloudy" or "sunny" etc.) Maybe the already present Weather endpoint of odh-tourism could be extended with the new data. For example we have the endpoint to retrieve the "districts" weather: https://tourism.opendatahub.bz.it/v1/Weather/District?language=en Here you can see, that we have for every district the current and forecast weather data structured like this (the following is the data for the district "Bolzano, Überetsch and Unterland"):

"BezirksForecast": [ { "date": "2023-11-24T00:00:00", "WeatherCode": "b", "WeatherDesc": "Partly cloudy", "WeatherImgUrl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_2.png", "Weathercode": "b", "Weatherdesc": "Partly cloudy", "WeatherImgurl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_2.png", "MaxTemp": 11, "MinTemp": -2, "Freeze": 3, "RainFrom": 0, "RainTo": 0, "Part1": 0, "Part2": 0, "Part3": 0, "Part4": 0, "Thunderstorm": 3, "SymbolId": 31, "Reliability": null }, { "date": "2023-11-25T00:00:00", "WeatherCode": "c", "WeatherDesc": "Cloudy", "WeatherImgUrl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_3.png", "Weathercode": "c", "Weatherdesc": "Cloudy", "WeatherImgurl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_3.png", "MaxTemp": 7, "MinTemp": 0, "Freeze": 2, "RainFrom": 0, "RainTo": 0, "Part1": 0, "Part2": 0, "Part3": 0, "Part4": 0, "Thunderstorm": 2, "SymbolId": 32, "Reliability": null }, { "date": "2023-11-26T00:00:00", "WeatherCode": "b", "WeatherDesc": "Partly cloudy", "WeatherImgUrl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_2.png", "Weathercode": "b", "Weatherdesc": "Partly cloudy", "WeatherImgurl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_2.png", "MaxTemp": 8, "MinTemp": 0, "Freeze": 2, "RainFrom": 0, "RainTo": 0, "Part1": 0, "Part2": 0, "Part3": 0, "Part4": 0, "Thunderstorm": 2, "SymbolId": 31, "Reliability": null }, { "date": "2023-11-27T00:00:00", "WeatherCode": "c", "WeatherDesc": "Cloudy", "WeatherImgUrl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_3.png", "Weathercode": "c", "Weatherdesc": "Cloudy", "WeatherImgurl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_3.png", "MaxTemp": 5, "MinTemp": -2, "Freeze": 3, "RainFrom": 0, "RainTo": 0, "Part1": 0, "Part2": 0, "Part3": 0, "Part4": 0, "Thunderstorm": 3, "SymbolId": 32, "Reliability": null }, { "date": "2023-11-28T00:00:00", "WeatherCode": "c", "WeatherDesc": "Cloudy", "WeatherImgUrl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_3.png", "Weathercode": "c", "Weatherdesc": "Cloudy", "WeatherImgurl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_3.png", "MaxTemp": 8, "MinTemp": 0, "Freeze": 2, "RainFrom": 0, "RainTo": 2, "Part1": 1, "Part2": 1, "Part3": 0, "Part4": 0, "Thunderstorm": 2, "SymbolId": 32, "Reliability": null }, { "date": "2023-11-29T00:00:00", "WeatherCode": "b", "WeatherDesc": "Partly cloudy", "WeatherImgUrl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_2.png", "Weathercode": "b", "Weatherdesc": "Partly cloudy", "WeatherImgurl": "http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_2.png", "MaxTemp": 6, "MinTemp": -2, "Freeze": 3, "RainFrom": 0, "RainTo": 0, "Part1": 0, "Part2": 0, "Part3": 0, "Part4": 0, "Thunderstorm": 3, "SymbolId": 31, "Reliability": null }

I didn't analysed the documentation above in detail, but could someone of you check, if the fields we are getting for the single municipalities is more ore less the same? fyi @RudiThoeni

rcavaliere commented 7 months ago

@niettadesantis there are similarities, but as said the main information is numeric. I don't know if we want to foresee also a (minimum) integration in our tourism part of the Open Data Hub, this would be somehow a replication... on the other hand it could be an interesting option for who is using the weather information from the tourism API. @RudiThoeni what do you think?

niettadesantis commented 7 months ago

Ok, thank you @rcavaliere. In fact we didn't talked about the needs, I have to discuss it internally. For the moment it's just an idea of me. But thank you, @RudiThoeni if you can briefly give your opinion on this.

rcavaliere commented 6 months ago

@dulvui I have finalized the tests of the Data Collector. Now is everything OK, super! We can put the Data Collector in production

dulvui commented 6 months ago

@rcavaliere Great! Data is now available in production.

WeatherForecast Stations https://mobility.api.opendatahub.com/v2/flat,node/WeatherForecast?where=sactive.eq.true Data https://mobility.api.opendatahub.com/v2/flat,node/WeatherForecast/*/latest?where=sactive.eq.true

WeatherForecastService Stations https://mobility.api.opendatahub.com/v2/flat,node/WeatherForecastService?where=sactive.eq.true Data https://mobility.api.opendatahub.com/v2/flat,node/WeatherForecastService/*/latest?where=sactive.eq.true

noi-techpark / bdp-commons

As STA I would like to integrate the new municipal weather forecasts of the Province of Bolzano, so to use them for the new infotainment system on the local trains #662