Add region: France - Githubissues

vbelloir commented 3 years ago

Hi, fridl, Thanks for your app!

All data for France, are free, and under ETALAB licence. They can be download on pdf or xml formats here : https://donneespubliques.meteofrance.fr/?fond=produit&id_produit=265&id_rubrique=50

Here is a example file for Aravis Area:

BRA.ARAVIS.20210117143853.txt

Can I do something more to help?

Vincent

fridlmue commented 3 years ago

Thank you for the Link! Do you know if France also issues reports in any other Language? I have to Investigate that, but I won't have much time the next week. The things to Figure out are (if you have time, you are very welcome to participate of course, otherwise I use it as my list of ToDo'S):

[x] Find the URL to query the XML for a region directly (without pressing 'Télécharger')
[ ] Extract a List with all the Regions and Sub-Regions (Massif -> Sub-Region or Département -> Sub-Region - I don't know what is better...)
[ ] Does the French Report fit the Structure of how we currently display the other "normal" reports or the Swiss structure or is another, third structure needed?
[ ] If it fits a Structure we need a Mapping of the XML-Datapoints to the Report-Structure.

Then on the Programming side it should be:

[ ] Writing a python xml parser for the french format
[ ] If needed, Implement a DangerPage for France.

And not to forget:

[ ] French Translation of the App, which includes:
[ ] Issue French Reports where Available (Swiss, ALBINA, Val'd Aran)

vbelloir commented 3 years ago

No, I don't find the export in other language, and when I saw that, i was thinking that it will be an issue ^^

Evne xml tags are in french...

I will see if i can give you some help on tasks you listed.

Vincent

fridlmue commented 3 years ago

As some key Information are part of the visual report representation, the Language should be ok in French (for sure more would be better, but we take what we get :wink: ) Most Important then is to have the Main Level, the dangerous Expositions and the Avalanche Problems as visuals. Most important the Avalanche Problems in a machine readable format, what we don't have in the Swiss regions at the moment. (Compare https://avalanche.report/education/avalanche-problems / https://fr.avalanche.report/education/avalanche-problems)

vbelloir commented 3 years ago

hum, It seems that the web page only list archive, and not the last report.

Last report on donneespubliques.meteofrance.fr website is from the 18th January 4:00 PM, while on meteofrance website, the bulletin is dated to 19th january 4:00PM.

I think that meteofrance keep last bulletin for its usage, and only archive are public.

for example, look at the report for Oisans area:

donneespubliques.meteofrance.fr BRA.OISANS.20210118144632.pdf
meteofrance.fr RisqueAvalanches- OISANS.pdf

fridlmue commented 3 years ago

I wrote on 27-12-2020 to Meteo France with a request about requesting the reports, but never got an answer. I have not been aware of the link you posted back then as my french is not good enough for googling around :wink: . The Mail was translated by a friend.

...
Dans le cadre du développement d'une application sous licence libre (logiciel 
libre, GPL3) pour les systèmes d'exploitation mobiles autres qu'Android et 
iOS, je souhaite également intégrer et afficher les rapports de situation 
d'avalanche français.
Les régions d'Autriche et d'Allemagne sont déjà couvertes dans l'application. 
Certaines régions italiennes également. La Slovénie et certaines régions 
espagnoles sont en cours de planification ou de mise en œuvre. Je serais 
heureux de couvrir autant de régions "EAWS" que possible. Sans les régions 
françaises déclarantes, la couverture est bien sûr loin d'être complète.
C'est pourquoi je voudrais savoir si les rapports de situation d'avalanche en 
France peuvent être interrogés via une API (par exemple en tant que CAAML-XML 
ou via JSON). Si cela est possible, j'aimerais les traiter et les afficher 
dans l'application, en indiquant la source bien sûr. 

Je serais très heureux de recevoir un bref retour d'information sur cette 
question.
...

vbelloir commented 3 years ago

Ok, I will continue to search for updated xml.

In parallel, I will also contact MeteoFrance. Can I point them to github repo?

fridlmue commented 3 years ago

Sure, absolutely no Proble!

vbelloir commented 3 years ago

Done, with the contact form of https://donneespubliques.meteofrance.fr/

I found a very interrest website (in French), that explains how to get xml files : https://blog.desgran.ge/post/opendata-et-donnees-neiges/

First, all files are available on a ftp : ftp://ftp.meteo.fr/FDPMSP/Pdf/BRA

each day, there is a new json file that list all new xml files, for each massif.
The json file is name like this : bra.%Y%m%d.json, and is available at this url: https://donneespubliques.meteofrance.fr/donnees_libres/Pdf/BRA/bra.%Y%m%d.json

For example, for today this is https://donneespubliques.meteofrance.fr/donnees_libres/Pdf/BRA/bra.20210119.json

With that file, we can now that for Oisans massif, {"massif": "OISANS", "heures": ["20210119144521"]} the bra is bra.oisans.20210119144521.xml available at https://donneespubliques.meteofrance.fr/donnees_libres/Pdf/BRA/BRA.OISANS.20210119144521.xml

EDIT : main issue is still that this report is not up to date, and expire at today 18:00 (6:00PM).

fridlmue commented 3 years ago

Lets check out, if we get a "valid" report tomorrow in the morning, then it wouldn't look so bad we would getting somehow a useful report. Very gut recherche so far, thanks for keeping me posted with all these findings!

vbelloir commented 3 years ago

Nothing more this morning 8:00 :(

fridlmue commented 3 years ago

Another way to determine a reports source is to check out where the official App requests its reports with a network monitoring tool or 'inspect' the apk.

vbelloir commented 3 years ago

I had yesrterday first responses for MeteoFrance services. They told me that live data are only avaliable from meteofrancewebsite, not donneespubliques. We already know that.

When I asked them if there is API on metefrance website, they give mu another email address, quite generic unfortunately. No response right now. On monday, I will ask again.

vbelloir commented 3 years ago

I had a final response today, form digital commercial service.
Report of the day can be accessed from an API or in xml format. I think that it exists, but reserved for some public services.

That's the end.

fridlmue commented 3 years ago

That's indeed sad news. I'll keep the Issue open for further news on the topic if things change. Or if someone wants to decompose the needed information form a App or Website... But, nevertheless: Thanks for Investigating!

fridlmue commented 3 years ago

@vbelloir Good news! I probably go a hint how to get the FR bulletins :stuck_out_tongue_winking_eye: Now i need some advice for the mapping of the content.

Attached you can find here a example bulletin. I removed the embedded Images. report_id1_example.txt

I'm able to extract up to two risk levels with elevations and the change over the day for any of these two (I'll present that as AM and PM report in AvaRisk). (line 6)
The Expositions are clear. I'll present that like in the Swiss report as "Dangerous Exposition"
Is there any common use of these Problems in the Swiss reports? I can't find any reference. https://www.avalanches.org/standards/avalanche-problems/
Can I use the content of (line 10) as the Headline for the Danger Report and (line 19) as Danger Report?
Can I use the content of (line 28) as Snowpack (and Weather) description?

If I store and present all of that, is there then any very important information for the FR-Report missing?

fridlmue commented 3 years ago

And these are the Region_ID's i've found so far (i think they are all). Would you order them differently?

  property ListModel regionListFrAlpesDuNord: ListModel{
      //Haute Savoie
      ListElement {region: qsTr("CHABLAIS");                                     RegionID: "FR-01"}
      ListElement {region: qsTr("ARAVIS");                                       RegionID: "FR-02"}
      ListElement {region: qsTr("MONT-BLANC");                                   RegionID: "FR-03"}
      //Savoie
      ListElement {region: qsTr("BAUGES");                                       RegionID: "FR-04"}
      ListElement {region: qsTr("BEAUFORTAIN");                                  RegionID: "FR-05"}
      ListElement {region: qsTr("HAUTE-TARENTAISE");                             RegionID: "FR-06"}
      ListElement {region: qsTr("MAURIENNE");                                    RegionID: "FR-09"}
      ListElement {region: qsTr("VANOISE");                                      RegionID: "FR-10"}
      ListElement {region: qsTr("HAUTE-MAURIENNE");                              RegionID: "FR-11"}
      //Isere                                                                    
      ListElement {region: qsTr("CHARTREUSE");                                   RegionID: "FR-07"}
      ListElement {region: qsTr("BELLEDONNE");                                   RegionID: "FR-08"}
      ListElement {region: qsTr("GRANDES-ROUSSES");                              RegionID: "FR-12"}
      ListElement {region: qsTr("VERCORS");                                      RegionID: "FR-14"}
      ListElement {region: qsTr("OISANS");                                       RegionID: "FR-15"}
  }

  property ListModel regionListFrAlpesDuSud: ListModel{
      //Hautes Alpes
      ListElement {region: qsTr("THABOR");                                       RegionID: "FR-13"}
      ListElement {region: qsTr("PELVOUX");                                      RegionID: "FR-16"}
      ListElement {region: qsTr("QUEYRAS");                                      RegionID: "FR-17"}
      ListElement {region: qsTr("DEVOLUY");                                      RegionID: "FR-18"}
      ListElement {region: qsTr("CHAMPSAUR");                                    RegionID: "FR-19"}
      ListElement {region: qsTr("EMBRUNAIS-PARPAILLON");                         RegionID: "FR-20"}
      ListElement {region: qsTr("UBAYE");                                        RegionID: "FR-21"}
      //Alpes Maritimes
      ListElement {region: qsTr("HAUT-VAR/HAUT-VERDON");                         RegionID: "FR-22"}
      ListElement {region: qsTr("MERCANTOUR");                                   RegionID: "FR-23"}
  }

  property ListModel regionListFrPyrenees: ListModel{
      //Pyrenees Atlantique
      ListElement {region: qsTr("PAYS-BASQUE");                                  RegionID: "FR-64"}
      ListElement {region: qsTr("ASPE-OSSAU");                                   RegionID: "FR-65"}
      //Hautes Pyrenees                                                      
      ListElement {region: qsTr("HAUTE-BIGORRE");                                RegionID: "FR-66"}
      ListElement {region: qsTr("AURE-LOURON");                                  RegionID: "FR-67"}
      //Haute Garonne                                                        
      ListElement {region: qsTr("LUCHONNAIS");                                   RegionID: "FR-68"}
      ListElement {region: qsTr("COUSERANS");                                    RegionID: "FR-69"}
      //Ariege                                                               
      ListElement {region: qsTr("HAUTE-ARIEGE");                                 RegionID: "FR-70"}
      ListElement {region: qsTr("ORLU  ST BARTHELEMY");                          RegionID: "FR-72"}
      //Andorre                                                              
      ListElement {region: qsTr("ANDORRE");                                      RegionID: "FR-71"}
      //Pyrenees Orientales
      ListElement {region: qsTr("CAPCIR-PUYMORENS");                             RegionID: "FR-73"}
      ListElement {region: qsTr("CERDAGNE-CANIGOU");                             RegionID: "FR-74"}
  }
    property ListModel regionListFrCorse: ListModel{
        ListElement {region: qsTr("CINTO-ROTONDO");                                RegionID: "FR-40"}
        ListElement {region: qsTr("RENOSO-INCUDINE");                              RegionID: "FR-41"}
    }

vbelloir commented 3 years ago

Hi @fridlmue It's a really good news!

I have never heard of picture representing Avalanche Problems. I think that there is no equivalent in french reports.

Yes, you are right, line 10 is the summary and line 19 is the detailed description of risk.

Line 28 describes weather and snow quality, so it can be "Snow and Weather Data" in swiss report, or "Snowpack description" in Italian ones.

With that, I think that most important info will be displayed!

Concerning region ID, this is the order used by meteoFrance, so, I would keep it!

fridlmue commented 3 years ago

@vbelloir You can find a first draft with localization and regions here: https://github.com/fridlmue/harbour-avarisk/releases/tag/V0.8-3

You can just play around with it a little. Some thinks need adjustments, like the string for "Treeline" is very long in :fr: (to long fir the DangerPage and the Cover at the moment, so we need to adjust the string or the layout). But sure there are more issues. Please report your findings. For more general things you can open also a new Issue.

As I told, my priority is to "harden" things at the pyAvaCore at the moment and then optimize the representation in the app. But it is very nice to see growing parts of the Alpine Area covered by regions AND localization :+1:

fridlmue commented 3 years ago

I forgot to mention: The App should now request the Swiss, Liechtenstein, Tirol, South Tirol, Trentino and Val d'Aran bulletins in French as well.

vbelloir commented 3 years ago

Great job!! I tested the rpm file. I have some string do correct, but nothing important. I will do that during the week.

fridlmue commented 3 years ago

Yeah, take your time. It would be great if you could also do sometimes a small checks on consistency with the official reports from meteo france. I do that at times, but 4 eyes see more than 2 :wink:

vbelloir commented 3 years ago

Ok, I will do.

I have a question: Some pictograms are under different names for France and for Swiss. is it normal? I'm talking of Avalanche Problem for France and Avalanche Prone Locations for swiss reports.

fridlmue commented 3 years ago

I know what you mean. Yeah, I need to figure out a good way to unify that.

The reason behind is: I try to structure the reports in a common format internally. But there are some variations in the reports. As the Swiss one was a very big deviation I gave it an individual page layout in the first run. While the (yet implemented) Austrian, German and Italian reports issue multiple Prone Locations and Elevations linked to different Avalanche Problems and, if necessary, two danger levels with a upper and lower range limit, in contrast the Swiss reports has one danger level with a general prone location (with other compass card) and only textual descriptions for the problems. The french report is somewhere in the middle with a general prone exposition not linked to problems but with up to two danger levels with a range limit and not the swiss compass. So it fitted the "not Swiss" layout better in the first run.

My long term target is to have a more adaptive DangerPage.qml and not a extra one for switzerland. But the first step would be to remove the Avalanche Problem section (and the Tendency Section) if the report is French and plot the compass next to the Elevation Data in this case. Ideas or concepts or a opinion from your side are very welcome! :wink:

fridlmue / harbour-avarisk

Add region: France #23