search

openradar / xradar

A tool to work in weather radar data in xarray
https://docs.openradarscience.org/projects/xradar
MIT License
104 stars 17 forks source link

implement ODIM BUFR readers #101

Open jfigui opened 1 year ago

jfigui commented 1 year ago

At the last OPERA ET meeting it emerged that there is a real interest for xradar to have and ODIM BUFR reader, in particular to read the French radar data. I am fully aware that that is not an easy task but considering that there are plans to open the OPERA data archive and that, particularly at the early stages of the OPERA program, many files are in BUFR format this would be very welcomed by the European radar community.

kmuehlbauer commented 1 year ago

@jfigui Thanks for bringing this to attention.

wradlib used to read BUFR back in the old days, but dropped support because of all kind of issues. The bufr related python ecosystem evolved much since then, so this might not be an issue any more.

If you can point to some resources on how to read french bufr data that would be great. Also docs on how to handle tables etc. would be nice to have.

jfigui commented 1 year ago

@kmuehlbauer ,

you probably know that but there are resources linked to OPERA Bufr here: https://www.eumetnet.eu/activities/observations-programme/current-activities/opera/

In particular: https://www.eumetnet.eu/wp-content/uploads/2017/01/OPERA_bufr_description_2012.pdf

https://www.eumetnet.eu/wp-content/uploads/2017/03/bufr_3.2.zip

I can probably send you some examples of polar volume bufr files as well if needed.

kmuehlbauer commented 1 year ago

Yeah, but good it's now linked here. I've thought there would be some low hanging fruit with new packages like pdbufr or even directly with eccodes. The code on that eumetnet-page is rather old, unfortunately.

It would be great if you could provide some bufr polar volumes. Do they need special tables or can they be read with the standard tables?

jfigui commented 1 year ago

I think French data need special tables. What I positively know is that eccodes cannot open the polar volume bufr data directly (I tried). I have always tried to stay away from BUFR files so I am definitely not an expert but I will ask for support at MF.

jfigui commented 1 year ago

I attach a copy of the tables that are used at Météo-France tables_bufr_opera.zip

jfigui commented 1 year ago

and here there is an example of so called PAM file:

jfigui commented 1 year ago

here there is an example of so called PAG file, these files are sent to the OPERA data center:

jfigui commented 1 year ago

One thing to keep in mind is that as far as I understood those files are not a single bufr file with multiple fields but a concatenation of bufr files. To be able to decode them properly you have to first cut the file and then read the bufr. Another problematic issue is that the field data is compressed using z lib so even when you manage to decode the bufr you still have to decompress the data.

kmuehlbauer commented 1 year ago

I feel like I'm remembering now, why we've dropped bufr in wradlib :grimacing:

jfigui commented 1 year ago

I can fully understand your decision! But if we could give it another try ... :)

kmuehlbauer commented 1 year ago

If at first you don't succeed...

mgrover1 commented 1 year ago

I am wondering if this might be of use here https://github.com/pytroll/trollbufr

jfigui commented 1 year ago

Last time I checked trollbufr was not able to decode the Météo France BUFR. Among other things it was not prepared to read BUFR version 2 or 3 but it can be starting point.

kmuehlbauer commented 1 year ago

There's also pybufrkit, which might be worth a test.

rzambranap commented 1 year ago

Hey everyone, have we had any success in this field ? I recently received a load of bufr radar files from météofrance, and I'm having little to no success in reading them with pybufrkit or even the opera bufr linux software listed in one of the comments up there. For me it is quite important and necessary to read these files and I use a python environment, if someone could give me any clues of how to move forward, I'll gladly be implementing a reader to use in an xarray environment. If someone advanced on this we could work together. I've attached a data sample and the tables they sent me. Thanks in advance!

202304090000.gz tables_bufr_281.zip

kmuehlbauer commented 1 year ago

No, there haven't been any advances here. The major pain point is, and this is a show stopper, that the provided tables are in some ".csv" format. eccodes does not read these tables per default and they will have to be converted for that purpose to the eccodes table format.

Please ask your table provider to provide the tables in ECMWF eccodes format.

Francesco-Uboldi commented 11 months ago

I am having a similar problem with some sample BUFR files I had from Météo-France. See here

kmuehlbauer commented 11 months ago

@Francesco-Uboldi Are your MF tables the same as linked above by @jfigui and @rzambranap?

And would you join a team to implement an eccodes based BUFR reader for xradar?

Francesco-Uboldi commented 11 months ago

I have version 279 instead of 281... I will try the 281 from @rzambranap . I can try and help, but I am quite new to both radar data in general and BUFR format. In particular, I never used eccodes: I met this software recently, of course... but the table problem stopped me, and I went towards OPERA instead. And I have limited knowledge of python! And of C. I come from Fortran and R...
By the way... wouldn't it be nice if some BUFR user from Météo-France agreed to be involved?!? Anyone out there?

kmuehlbauer commented 11 months ago

@Francesco-Uboldi As I see the situation it all depends on the tables. We would need those tables in the format accepted by eccodes. Or there would need to be some transformation tool to convert the tables. It's a pity that BUFR data is so unaccessible.

rzambranap commented 11 months ago

Hey everyone, I wasn't able to implement something on Python, but I was able to sort it out via Python using the subprocess library, calling the installed BUFR software locally. Two caveats: I couldn't launch it on VMs, and there was something about 32 vs 64-bit architecture that somehow worked on my PC but not on my office's servers. I can send you the tables Météo-France sent to me and the functions I did in Python calling subprocess and everything, but it's not up to the standards you might be accustomed to...

On Wed, Dec 13, 2023 at 10:06 AM Kai Mühlbauer @.***> wrote:

@Francesco-Uboldi https://github.com/Francesco-Uboldi As I see the situation it all depends on the tables. We would need those tables in the format accepted by eccodes. Or there would need to be some transformation tool to convert the tables. It's a pity that BUFR data is so unaccessible.

— Reply to this email directly, view it on GitHub https://github.com/openradar/xradar/issues/101#issuecomment-1853521869, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO6FHGG2GQ5OLQG2IZX2B3LYJFVYTAVCNFSM6AAAAAAWDTIYCCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJTGUZDCOBWHE . You are receiving this because you were mentioned.Message ID: @.***>

-- Rodrigo Zambrana