search

ECMWFCode4Earth / challenges_2021

Discover the ECMWF Summer of Weather Code 2021 challenges!
47 stars 13 forks source link

Challenge #31 -ECMWF forecast data in 3D with Blender #4

Open EsperanzaCuartero opened 3 years ago

EsperanzaCuartero commented 3 years ago

Challenge 31- ECMWF forecast data in 3D with Blender

Stream 3 - Visualize weather, climate and atmosphere

Goal

Extend existing visualisation software to support ECMWF forecast data + examples

Mentors and skills

Challenge description

For creating high quality visualisations of forecast data, Blender has proven a valuable tool. There has been various attempts to load GRIB data but they have not been sustainable and scalable. A new project started called BlenderNC to offer a Python add-on using xarray to load NetCDF data. This project should use the xarray extension cfgrib to extend BlenderNC to be able to read and visualise GRIB data. It is also expected that the documentation at https://blendernc.readthedocs.io will be updated with examples of using GRIB data. Interested candidates have a chance to work on an exciting Python project and learn how to handle weather model data and how to visualise it.


ESoWC

whatnick commented 3 years ago

I would be interested in this challenge. I have the relevant experience in Python, Xarray and Blender. I work regularly with NetCDF and more recently with GRIB files for forcing WRF models.

EsperanzaCuartero commented 3 years ago

Hi Tisham, thanks for your interest. If you have a question, you can start a conversation with the mentors of this challenge and tailor your proposal before submitting it. The deadline for applications is on 16 April. Best, Esperanza

jwagemann commented 3 years ago

Hi, join us for the ECMWF Summer of Weather Code Ask Me Anything session and learn all things ESoWC.

When: Wednesday, 24 March 2021 at 4 pm GMT

What: learn everything about ESoWC - how it works, the challenges this year, some tips for your proposal and listen to ESoWC experiences from previous participants

How: register here.

whatnick commented 3 years ago

Hi @sylvielamythepaut @StephanSiemen , @josuemtzmo and I are putting together a joint proposal for this challenge. Josue is a maintainer for the BlenderNC project and in Canberra with myself. We look forward to working on BlenderNC.

StephanSiemen commented 3 years ago

Hi @whatnick, Thanks for letting us know. Sounds great! Looking forward to your proposal :-)

whatnick commented 3 years ago

@StephanSiemen I currently have this challenge loading the ECMWF forecast data I have using cfgrib. https://github.com/ecmwf/cfgrib/issues/232 , some assistance will be appreciated.

iainrussell commented 3 years ago

Hi @whatnick , I'm writing a comment into the cfgrib issue, but in the meantime can you successfully use the ecCodes tool 'grib_to_netcdf' as described here if you only want to do a file conversion? https://confluence.ecmwf.int/display/ECC/grib_to_netcdf You will need to install ecCodes, perhaps through conda-forge. It might work, but mixing GRIB editions could be problematic, let's see.

josuemtzmo commented 3 years ago

Hello, As far as I know, BlenderNC could support cgrib files by implementing:

 xr.load_dataset('filename.grib', engine='cfgrib')

an xarray example can be found at: http://xarray.pydata.org/en/stable/examples/ERA5-GRIB-example.html

iainrussell commented 3 years ago

Another thought, if cfgrib is complaining because of a mixture of GRIB1 and GRIB2 messages, you can separate them using the filter_by_keys argument described here: https://github.com/ecmwf/cfgrib Untested suggestion: g1 = xr.open_dataset('mydata.grib', engine='cfgrib', backend_kwargs={'filter_by_keys': {'edition': '1'}}) g2 = xr.open_dataset('mydata.grib', engine='cfgrib', backend_kwargs={'filter_by_keys': {'edition': '2'}}) Then you can save them into separate netCDF files if that's useful.

sylvielamythepaut commented 3 years ago

Hi,

I just created a shared forder on Dropbox . I put 1 set of data ( in Grib and NetCDF)

This is a 10 days forecast for 5 parameters: 2t : 2m temperature lcc, mcc, hcc : low, Medium, High Cloud Cover tp : Total precipitation ( I think they are accumulated .. May be we need to do a bit of processing on this one ) That should allow us to start

whatnick commented 3 years ago

I have downloaded and verified that this data can be loaded successfully. What are the license conditions for this data. If this data can be CC licensed we could place it in our Github under LFS and use as test data in CI. I have put my exploratory code in a notebook here. https://gist.github.com/whatnick/0e2821f1abacba8d57c1ebdeea6487d7

More details on GIT LFS are available here. https://git-lfs.github.com/ . If a suitable licence can be obtained then please feel free to make a PR to blenderNC with this data in say a data folder.

EsperanzaCuartero commented 3 years ago

Hi Tisham,

I've already sent an invite for you to create a repository under ESoWC space GitHub. I've used the GitHub handle you provided in the form https://github.com/josuemtzmo

Thanks.

Best Esperanza

From: Tisham Dhar @.> Sent: Saturday, May 29, 2021 8:53 AM To: esowc/challenges_2021 @.> Cc: Esperanza Cuartero @.>; Author @.> Subject: Re: [esowc/challenges_2021] Challenge #31 -ECMWF forecast data in 3D with Blender (#4)

I have downloaded and verified that this data can be loaded successfully. What are the license conditions for this data. If this data can be CC licensed we could place it in our Github under LFS and use as test data in CI. https://gist.github.com/whatnick/0e2821f1abacba8d57c1ebdeea6487d7https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fwhatnick%2F0e2821f1abacba8d57c1ebdeea6487d7&data=04%7C01%7Cesperanza.cuartero%40ecmwf.int%7C02dcd458111c4cabb9f608d9227f4909%7C21b711c6aab74d369ffbac0357bc20ba%7C1%7C0%7C637578752340146347%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=lq%2BLpcZEDXDc6FJ6K8QY2uarjQYzxJltRVokiSfnRrc%3D&reserved=0

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fesowc%2Fchallenges_2021%2Fissues%2F4%23issuecomment-850798597&data=04%7C01%7Cesperanza.cuartero%40ecmwf.int%7C02dcd458111c4cabb9f608d9227f4909%7C21b711c6aab74d369ffbac0357bc20ba%7C1%7C0%7C637578752340146347%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Zz71dmKo7qsWoxbtlY3V7msVD67GOEQ7%2Bp3Ft94A1gE%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALENNJO347Y7HX3AEEFAYDTTQCTSBANCNFSM4WW66S3Q&data=04%7C01%7Cesperanza.cuartero%40ecmwf.int%7C02dcd458111c4cabb9f608d9227f4909%7C21b711c6aab74d369ffbac0357bc20ba%7C1%7C0%7C637578752340156291%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FzGVereRq0ggbjQFUGK2CUBUvTpyv1KY2NNsFYLMbbk%3D&reserved=0.

sylvielamythepaut commented 3 years ago

I do not think there is any issue with the license, but I will check and come back to you.

josuemtzmo commented 3 years ago

Hi Tisham, I've already sent an invite for you to create a repository under ESoWC space GitHub. I've used the GitHub handle you provided in the form https://github.com/josuemtzmo Thanks. Best Esperanza …

Hello Esperanza, I've accepted the invitation, I've created an issue to draft the README file Issue 51.