Ouranosinc / raven

WPS services related to hydrological modeling
https://pavics-raven.readthedocs.io
MIT License
37 stars 12 forks source link

Use smaller testdata for Raven tutorial notebook so can enable Raven notebooks on Jenkins #383

Closed tlvu closed 2 years ago

tlvu commented 3 years ago

The goal is to enable Raven tutorial notebooks on Jenkins, just like Finch and Pavics-sdi notebooks currently.

This means Raven notebooks should pass on all test deployments of PAVICS, not just on prod deployment. So we need to replicate testdata to all the test deployments.

This notebook https://github.com/Ouranosinc/raven/blob/3c768f167fbb7515206b63b90020e57708ebbc4d/docs/source/notebooks/Full_process_example_1.ipynb use two gigantic testdata file (ERA5_tas = "https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/ecmwf/era5_rechunk/tas_era5_reanalysis_hourly_2020.nc", ERA5_pr = "https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/ecmwf/era5_rechunk/pr_era5_reanalysis_hourly_2020.nc") that are 17G each !

Either:

matprov commented 3 years ago

@tlvu Just by curiosity, any update on this one? This would allow us to run the e2e raven tests, which are required for all the PRs regarding raven.

tlvu commented 3 years ago

@tlvu Just by curiosity, any update on this one? This would allow us to run the e2e raven tests, which are required for all the PRs regarding raven.

@huard ? I have to defer to you as I do not understand the notebook enough to be able to make the call.

huard commented 3 years ago

I made this choice initially because the performance with the NcML was terrible. It just hanged there forever. Since the NcML is now using rechunked date, we should check if performance is acceptable.

richardarsenault commented 2 years ago

Smaller files have been provided and the Full_process notebook has been deprecated.