EUREC4A-UK / lagtraj

Python trajectory code for Lagrangian simulations
MIT License
11 stars 9 forks source link

Handle out-of-order ERA5 time coord issue #189

Closed leifdenby closed 1 year ago

leifdenby commented 1 year ago

It appears that ECMWF's CDS has started returning data where the time-coordinate isn't monotonic anymore and the timesteps are randomly ordered. This commit adds sorting by time during load-time. This may slow down calculations and a fix where input files are overwritten could be made in future

Related to https://github.com/EUREC4A-UK/lagtraj/issues/183

leifdenby commented 1 year ago

@sjboeing I've manually verified that this works. We can't test it with our test-data because we don't download data from ECMWF during testing. Could you give it a try too and let me know what you think? Thanks :)

sjboeing commented 1 year ago

Hi Leif,

This looks good and it seems to work for me. While working on this, I did run into an issue with domain versioning: "Exception: The domain data in /home/stevenboeing/lagtraj_0922/lagtraj/data/domains/eurec4a_circle_data doesn't match the version stored in the data definition for lagtraj://eurec4a_circle (unversioned != 1.0.0). Please delete the domain data and re-download." To make it work, I removed the version from: lagtraj/input_definitions/examples_files/domains/eurec4a_circle.yaml Let me know if this sounds familiar, if not I will make an issue for us to investigate.

leifdenby commented 1 year ago

"Exception: The domain data in /home/stevenboeing/lagtraj_0922/lagtraj/data/domains/eurec4a_circle_data doesn't match the version stored in the data definition for lagtraj://eurec4a_circle (unversioned != 1.0.0). Please delete the domain data and re-download." To make it work, I removed the version from: lagtraj/input_definitions/examples_files/domains/eurec4a_circle.yaml

Hmm.. I think this means you downloaded this domain data before we added the versioning (https://github.com/EUREC4A-UK/lagtraj/pull/87). But this is fine :) Another way to solve it is just create a filename called VERSION in domains/eurec4a_circle_data which just contains 1.0.0

leifdenby commented 1 year ago

I'll merge this in once the tests pass

@sjboeing with this change the changelog is getting quite long: https://github.com/leifdenby/lagtraj/blob/fix-era5-time-monotinicity-issue/CHANGELOG.md - how do you feel about me creating release v0.1.2 once this pull-request has been merged?

sjboeing commented 1 year ago

Ah, thanks Leif. merging this and creating a new release sounds good.

The other issue occurred with data downloaded just this week. I will check and see if it is something I missed and create an issue if it persists.