search

COSIMA / cosima-cookbook

Framework for indexing and querying ocean-sea ice model output.
https://cosima-recipes.readthedocs.io/en/latest/
Apache License 2.0
57 stars 27 forks source link

Easterlies perturbation data moved #332

Open aidanheerdegen opened 8 months ago

aidanheerdegen commented 8 months ago

The easterlies perturbation experiments 01deg_jra55v13_ryf9091_easterlies_down10 and 01deg_jra55v13_ryf9091_easterlies_up10 both seem to have been moved from ik11 to cj50, but the DB still has entries pointing to the ik11 location.

This leads to errors like

FileNotFoundError: [Errno 2] No such file or directory: '/g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091_easterlies_down10/output1047/ocean/ocean_scalar.nc'
access-hive-bot commented 8 months ago

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/unable-to-access-access-om2-01-future-perturbation-outputs/1473/11

adele-morrison commented 8 months ago

Should the database indexing pick up changes like this? And if so, do we know why it isn't?

aidanheerdegen commented 8 months ago

Should the database indexing pick up changes like this? And if so, do we know why it isn't?

Yes and yes.

The reason it doesn't is that it doesn't have any logic to detect this condition. It relies on the directory being present and then it sees the files have been deleted and removes them from the DB.

It is perfectly possible to add the logic required to detect this condition.

In an ideal world there would be a script to move data from one location to another. The script could do the move and make the required changes to the database, so the data doesn't have to be re-indexed.

Of course this would still require people to use it.