ACCESS-NRI / access-nri-intake-catalog

Tools and configuration info used to manage ACCESS-NRI's intake catalogue
https://access-nri-intake-catalog.rtfd.io
Apache License 2.0
8 stars 1 forks source link

[BUG] Stripping dates to produce file_id produces broken datasets #87

Closed dougiesquire closed 1 year ago

dougiesquire commented 1 year ago

Describe the bug

I'd say this is really a data issue, but it's one the catalog is probably just expected to deal with. The ACCESS-OM2 01deg_jra55v13_ryf9091 product includes files with the following names, e.g.:

Stripping the time info to produce the file_id results in both sets of files being included in the same dataset. However, in this case the ocean_daily_3d_temp.nc files are on a different grid than the ocean_daily_3d_temp_01.nc files.

This type of issue is potentially also occurring in other (probably COSIMA) data products also.

It's still not an ideal solution, but it would be safer to replace dates with something (e.g. XX in this case) instead of stripping them.