search

NOAA-GFDL / CatalogBuilder

Toolset to build, validate, modify, and use intake-esm based data catalogs
3 stars 4 forks source link

ERA5 catalogs -- filename accommodation #80

Open meteorologist15 opened 1 month ago

meteorologist15 commented 1 month ago

Adjusts crawler script to scan for files not containing a "." character within the name if the file comes from the UDA.

aradhakrishnanGFDL commented 1 month ago

@meteorologist15 recap of what we discussed: We can use activity_id = "UDA" here and the csv will have this info then. For future, we can also consider using this project being passed around in the code for handling certain paths differently: https://github.com/NOAA-GFDL/CatalogBuilder/blob/0f18e03ebc789f779d536f321a3949035d3795fd/catalogbuilder/intakebuilder/getinfo.py#L24

aradhakrishnanGFDL commented 1 month ago

@meteorologist15 recap of what we discussed: We can use activity_id = "UDA" here and the csv will have this info then. For future, we can also consider using this project being passed around in the code for handling certain paths differently:

https://github.com/NOAA-GFDL/CatalogBuilder/blob/0f18e03ebc789f779d536f321a3949035d3795fd/catalogbuilder/intakebuilder/getinfo.py#L24

uda string could be grabbed from the input_path

aradhakrishnanGFDL commented 1 month ago

@meteorologist15 documenting minutes for reference

using project as uda for the special case maneuvering in the code sounds great. For the sprint, focus on cloud variables for Ray's analysis. Ensure frequency and time is captured from the DRS or filename. You may have to tweak your existing configuration. See examples in configs or tests in the repository.

variable_id: lets revisit this after tomorrow. It's possible that you want to capture the descriptive names in the filename as standard_name in the catalog. And, then come back to the mapping route or the netcdf look up route.

If it's the descriptive names that's captured, we have to tweak Ray's analysis a bit, but should be do-able. Keeping all options open!