ACCESS-NRI / access-nri-intake-catalog

Tools and configuration info used to manage ACCESS-NRI's intake catalogue
https://access-nri-intake-catalog.rtfd.io
Apache License 2.0
8 stars 1 forks source link

[BUG] Build function takes min/max version from live directory always, instead of build directory #263

Closed marc-white closed 1 day ago

marc-white commented 3 days ago

Describe the bug

While doing testing for #175 , I've discovered that the build function is computing the minimum (and presumably the maximum) catalog version from the live catalog location on g/data, even when the build is taking place somewhere else. The system should be looking for min and max catalog versions in the build directory.

To Reproduce

Running a test build of #175 with MOM6 sources only into a blank test directory on Gadi /scratch yields the following catalog.yaml:

sources:
  access_nri:
    args:
      columns_with_iterables:
      - model
      - realm
      - frequency
      - variable
      mode: r
      name_column: name
      path: /scratch/tm70/mcw120/intake-cat-test/{{version}}/metacatalog.csv
      yaml_column: yaml
    description: ACCESS-NRI intake catalog
    driver: intake_dataframe_catalog.core.DfFileCatalog
    metadata:
      storage: gdata/None+gdata/ik11
      version: '{{version}}'
    parameters:
      version:
        default: v2024-11-12
        description: Catalog version
        max: v2024-11-12
        min: v2023-07-10   # HERE - This should be equal to max and default, as there were no other existing catalogs here
        type: str

The min value returned corresponds to the current minimum value present in /g/data/xp65/public/apps/access-nri-intake-catalog (i.e., the live catalog directory).

Additional context

Doesn't block #175, but should be fixed ASAP. Should also include a test to prevent this from happening again.