Testing with MDTF timeslice run

[Wen-hao-Dong]

I was trying to build the data catalog for MDTF_timeslice_run (/archive/wnd/MDTF_timeslice_run) using the command: _fre catalog buildcatalog -i /archive/wnd/MDTF_timeslice_run -o MDTF_timeslice_runcatalog but got the following error message.

Please note there are some differences between the MDTF_timeslice_run and the default say AM5 simulation: (1) the unusual name of the subfolder (like atmos_inst, ocean_1x1deg, etc.) under /pp (2) the different length of the output in each subfolder as this simulation was done in two steps: We restarted it to modify the diag table after year 2 as we only need some high-temporal data for the first 2 years.

Error message:

wnd:/home/Wenhao.Dong> fre catalog buildcatalog -i /archive/wnd/MDTF_timeslice_run -o MDTF_timeslice_run_catalog Traceback (most recent call last): File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/intakebuilder/getinfo.py", line 134, in getInfoFromGFDLDRS dictInfo[builderconfig.output_path_template[i]] = stemdir[(j)]

IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fms/local/opt/fre-commands/canopy/bin/fre", line 10, in <module>
    sys.exit(fre())
             ^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/fre/fre.py", line 427, in buildCatalog
    context.forward(frecatalog.frecatalog.buildCatalog)
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/core.py", line 804, in forward
    return __self.invoke(__cmd, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/fre/frecatalog/frecatalog.py", line 38, in buildCatalog
    context.forward(build)
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/core.py", line 804, in forward
    return __self.invoke(__cmd, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/fre/frecatalog/get_intake_gfdl.py", line 66, in build
    list_files = gfdlcrawler.crawlLocal(project_dir, dictFilter, dictFilterIgnore,logger)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/intakebuilder/gfdlcrawler.py", line 40, in crawlLocal
    dictInfo = getinfo.getInfoFromGFDLDRS(dirpath, projectdir, dictInfo)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/fms/conda/envs/fre-cli/lib/python3.12/site-packages/intakebuilder/getinfo.py", line 136, in getInfoFromGFDLDRS
    sys.exit("oops in getInfoFromGFDLDRS"+str(i)+str(j)+builderconfig.output_path_template[i]+stemdir[j])
                                                                                              ~~~~~~~^^^
IndexError: list index out of range

[Ciheim]

The fre-cli catalog tool used here doesn't yet include true configuration functionality. Because of this, the tool attempted to build the catalog with the default configuration. Easy configuration functionality is one of our top priorities and will be implemented it in the coming weeks. We also want to avoid confusing error messages like the one seen here so we'll be updating the tool with much cleaner error reporting. (#118)

For the time being, users should clone this repository and configure the builder by editing intakebuilder/builderconfig.py. For this case, a catalog can be generated by changing _output_pathtemplate to ['NA','NA','experiment_id','modeling_realm','cell_methods','frequency','chunk_freq'].

The builder can instead be called by running scripts/gen_intake_gfdl.py /archive/wnd/MDTF_timeslice_run MDTF_timeslice_run_catalog