search

aradhakrishnanGFDL / CatalogBuilder

CatalogBuilder for data discovery and analysis
3 stars 4 forks source link

116.config yaml #125

Closed Ciheim closed 1 month ago

Ciheim commented 1 month ago

TESTING:

  1. Module load conda
  2. Conda env create -f intakebuilder/environment.yml (jsondiff package added)
  3. Conda activate catalogbuilder
  4. cp configs/config-template.yaml config/config.yaml
  5. Leave headerlist, OPT, and OFT, the same. Set input path as '/archive/am5/am5/am5f3b1r0/c96L65_am5f3b1r0_pdclim1850F/gfdl.ncrc5-deploy-prod-openmp/pp' and set output path as 'catalog' (note: '~' doesn't work here)
  6. Run scripts/gen_intake_gfdl.py --config configs/config.yaml
  7. Run scripts/gen_intake_gfdl.py /archive/am5/am5/am5f3b1r0/c96L65_am5f3b1r0_pdclim1850F/gfdl.ncrc5-deploy-prod-openmp/pp catalog2
  8. Run diff catalog catalog2
aradhakrishnanGFDL commented 1 month ago

If I comment out output_path_template and output_file_template in builderconfig.py, errors are thrown.

scripts/gen_intake_gfdl.py --config config.yaml The module intakebuilder is not installed. Do you have intakebuilder in your sys.path or have you activated the conda environment with the intakebuilder package in it? Attempting again with adjusted sys.path input-path None config.yaml /archive/Chris.Blanton/am5/am5f7b11r0/c96L65_am5f7b11r0_amip/gfdl.ncrc5-deploy-prod-openmp/pp catalog Traceback (most recent call last): File "scripts/gen_intake_gfdl.py", line 102, in main() File "/nbhome/Aparna.Radhakrishnan/conda/envs/intake/lib/python3.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/nbhome/Aparna.Radhakrishnan/conda/envs/intake/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/nbhome/Aparna.Radhakrishnan/conda/envs/intake/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/nbhome/Aparna.Radhakrishnan/conda/envs/intake/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "scripts/gen_intake_gfdl.py", line 81, in main list_files = gfdlcrawler.crawlLocal(project_dir, dictFilter, dictFilterIgnore, logger) File "/nbhome/a1r/CatalogBuilder/intakebuilder/gfdlcrawler.py", line 40, in crawlLocal dictInfo = getinfo.getInfoFromGFDLDRS(dirpath, projectdir, dictInfo) File "/nbhome/a1r/CatalogBuilder/intakebuilder/getinfo.py", line 132, in getInfoFromGFDLDRS nlen = len(builderconfig.output_path_template) AttributeError: module 'intakebuilder.builderconfig' has no attribute 'output_path_template'