$USER in OutputPath no longer works

[msulprizio]

Name and Institution (Required)

Name: Melissa Sulprizio Institution: Harvard

Description of your issue or question

The use of $USER in a file path no longer works. In the config files for Cannon (i.e. running on out local cluster) we currently have by default:

## Path for IMI runs and output
OutputPath: "/n/holylfs05/LABS/jacob_lab/$USER"

This now causes errors in several python scripts due to the use of lines lines like those included below which fail to properly interpret $USER within the OutputPath string.

In imi_preview.py:

prior_cache = os.path.join(config["OutputPath"], config["RunName"], "prior_run/OutputDir")

In plumes.py:

basedir = f'{self.config["OutputPath"]}/{self.config["RunName"]}'

One workaround is to define the full file paths in the relevant shell scripts (e.g. preview.sh, statevector.sh) and pass them down into the python routines. Another is to remove use of the $USER in the config files.

Is this something that would be useful to fix? Using $USER in the Harvard Cannon config files was meant to simplify things so users did not need to manually update that path but I'm not sure if others actually use that or if they tend to hardcode their own paths.

[nicholasbalasus]

I thought I had fixed this here: https://github.com/geoschem/integrated_methane_inversion/blob/09690d4b78268ba1846eaa40423d9164a26c72ac/src/inversion_scripts/imi_preview.py#L115-L117

You can probably remove this code with your fix.

[laestrada]

@nicholasbalasus, my fix expands when the config files are read in by the bash scripts, but if the config file is read directly via python expanding the variables is still necessary (as your fix does).

I would have thought that bash expansion would happen automatically with eval $(python parse_yaml.py config.yml), but this fix should work anyways

[msulprizio]

I tried running with the bugfix branch but am still getting:

Traceback (most recent call last):
  File "/var/slurmd/spool/slurmd/job39287365/slurm_script", line 631, in <module>
    sensitivities = estimate_averaging_kernel(*sensitivity_args)
  File "/n/holylfs05/LABS/jacob_lab/Users/msulprizio/IMI/src/inversion_scripts/imi_preview.py", line 445, in estimate_averaging_kernel
    prior_ds = get_mean_emissions(startday, endday, prior_cache)
  File "/n/holylfs05/LABS/jacob_lab/Users/msulprizio/IMI/src/inversion_scripts/utils.py",
line 443, in get_mean_emissions
    prior_files = [f for f in os.listdir(prior_cache_path) if "HEMCO_sa_diagnostics" in f]
FileNotFoundError: [Errno 2] No such file or directory: '/n/holylfs05/LABS/jacob_lab/$USER/Test_IMI_Global/prior_run/OutputDir'

Note $USER is still in the file path in the last line.

[laestrada]

Closed by #241