HRRR Herbie package permission issues when running the RAiDER in the Docker

xhuang-jpl commented 1 year ago

Describe the bug

Hi @bbuzz31 and @jhkennedy ,

When we run the ISCE3 InSAR workflow using the Docker, there are some permission issues related to the HRRR Herbie package with the error showing below,

/opt/conda/lib/python3.9/site-packages/nisar/workflows/ionosphere.py:1063: UserWarning: swmr=True only affects read ('r') mode. For swmr write mode, set f.swmr_mode = True after opening the file.
  with h5py.File(iono_output, 'a', libver='latest', swmr=True) as dst_h5:
/opt/conda/lib/python3.9/site-packages/nisar/workflows/geocode_insar.py:1081: UserWarning: swmr=True only affects read ('r') mode. For swmr write mode, set f.swmr_mode = True after opening the file.
  with h5py.File(output_hdf5, "a", libver='latest', swmr=True) as dst_h5:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/pathlib.py", line 1323, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/.config/herbie'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.9/site-packages/nisar/workflows/insar.py", line 123, in <module>
    run(insar_runcfg.cfg, out_paths, persist.run_steps)
  File "/opt/conda/lib/python3.9/site-packages/nisar/workflows/insar.py", line 92, in run
    troposphere.run(cfg, out_paths['GUNW'])
  File "/opt/conda/lib/python3.9/site-packages/nisar/workflows/troposphere.py", line 413, in run
    tropo_delay_datacube = compute_troposphere_delay(cfg, gunw_hdf5)
  File "/opt/conda/lib/python3.9/site-packages/nisar/workflows/troposphere.py", line 157, in compute_troposphere_delay
    from RAiDER.models.hres import HRES
  File "/opt/conda/lib/python3.9/site-packages/RAiDER/models/__init__.py", line 5, in <module>
    from .hrrr import HRRR, HRRRAK
  File "/opt/conda/lib/python3.9/site-packages/RAiDER/models/hrrr.py", line 8, in <module>
    from herbie import Herbie
  File "/opt/conda/lib/python3.9/site-packages/herbie/__init__.py", line 136, in <module>
    _config_path.parent.mkdir(parents=True, exist_ok=True)
  File "/opt/conda/lib/python3.9/pathlib.py", line 1327, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/opt/conda/lib/python3.9/pathlib.py", line 1323, in mkdir
    self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/.config'
total processing time:  0.029475927352905273  sec
file saved: scratch/rubbersheet_offsets/groundTrackVelocity.tif
 ╭─────────────────────────────────────────────────╮
 │ I'm building Herbie's default config file.      │
 ╰╥────────────────────────────────────────────────╯
 👷🏻‍♂️
2023-10-31T20:32:39.807735Z, Error, PGE::insar_pge.py, run_sas_program, 703004, /opt/pge/l1_l2/sas_runner.py:79, "SAS program failed with exit code 1"
2023-10-31T20:32:39.830471Z, Info, PGE::insar_pg

To Reproduce Steps to reproduce the behavior:

Download
Command used
Error Output

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

RAiDER 0.4.5
AWS Linux instance

Additional context Add any other context about the problem her

jhkennedy commented 1 year ago

@xhuang-jpl I'm not super familiar with Herbie, but that stack trace indicates Herbie is looking for a config file at /.config and doesn't have permission to create/read from that directory. But, Herbie should be looking in ${HOME}/.config/:

https://herbie.readthedocs.io/en/2023.3.0/user_guide/configure.html#configure

Herbie is using pathlib.Path.expand:

https://github.com/blaylockbk/Herbie/blob/main/herbie/__init__.py#L73 but the expand method is not a pathlib.Path method and is a custom one added to it by Herbie (:grimacing: ) here:
https://github.com/blaylockbk/Herbie/blob/main/herbie/__init__.py#L45-L68

In the end, this is the actual python code being run to set the Herbie config path:

import os
from pathlib import Path

_config_path = Path(os.path.expandvars("~/.config/herbie/config.toml")).expanduser()

And in the end, os.expanduser()will be used to expand that ~.

I don't have access to the docker container, but I suspect, os.expanduser() ends up with an empty string for HOME. Here's how os.expanduser() determines HOME:

On Unix, an initial ~ is replaced by the environment variable HOME if it is set; otherwise the current user’s home directory is looked up in the password directory through the built-in module pwd. An initial ~user is looked up directly in the password directory.

Can you hop into the container and determine what os.expanduser('~') returns?

dbekaert commented 1 year ago

Is herbie required in the imports? Herbie is only to be used upon pulling of hrrr model to my understanding ?

On Thu, Nov 2, 2023 at 2:58 PM Joseph H Kennedy @.***> wrote:

@xhuang-jpl https://github.com/xhuang-jpl I'm not super familiar with Herbie, but that stack trace indicates Herbie is looking for a config file at /.config and doesn't have permission to create/read from that directory. But, Herbie should be looking in ${HOME}/.config/:

- https://herbie.readthedocs.io/en/2023.3.0/user_guide/configure.html#configure

Herbie is using pathlib.Path.expand:

https://github.com/blaylockbk/Herbie/blob/main/herbie/__init__.py#L73 but the expand method is not a pathlib.Path method and is a custom one added to it by Herbie (😬 ) here:

https://github.com/blaylockbk/Herbie/blob/main/herbie/__init__.py#L45-L68

In the end, this is the actual python code being run to set the Herbie config path:

import osfrom pathlib import Path _config_path = Path(os.path.expandvars("~/.config/herbie/config.toml")).expanduser()

And in the end, os.expanduser() https://docs.python.org/3/library/os.path.html#os.path.expanduserwill be used to expand that ~.

I don't have access to the docker container, but I suspect, os.expanduser() ends up with an empty string for HOME. Here's how os.expanduser() determines HOME:

On Unix, an initial ~ is replaced by the environment variable HOME if it is set; otherwise the current user’s home directory is looked up in the password directory through the built-in module pwd https://docs.python.org/3/library/pwd.html#module-pwd. An initial ~user is looked up directly in the password directory.

Can you hop into the container and determine what os.expanduser('~') returns?

— Reply to this email directly, view it on GitHub https://github.com/dbekaert/RAiDER/issues/608#issuecomment-1791595784, or unsubscribe https://github.com/notifications/unsubscribe-auth/AESZPSOFFCSEE4GLSMUDYPTYCQJP5AVCNFSM6AAAAAA6ZXJALGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJRGU4TKNZYGQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

jhkennedy commented 1 year ago

@dbekaert Herbie setup will be triggered regardless of what model is used because:

it's a top-level import in RAiDER.models.hrrr.py: https://github.com/dbekaert/RAiDER/blob/dev/tools/RAiDER/models/hrrr.py#L8
all the models are imported into RAiDER.models.__init__.py: https://github.com/dbekaert/RAiDER/blob/dev/tools/RAiDER/models/__init__.py

That said, this does not appear to me to be a RAiDER problem -- it's entirely customary to put config files in your $HOME directory and RAiDER also looks for and potentially creates .netrc, .cdsapirc, and .ecmwfapirc files there.

hfattahi commented 1 year ago

@dbekaert Herbie setup will be triggered regardless of what model is used because:

it's a top-level import in RAiDER.models.hrrr.py: https://github.com/dbekaert/RAiDER/blob/dev/tools/RAiDER/models/hrrr.py#L8

all the models are imported into RAiDER.models.__init__.py: https://github.com/dbekaert/RAiDER/blob/dev/tools/RAiDER/models/__init__.py

That said, this does not appear to me to be a RAiDER problem -- it's entirely customary to put config files in your $HOME directory and RAiDER also looks for .netrc, .cdsapirc, and .ecmwfapirc files there.

@jhkennedy I understand that raider or other software may look for .netrc etc in $HOME . But here the issue is not looking for a file in $HOME. The issue is that this package attempts to create a file at $HOME. You can imagine that users may not have that permission. This is a blocking issue for the pipeline currently. @dbekaert is raider team looking into this issue?

jhkennedy commented 1 year ago

The issue is that this package attempts to create a file at $HOME. You can imagine that users may not have that permission.

A user unable to write to their home directory is very non-standard.

This is a blocking issue for the pipeline currently. @dbekaert is raider team looking into this issue?

@hfattahi even if we decided to change our public API to prevent Herbie from writing a config file unless HRRR was explicitly selected, you'd need to rebuild your container with the new package version.

It will be much simpler and faster for you to either:

set an appropriate home directory in your container image
include a readable Herbie config file in your container image at /.config/herbie/config.toml that looks like:
```
[default]
model = "hrrr"
fxx = 0
save_dir = "~/data"
overwrite = false
verbose = true
```

jhkennedy commented 1 year ago

@hfattahi @xhuang-jpl in #612 I took a stab at delaying weather model imports so only the weather model of interest is imported when selected and run, which should prevent Herbie from initializing it's config for models other than HRRR.

I've confirmed that this change would mean Herbie won't initialize when running the these RAiDER import in ICE3/NISAR: https://github.com/isce-framework/isce3/blob/develop/python/packages/nisar/workflows/troposphere.py#L150-L154

(I can't guarantee the subsequent code execution won't still hit it)

However, this change would require a significant refactor of prepFromGUNW.py and the RAiDER.models sub-package, specifically relating to these lines: https://github.com/dbekaert/RAiDER/blob/dev/tools/RAiDER/aria/prepFromGUNW.py#L129-L138

I could rework that to use a dispatch dictionary, which would be cleaner anyways, but the logical place to put that would be in models.__init__.py, which would put us right be back to square one, so I'd have to come up with an alternative.

I won't have capacity anytime soon for a refactor like that, especially for such a peculiar issue, but you or the NISAR team are welcome to pick up where I left off in #612 and move it forward. This at least confirms that options (1) and (2) above are significantly easier than anything here.

Note: I don't particularly like the alternative of just moving the herbie import to right before it's used as it prevents package/environment issues from quickly failing, and doesn't solve this issue in general.

jlmaurer commented 1 year ago

@xhuang-jpl @hfattahi we'll leave this issue ticket open for now; hopefully a workaround as described on this Herbie issue will resolve the issue for now.

dbekaert / RAiDER

HRRR Herbie package permission issues when running the RAiDER in the Docker #608