Temporary products not written to scratch directory defined by RunConfig

collinss-jpl commented 3 months ago

While testing the DSWx-S1 and DSWx-NI deliveries, I discovered that changing the docker working directory to /home/dswx_user (from the default of /home/dswx_user/scratch) resulted in the following permission error:

Traceback (most recent call last):
  File "/home/dswx_user/miniconda3/envs/DSWX-SAR/bin/dswx_ni.py", line 103, in <module>
    main()
  File "/home/dswx_user/miniconda3/envs/DSWX-SAR/bin/dswx_ni.py", line 98, in main
    dswx_ni_workflow(cfg)
  File "/home/dswx_user/miniconda3/envs/DSWX-SAR/bin/dswx_ni.py", line 75, in dswx_ni_workflow
    refine_with_bimodality.run(cfg)
  File "/home/dswx_user/miniconda3/envs/DSWX-SAR/lib/python3.11/site-packages/dswx_sar/refine_with_bimodality.py", line 1495, in run
    dswx_sar_util.merge_binary_layers(
  File "/home/dswx_user/miniconda3/envs/DSWX-SAR/lib/python3.11/site-packages/dswx_sar/dswx_sar_util.py", line 1133, in merge_binary_layers
    write_raster_block(
  File "/home/dswx_user/miniconda3/envs/DSWX-SAR/lib/python3.11/site-packages/dswx_sar/dswx_sar_util.py", line 939, in write_raster_block
    _save_as_cog(out_raster, scratch_dir)
  File "/home/dswx_user/miniconda3/envs/DSWX-SAR/lib/python3.11/site-packages/dswx_sar/dswx_sar_util.py", line 349, in _save_as_cog
    temp_file = tempfile.NamedTemporaryFile(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dswx_user/miniconda3/envs/DSWX-SAR/lib/python3.11/tempfile.py", line 563, in NamedTemporaryFile
    file = _io.open(dir, mode, buffering=buffering,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dswx_user/miniconda3/envs/DSWX-SAR/lib/python3.11/tempfile.py", line 560, in opener
    fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dswx_user/miniconda3/envs/DSWX-SAR/lib/python3.11/tempfile.py", line 256, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/home/dswx_user/tmp88whrsxh.tif'

This is because there are two locations in the code where temporary products are written to the current working directory ('.') instead of the scratch directory defined by the RunConfig:

Redefining the container working directory is useful for PGE/SDS integration, so it would be nice if these temp products could get written to defined scratch directory like other temporary products.

oberonia78 commented 2 months ago

Hi @collinss-jpl This issues were resolved in the most recent patch release (DSWx-S1 v0.4.2 ).

collinss-jpl commented 2 months ago

Hi @collinss-jpl This issues were resolved in the most recent patch release (DSWx-S1 v0.4.2 ).

Hi @oberonia78, I was just able to verify the fix during acceptance testing of the 4.2 delivery, thanks for the quick turnaround!

collinss-jpl commented 3 weeks ago

Hi @oberonia78, I've been seeing this issue crop up again in the interface delivery for DSWx-NI. Seems like the fix for the issue was only merged to the dswx-s1-calval branch, and not the main branch, so it was not included with the DSWx-NI delivery.

oberonia78 commented 3 weeks ago

Thanks @collinss-jpl for reporting the issue. We are planning to merge the recent changes for dswx-s1-calval to the main branch (https://github.com/opera-adt/DSWX-SAR/pull/89). This issue will be fixed in the next delivery.

opera-adt / DSWX-SAR

Temporary products not written to scratch directory defined by RunConfig #78