files saved in output directory not where we expect them and special characters in filenames

[asaxton]

On hydro's a100 node I ran

python pipeline.py -v \
       --inactive_timeout 60 \
       --data /projects/bbym/shared/commonDebug/data \
       --output /projects/bbym/shared/commonDebug/output \
       --legends /projects/bbym/shared/commonDebug/legends \
       --layout /projects/bbym/shared/commonDebug/layouts \
       --feedback /projects/bbym/shared/commonDebug/feedback \
       --output_types cdr_json raster_masks \
       --log /logs/gpu-${SLURM_JOBID}.log \
       --amqp amqp://${UNAME}:${PASSWD}@criticalmaas.ncsa.illinois.edu:5672/%2F \
       --model golden_muscat &> log.${SLURM_JOBID}_${HOSTNAME}.txt &

On the rabbitMQ process_golden_muscat queue the json message was

{"cog_id": "6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1", "cog_url": "https://s3.amazonaws.com/public.cdr.land/cogs/6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1.cog.tif", "map_area": "https://criticalmaas.ncsa.illinois.edu/cdr/download/6a/4f/6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1.cog_area.json", "models": ["golden_muscat"], "image_filename": "6a/4f/6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1/6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1.cog.tif", "json_filename": "6a/4f/6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1/6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1.cog_area.json"}

the contents of the output directory was

(venv) (CMAAS_py_3.10) BW[hydrol1: (Wed Jun 05 16:15:02) output saxton>$ ls -lR
.:
total 4
drwxrws---+ 2 saxton hydro_bbym 4096 Jun  5 16:05 tmp

./tmp:
total 43528
-rw-rw----+ 1 saxton hydro_bbym  1099340 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_5□tif
-rw-rw----+ 1 saxton hydro_bbym 28646979 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_cdr□json
-rw-rw----+ 1 saxton hydro_bbym  1423758 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_Eq□tif
-rw-rw----+ 1 saxton hydro_bbym  1077382 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_Jd□tif
-rw-rw----+ 1 saxton hydro_bbym  1181473 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_Ji□tif
-rw-rw----+ 1 saxton hydro_bbym  1046421 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_NO_OCR_RESULT_0□tif
-rw-rw----+ 1 saxton hydro_bbym  1957785 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_NO_OCR_RESULT_1□tif
-rw-rw----+ 1 saxton hydro_bbym  2190387 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_NO_OCR_RESULT_2□tif
-rw-rw----+ 1 saxton hydro_bbym  1320681 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_Pq□tif
-rw-rw----+ 1 saxton hydro_bbym  1089345 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_Pv□tif
-rw-rw----+ 1 saxton hydro_bbym  1155817 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_Rc□tif
-rw-rw----+ 1 saxton hydro_bbym  1202222 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_Re□tif
-rw-rw----+ 1 saxton hydro_bbym  1157750 Jun  5 16:05 6a4f4a4c7ac57ec75e876f872152256335a211e74f9749a713c712de09db4de1□cog_Rg□tif

The output should be placed in the directory f'output/{cog_id[0:2]}/{cog_id[2:4]}'. Also, there seems to be "□" in the file names. Is that expected?

[asaxton]

for clarity, this was tested with code on main branch at commit 0fc58b10730a12104d3ca13b26b866a7679b7cc8. The environment was setup using the process described in README.md > Quick Start > Installing> For Developers

[abodeuis]

Pushed commit 4b61114d9e4f849da6338e80d4d65465dbd08d53 to main should address the □ character replacing '.' in the extension. Will discuss this further with rob for a more permanent solution.

[asaxton]

Fixing the "□ character replacing '.'" problem seems to have fixed the "The output should be placed in the directory f'output/{cog_id[0:2]}/{cog_id[2:4]}'" problem too. I think this issue can be closed until we find it pop up again.