rbeyer / hiproc

Python libraries and programs for processing HiRISE data.
Apache License 2.0
4 stars 3 forks source link

empty 'sys' and 're' files created #2

Closed AndrewAnnex closed 3 years ago

AndrewAnnex commented 3 years ago

Description

When testing the full hiproc pipeline with the hiproc command with hijack force enabled, two empty (0B) files 'sys' and 're' are created by hiproc. I am not sure where these files are being created as nothing obvious in the code is making them and they share the names of parts of the python standard library so it's not clear if they are some intentional intermediate product that keep wants to retain, all I know is they were created near the final 3 minutes of the pipeline so that points to one of the later processing steps being responsible.

What I Did

hiproc -k -p --logfile hiproc.log ./ESP_016657_1890*.IMG
rbeyer commented 3 years ago

Wild. I can't seem to reproduce. I tried with Python 3.9.5 and I tried with Python 3.6. No weirdly-named, zero-length files. The only difference is maybe that I'm testing on macOS and not Ubuntu, but that seems unlikely.

Since you "kept" all of the intermediate files, could you give me a time-based directory listing, so I can see what files get made immediately before and immediately after these weird empty files, that might help me at least narrow down what might be emitting them from on your system, but quite honestly, I'm not entirely sure how they'd get made with those names

Also your --logfile is probably pretty empty, you might want to consider also throwing the -v switch.

AndrewAnnex commented 3 years ago

this is weird, as I only saw it with this one hirise image so far. Looking through the logs and times doesn't really show anything as so I am going to clean the state and re-run the job overnight and see what happens.

AndrewAnnex commented 3 years ago

I am closing this issue as I wasn't able to reproduce it so it may be caused by something else external to hiproc. closing for now