lwellerastro
commented
2 days ago Example of contents of WORKDIR=TEMP:
TEMP/
from_images_segment1.lis
from_images_segment2.lis
M1214450277LE.lev1.segment1.cub
M1214450277LE.lev1.segment2.cub
M1214450277RE.lev1.segment1.cub
M1214450277RE.lev1.segment2.cub
M156671044LE_cubes_ff_1_from_images_segment1_overlap_fromlist.lis
M156671044LE_cubes_ff_1_from_images_segment1.overlaps
M156671044LE_cubes_ff_1_from_images_segment2_overlap_fromlist.lis
M156671044LE_cubes_ff_1_from_images_segment2.overlaps
M156671044LE_cubes_ff_2_from_images_segment1_overlap_fromlist.lis
M156671044LE_cubes_ff_2_from_images_segment1.overlaps
M156671044LE_cubes_ff_2_from_images_segment2_overlap_fromlist.lis
M156671044LE_cubes_ff_2_from_images_segment2.overlaps
M156671044LE.lev1.segment1.cub
M156671044LE.lev1.segment2.cubIt would be preferable to exclude the segmentcub files from this directory.
This particular example does not run into the problem with /tmp getting full if WORKDIR is not used because it only involves 3 images, but I wanted to get something up that was more descriptive.
ISIS version(s) affected: I8.2.0RC1 (not yet released update)
Description
The latest unreleased version of the script includes a new parameter WORKDIR which was added to save temporary lists from findimageoverlaps and overlapstats runs. This was originally requested for debugging purposes in other script posts.
The new parameter and directory work when specified, however, the image segments are now being written to this area when they hadn't before. The problem occurs if the user doesn't specify WORKDIR and input/output lists along with the image segments are written to /tmp. Most /tmp areas are very small (cluster nodes and astrovm's have about 9G each for /tmp; my VM has 3G) and gets filled up very quickly when the LROC NAC segments are written there, especially for numerous runs on a cluster, and the process dies almost immediately. A long LROC NAC image can be 0.5G in size, which is then duplicated when segmented. The image being worked on and all of the potentially overlapping images in the FROMLIST are also present, so a lot of space is needed. /tmp is not an appropriate area for these data.
Originally the script wrote the newly made image segments to the location where the script was being run - we need to go back to this please. It was up to the user to remove the files after the script completed and that should remain as well. The physical segments are needed for debugging the script (because the overlap lists point to the segments as do intermediate networks) so they shouldn't just be temporary. They are easy to remove after the fact.
How to reproduce
Run the new script on a moderate sized test set, and don't set new WORKDIR parameter so the /tmp is used. I'll point to something that will should fail by default in the comments.
Possible Solution
The only thing that was requested and that should go into the WORKDIR are the findimageoverlaps FROMLIST and OVERLAPLIST and overlapstats FROMLIST, OVERLAPLIST and TOLIST (not sure this is even created) if different from findimageoverlaps.
The description for WORKDIR is as follows:
Perhaps the parameter name should be something new like TEMPDIR, and if the user creates it, it shouldn't be deleted when the program terminates.
Note that the script has a lot of parameters that are reserved for the findfeatures program run including DEBUG and DEBUGLOG, so those can't be used for the temporary area. WORDIR to me says a lot things beyond how it is currently being used and doesn't say "temporary" to me at all. TEMPDIR might not be the best choice either, but I'm not sure what else to recommend since so many other potential words are reserved.