FORCE temporary directory

[lucadelu]

I got an error during FORCE processing related to temporary directory, it seems to write into /tmp/ but in my config.yml file I set temp_force_dir: '/scratch3/delucchil/sadasadam/temp_force/' (/scratch3/ has some free TB)

14 images enqueued. Start processing with 1 CPUs
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "C.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "C.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

Computers / CPU cores / Max jobs to run
1:local / 56 / 1

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 658s Left: 7 AVG: 94.14s  local:1/7/100%/94.4s gsl: simplex2.c:372: ERROR: non-finite function value encountered
Default GSL error handler invoked.
/home/docker/bin/force-l2ps_: line 161: 44572 Aborted                 (core dumped) $EXE $TODO $PRM >> $LOGFILE
ETA: 639s Left: 6 AVG: 106.75s  local:1/8/100%/107.0s gsl: simplex2.c:372: ERROR: non-finite function value encountered
Default GSL error handler invoked.
/home/docker/bin/force-l2ps_: line 161: 46535 Aborted                 (core dumped) $EXE $TODO $PRM >> $LOGFILE
ETA: 565s Left: 5 AVG: 113.11s  local:1/9/100%/113.3s parallel: Error: Output is incomplete. Cannot append to buffer file in /tmp. Is the disk full?
parallel: Error: Change $TMPDIR with --tmpdir or use --compress.
Warning: unable to close filehandle properly: No space left on device during global destruction.
/usr/local/lib/python3.8/dist-packages/sadasadam/force.py:47: UserWarning: Directory /scratch3/delucchil/sadasadam/output already exists, skipping...
  warnings.warn(f"Directory {directory} already exists, skipping...")
FORCE log files copied to /scratch3/delucchil/sadasadam/output
Creation of same day mosaics...
Traceback (most recent call last):
  File "/usr/local/bin/sadasadam", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/sadasadam/cli.py", line 252, in main
    force_proc.run_force_mosaic()
  File "/usr/local/lib/python3.8/dist-packages/sadasadam/force.py", line 348, in run_force_mosaic
    run_subprocess(cmd_list)
  File "/usr/local/lib/python3.8/dist-packages/sadasadam/force.py", line 90, in run_subprocess
    raise Exception(f"Error running process {cmd_str}:\n {stderr_dec}")
Exception: Error running process force-mosaic -m mosaic /scratch3/delucchil/sadasadam/temp_force/force_dir_20231122_115245/level2:
 perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "C.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "C.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "C.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Error in tempfile() using template /tmp/parXXXXX.par: Could not create temp file /tmp/parA5XPl.par: No space left on device at /usr/bin/parallel line 3889.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "C.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "C.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Error in tempfile() using template /tmp/parXXXXX.par: Could not create temp file /tmp/parf0sJq.par: No space left on device at /usr/bin/parallel line 3889.

[griembauer]

The temp_force_dir is used by sadasadam to store intermediate FORCE results in a specific folder with a timestamp (in the example above: scratch3/delucchil/sadasadam/temp_force/force_dir_20231122_115245). Data stored there are for example the whole level-2 FORCE datacube and the FORCE mosaics.

Error: Output is incomplete. Cannot append to buffer file in /tmp. Is the disk full? parallel: Error: Change $TMPDIR with --tmpdir or use --compress --> It may be that FORCE itself uses the /tmp directory to store some further temporary data during processing. Maybe setting the $TMPDIR env variable helps?

If the data has not been cleaned up (or if you used remove_force_data: False), you could also check the data in /scratch3/delucchil/sadasadam/temp_force/force_dir_20231122_115245/level2 as well as the FORCE log files.

This issue seems to be related to the GNU parallel usage of FORCE level-2 processing. I found a similar issue (unrelated to FORCE) here. I am not sure whether the issue lies in sadasadam or FORCE itself. If you still have the downloaded data, you can try to directly run the FORCE command instead of sadasadam to isolate the problem further, for this you would need to define a FORCE parameter file.

[lucadelu]

The temp_force_dir is used by sadasadam to store intermediate FORCE results in a specific folder with a timestamp (in the example above: scratch3/delucchil/sadasadam/temp_force/force_dir_20231122_115245). Data stored there are for example the whole level-2 FORCE datacube and the FORCE mosaics.

Error: Output is incomplete. Cannot append to buffer file in /tmp. Is the disk full? parallel: Error: Change $TMPDIR with --tmpdir or use --compress --> It may be that FORCE itself uses the /tmp directory to store some further temporary data during processing. Maybe setting the $TMPDIR env variable helps?

ok, I run it in a different blade and the error didn't appear so probably the issue of full disk was not related to me, however to be sure I'll try to set different $TMPDIR env variable

If the data has not been cleaned up (or if you used remove_force_data: False), you could also check the data in /scratch3/delucchil/sadasadam/temp_force/force_dir_20231122_115245/level2 as well as the FORCE log files.

This issue seems to be related to the GNU parallel usage of FORCE level-2 processing. I found a similar issue (unrelated to FORCE) here. I am not sure whether the issue lies in sadasadam or FORCE itself. If you still have the downloaded data, you can try to directly run the FORCE command instead of sadasadam to isolate the problem further, for this you would need to define a FORCE parameter file.

ok, I could try, however now it is running but I have other errors, I'll report tomorrow new issues.

For now we can keep this issue open but nothing to do, thanks