Issue in preprocessing

[RenzoTale88]

Hi again, I'm testing cactus v2 on a UGE system using the following command line:

cactus jobStore $1 $2.hal \
        --maxCores 8 \
        --logFile cactus.${logcode}.log \
        --binariesMode local \
        --writeLogs ${PWD}/LOGS/ \
        --workDir ${PWD}/WDIR \
        --batchSystem grid_engine \
        --disableCaching \
        --defaultDisk 8G \
        --maxDisk 50G \
        --nodeStorage 20 \
        --consCores 8 \
        --defaultMemory 8G \
        --maxLocalJobs 50 \
        --maxServiceJobs 8 \
        $flag

I'm creating the WDIR and specifying a local temporary directory exporting the TMPDIR environment variable prior runtime. However, I keep coming across the following issue after the initial pre-processing, when starting the lastz masking processes:

[2021-06-20T10:50:16+0100] [MainThread] [W] [toil.leader] Log from job "kind-LastzRepeatMaskJob/instance-1p6e9aoy" follows:
=========>
        Traceback (most recent call last):
          File "/PATH/TO/Andrea/cactus-bin-v2.0.1/venv/bin/_toil_worker", line 8, in <module>
            sys.exit(main())
          File "/PATH/TO/Andrea/cactus-bin-v2.0.1/venv/lib/python3.6/site-packages/toil/worker.py", line 683, in main
            with in_contexts(options.context):
          File "/PATH/TO/python/3.6.8/lib/python3.6/contextlib.py", line 81, in __enter__
            return next(self.gen)
          File "/PATH/TO/Andrea/cactus-bin-v2.0.1/venv/lib/python3.6/site-packages/toil/worker.py", line 658, in in_contexts
            with manager:
          File "/PATH/TO/Andrea/cactus-bin-v2.0.1/venv/lib/python3.6/site-packages/toil/batchSystems/abstractBatchSystem.py", line 506, in __enter__
            self.arena.enter()
          File "/PATH/TO/Andrea/cactus-bin-v2.0.1/venv/lib/python3.6/site-packages/toil/lib/threading.py", line 447, in enter
            fcntl.lockf(self.lockfileFD, fcntl.LOCK_EX)
          File "/PATH/TO/python/3.6.8/lib/python3.6/contextlib.py", line 88, in __exit__
            next(self.gen)
          File "/PATH/TO/Andrea/cactus-bin-v2.0.1/venv/lib/python3.6/site-packages/toil/lib/threading.py", line 365, in global_mutex
            os.unlink(lock_filename)
        FileNotFoundError: [Errno 2] No such file or directory: '/PATH/TO/WDIR/toil-mutex-d92ca847-538c-49e8-9f4d-9e41d11c7e0c-cleanup-arena-lock'
<=========

I'm not entirely sure how to fix this, any help would be much appreciated!

Best Andrea

[glennhickey]

is your file system NFS? I don't think Toil works on that...

[RenzoTale88]

@glennhickey good question, I'll check with IT. Thanks!

[naturalstay]

I also encountered the same problem when using the cactus-prepare-toil command. I want to ask how to solve it. Thank you very much! @glennhickey @RenzoTale88