search

gsneha26 / SegAlign

A Scalable GPU-Based Whole Genome Aligner, published in SC20: https://doi.ieeecomputersociety.org/10.1109/SC41405.2020.00043
MIT License
66 stars 16 forks source link

"grep: *.err" and "m: cannot remove '*.segments'" errors #46

Closed thiagogenez closed 3 years ago

thiagogenez commented 3 years ago

Some basic file commands such as grep and rm in the run_segalign script are called without checking if the input files exist. As a result, the script crash and Cactus execution fail (attached below).

I was wondering with the files named with .err, .segments, [`.plus.](https://github.com/gsneha26/SegAlign/blob/dfa1364d7e3199764477f60f0160abeed5bebb60/scripts/run_segalign#L150), and [.minus.*](https://github.com/gsneha26/SegAlign/blob/dfa1364d7e3199764477f60f0160abeed5bebb60/scripts/run_segalign#L151) **must** be created every time thesegalign` script is called.

[Update] The input tmp2e_55g2q.tmp and tmprx912f4d.tmp files are available at https://www.dropbox.com/sh/9bf6o0tij7drafm/AACdWCSz6nkbW7hy6zbpDgeea?dl=0

Cheers, Thiago

Log from job "kind-RunBlast/instance-v7g6i_gt" follows:
=========>
    [2021-03-08T18:54:14+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
    [2021-03-08T18:54:14+0000] [MainThread] [I] [toil] Running Toil version 5.2.0-047d0c4f2949c576c80e452a0807c5be6355c63d on host tf-cactus-slurm-compute-3-0.
    [2021-03-08T18:54:14+0000] [MainThread] [D] [toil] Configuration: {'workflowID': '5d8e7caf-fad5-4d3b-9303-ae20634083f8', 'workflowAttemptNumber': 0, 'jobStore': 'file:/home/thiagogenez_ebi_ac_uk/viruses/run/jobStore', 'logLevel': 'Debug', 'workDir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'never', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 8, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 1000, 'scale': 1.0, 'linkImports': True, 'moveExports': False, 'mesosMasterAddress': '10.0.0.12:5050', 'allocate_mem': True, 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 3600, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 5, 'enableUnlimitedPreemptableRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': True, 'disableChaining': True, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 64000, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
    [2021-03-08T18:54:14+0000] [MainThread] [D] [toil.deferred] Running for file /tmp/node-5d8e7caf-fad5-4d3b-9303-ae20634083f8-9a39eee8-e096-45e7-8d8e-a2a3f2cb7837/deferred/funceiihipwj
    [2021-03-08T18:54:14+0000] [MainThread] [D] [toil.worker] Parsed job description
    [2021-03-08T18:54:14+0000] [MainThread] [I] [toil.worker] Working on job 'RunBlast' kind-RunBlast/instance-v7g6i_gt
    [2021-03-08T18:54:14+0000] [MainThread] [D] [toil.worker] Got a command to run: _toil files/for-job/kind-RunBlast/instance-v7g6i_gt/cleanup/file-ec7bc1a3a798485ab637ad8f261e43e4/stream /home/thiagogenez_ebi_ac_uk/.local/bin/cactus-bin-v1.3.0/venv/lib/python3.6/site-packages cactus.blast.blast True
    [2021-03-08T18:54:14+0000] [MainThread] [D] [toil.job] Loading user module ModuleDescriptor(dirPath='/home/thiagogenez_ebi_ac_uk/.local/bin/cactus-bin-v1.3.0/venv/lib/python3.6/site-packages', name='cactus.blast.blast', fromVirtualEnv=True).
    [2021-03-08T18:54:14+0000] [MainThread] [I] [toil.worker] Loaded body Job('RunBlast' kind-RunBlast/instance-v7g6i_gt) from description 'RunBlast' kind-RunBlast/instance-v7g6i_gt
    [2021-03-08T18:54:14+0000] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
    [2021-03-08T18:54:14+0000] [MainThread] [D] [toil.deferred] Running job
    [2021-03-08T18:54:14+0000] [MainThread] [I] [cactus.shared.common] Docker work dir: /tmp/node-5d8e7caf-fad5-4d3b-9303-ae20634083f8-9a39eee8-e096-45e7-8d8e-a2a3f2cb7837/tmph97m_giv/ee1594c4-2d84-45b0-b8ef-4ed978d7201c
    [2021-03-08T18:54:14+0000] [MainThread] [I] [cactus.shared.common] Running the command ['singularity', '--silent', 'run', '--nv', '/home/thiagogenez_ebi_ac_uk/viruses/run/cactus.img', 'run_segalign', 'tmp2e_55g2q.tmp', 'tmprx912f4d.tmp', '--format=cigar', '--notrivial', '--step=2', '--ambiguous=iupac,100,100', '--ydrop=3000']
    [2021-03-08T18:54:14+0000] [MainThread] [D] [toil.statsAndLogging] Suppressing the following loggers: {'websocket', 'bcdocs', 'urllib3', 'sonLib', 'google', 'requests_oauthlib', 'humanfriendly', 'galaxy', 'dill', 'prov', 'oauthlib', 'kubernetes', 'cactus', 'botocore', 'salad', 'boto', 'cachecontrol', 'rdflib', 'boto3', 'docker', 'requests'}
    [2021-03-08T18:54:14+0000] [MainThread] [I] [toil-rt] 2021-03-08 18:54:14.635398: Running the command: "singularity --silent run --nv /home/thiagogenez_ebi_ac_uk/viruses/run/cactus.img run_segalign tmp2e_55g2q.tmp tmprx912f4d.tmp --format=cigar --notrivial --step=2 --ambiguous=iupac,100,100 --ydrop=3000"
    /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.UTF-8)

    Converting fasta files to 2bit format

    Executing: "segalign /tmp/node-5d8e7caf-fad5-4d3b-9303-ae20634083f8-9a39eee8-e096-45e7-8d8e-a2a3f2cb7837/tmph97m_giv/ee1594c4-2d84-45b0-b8ef-4ed978d7201c/tmp2e_55g2q.tmp /tmp/node-5d8e7caf-fad5-4d3b-9303-ae20634083f8-9a39eee8-e096-45e7-8d8e-a2a3f2cb7837/tmph97m_giv/ee1594c4-2d84-45b0-b8ef-4ed978d7201c/tmprx912f4d.tmp /tmp/node-5d8e7caf-fad5-4d3b-9303-ae20634083f8-9a39eee8-e096-45e7-8d8e-a2a3f2cb7837/tmph97m_giv/ee1594c4-2d84-45b0-b8ef-4ed978d7201c/output_11930/data_5519/  --format=cigar --notrivial --step=2 --ambiguous=iupac,100,100 --ydrop=3000"
    Using 8 threads
    Using 1 GPU(s)

    Reading query file ...

    Reading target file ...

    Start alignment ...

    Sending reference block 0 ...

    Sending query block 0 with buffer 0 ...
    Query block 0, interval 1/1 (0:28184) with buffer 0

    real    0m1.184s
    user    0m0.106s
    sys 0m1.012s
    grep: *.err: No such file or directory
    rm: cannot remove '*.segments': No such file or directory
    [2021-03-08T18:54:16+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
    [2021-03-08T18:54:16+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-BlastSequencesAgainstEachOther/instance-7jk5yz3n/cleanup/file-e798bb1571b344409a9fdef384794920/0' to path '/tmp/node-5d8e7caf-fad5-4d3b-9303-ae20634083f8-9a39eee8-e096-45e7-8d8e-a2a3f2cb7837/tmph97m_giv/ee1594c4-2d84-45b0-b8ef-4ed978d7201c/tmp2e_55g2q.tmp'
    [2021-03-08T18:54:16+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-BlastSequencesAgainstEachOther/instance-7jk5yz3n/cleanup/file-7a446f73b95d4f5f938fc2ba16ebb915/0' to path '/tmp/node-5d8e7caf-fad5-4d3b-9303-ae20634083f8-9a39eee8-e096-45e7-8d8e-a2a3f2cb7837/tmph97m_giv/ee1594c4-2d84-45b0-b8ef-4ed978d7201c/tmprx912f4d.tmp'
    [2021-03-08T18:54:16+0000] [MainThread] [D] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job files/for-job/kind-RunBlast/instance-v7g6i_gt/cleanup/file-ec7bc1a3a798485ab637ad8f261e43e4/stream used 0.00% (48.0 KB [49152B] used, 1.0 GB [1677721600B] requested) at the end of its run.
    [2021-03-08T18:54:16+0000] [MainThread] [D] [toil.deferred] Running own deferred functions
    [2021-03-08T18:54:16+0000] [MainThread] [D] [toil.deferred] Out of deferred functions!
    [2021-03-08T18:54:16+0000] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
    Traceback (most recent call last):
      File "/home/thiagogenez_ebi_ac_uk/.local/bin/cactus-bin-v1.3.0/venv/lib/python3.6/site-packages/toil/worker.py", line 394, in workerScript
        job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/home/thiagogenez_ebi_ac_uk/.local/bin/cactus-bin-v1.3.0/venv/lib/python3.6/site-packages/cactus/shared/common.py", line 1424, in _runner
        fileStore=fileStore, **kwargs)
      File "/home/thiagogenez_ebi_ac_uk/.local/bin/cactus-bin-v1.3.0/venv/lib/python3.6/site-packages/toil/job.py", line 2359, in _runner
        returnValues = self._run(jobGraph=None, fileStore=fileStore)
      File "/home/thiagogenez_ebi_ac_uk/.local/bin/cactus-bin-v1.3.0/venv/lib/python3.6/site-packages/toil/job.py", line 2280, in _run
        return self.run(fileStore)
      File "/home/thiagogenez_ebi_ac_uk/.local/bin/cactus-bin-v1.3.0/venv/lib/python3.6/site-packages/cactus/blast/blast.py", line 466, in run
        gpuLastz = self.blastOptions.gpuLastz)
      File "/home/thiagogenez_ebi_ac_uk/.local/bin/cactus-bin-v1.3.0/venv/lib/python3.6/site-packages/cactus/shared/common.py", line 833, in runLastz
        parameters=[lastzCommand, seq1, seq2, "--format=cigar", "--notrivial"] + lastzArguments.split())
      File "/home/thiagogenez_ebi_ac_uk/.local/bin/cactus-bin-v1.3.0/venv/lib/python3.6/site-packages/cactus/shared/common.py", line 1357, in cactus_call
        raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
    RuntimeError: Command ['singularity', '--silent', 'run', '--nv', '/home/thiagogenez_ebi_ac_uk/viruses/run/cactus.img', 'run_segalign', 'tmp2e_55g2q.tmp', 'tmprx912f4d.tmp', '--format=cigar', '--notrivial', '--step=2', '--ambiguous=iupac,100,100', '--ydrop=3000'] exited 1: stdout=None
    [2021-03-08T18:54:16+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host tf-cactus-slurm-compute-3-0
<=========
gsneha26 commented 3 years ago

Hi Thiago,

Thank you for pointing out this issue. I have fixed it in the latest commit. The files *.err, .segments, *.plus.*, and *.minus.* are not generated if there is no alignment between the 2 sequences.

Hope this helps!

Sneha

thiagogenez commented 3 years ago

Thank you @gsneha26 for your reply. Cool. I will pull your lastest commit and check it out =)

Cheers, Thiago