search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
481 stars 106 forks source link

ValueError in redMasking.py due to invalid literal for int() with base 10 #1408

Closed thiagogenez closed 2 weeks ago

thiagogenez commented 2 weeks ago

Issue Description

A ValueError occurs when running the RedMaskJob in the cactus preprocessor. The error message indicates that an invalid literal for int() with base 10 was encountered: '4.31768e+09'. This issue arises during the execution of cactus_redPrefilter.

=========>
        [2024-06-12T14:30:19+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
        [2024-06-12T14:30:19+0100] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host codon-gpu-008.ebi.ac.uk.
        [2024-06-12T14:30:19+0100] [MainThread] [D] [toil] Configuration: {'cwl': False, 'workflowID': '88d241ba-cd70-489b-ae18-50b7e0bb2d36', 'kill_polling_interval': 5, 'jobStore': 'file:/hps/nobackup/flicek/ensembl/compara/thiagogenez/work/wheat/run2/batch/1-preprocessors/jobstore/1', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'onSuccess', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'aws_batch_region': None, 'aws_batch_queue': None, 'aws_batch_job_role_arn': None, 'scale': 1, 'allocate_mem': True, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': [0], 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': False, 'retryCount': 0, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 104857600, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'Debug', 'workflowAttemptNumber': 0}
        [2024-06-12T14:30:19+0100] [MainThread] [D] [toil.deferred] Opened with own state file /tmp/toilwf-dc39984165cf536ebcc8073825e10d39/deferred/funcaiel1nb2
        [2024-06-12T14:30:19+0100] [MainThread] [D] [toil.worker] Parsed job description
        [2024-06-12T14:30:19+0100] [MainThread] [I] [toil.worker] Working on job 'RedMaskJob' kind-RedMaskJob/instance-5hl88ec2 v1
        [2024-06-12T14:30:19+0100] [MainThread] [D] [toil.worker] Got a command to run: _toil files/for-job/kind-RedMaskJob/instance-5hl88ec2/cleanup/file-054baea8b68e484db3faf3dc400885b4/stream /home/cactus/cactus_env/lib/python3.8/site-packages cactus.preprocessor.redMasking True
        [2024-06-12T14:30:19+0100] [MainThread] [D] [toil.job] Loading user module ModuleDescriptor(dirPath='/home/cactus/cactus_env/lib/python3.8/site-packages', name='cactus.preprocessor.redMasking', fromVirtualEnv=True).
        [2024-06-12T14:30:19+0100] [MainThread] [I] [toil.worker] Loaded body Job('RedMaskJob' kind-RedMaskJob/instance-5hl88ec2 v1) from description 'RedMaskJob' kind-RedMaskJob/instance-5hl88ec2 v1
        [2024-06-12T14:30:19+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
        [2024-06-12T14:30:19+0100] [MainThread] [D] [toil.deferred] Ran orphaned deferred functions from 0 abandoned state files
        [2024-06-12T14:30:19+0100] [MainThread] [D] [toil.deferred] Running job
        [2024-06-12T14:30:48+0100] [MainThread] [I] [cactus.shared.common] Running the command ['cactus_redPrefilter', '/tmp/toilwf-dc39984165cf536ebcc8073825e10d39/1b71/3729/tmpgdluw72m/triticum_urartu.fa', '-m', '20000', '-b', '0.98']
        [2024-06-12T14:30:48+0100] [MainThread] [D] [toil.statsAndLogging] Suppressing the following loggers: {'botocore', 'docker', 'sonLib', 'dill', 'boto', 'asyncio', 'cactus', 'bcdocs', 'concurrent', 'urllib3', 'requests', 'charset_normalizer', 'boto3'}
        [2024-06-12T14:30:48+0100] [MainThread] [I] [toil-rt] 2024-06-12 14:30:48.596012: Running the command: "cactus_redPrefilter /tmp/toilwf-dc39984165cf536ebcc8073825e10d39/1b71/3729/tmpgdluw72m/triticum_urartu.fa -m 20000 -b 0.98"
        [2024-06-12T14:31:23+0100] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.bytes2human()" instead."
        [2024-06-12T14:31:23+0100] [MainThread] [I] [toil-rt] 2024-06-12 14:31:23.042710: Successfully ran: "cactus_redPrefilter /tmp/toilwf-dc39984165cf536ebcc8073825e10d39/1b71/3729/tmpgdluw72m/triticum_urartu.fa -m 20000 -b 0.98" in 34.4389 seconds and 1.3 Gi memory
        [2024-06-12T14:31:23+0100] [MainThread] [I] [toil-rt] 2024-06-12 14:31:23.044211: Running the command: "cactus_softmask2hardmask -b /tmp/toilwf-dc39984165cf536ebcc8073825e10d39/1b71/3729/tmpgdluw72m/red-in-triticum_urartu/triticum_urartu.filter.fa"
        [2024-06-12T14:31:50+0100] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.bytes2human()" instead."
        [2024-06-12T14:31:50+0100] [MainThread] [I] [toil-rt] 2024-06-12 14:31:50.000809: Successfully ran: "cactus_softmask2hardmask -b /tmp/toilwf-dc39984165cf536ebcc8073825e10d39/1b71/3729/tmpgdluw72m/red-in-triticum_urartu/triticum_urartu.filter.fa" in 26.9457 seconds and 1.3 Gi memory
        [2024-06-12T14:31:50+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
        [2024-06-12T14:31:50+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-PreprocessSequence/instance-39t34pah/cleanup/file-360809766d394cb0b4fe08f9205e9274/tmptl3e7v04.tmp' to path '/tmp/toilwf-dc39984165cf536ebcc8073825e10d39/1b71/3729/tmpgdluw72m/triticum_urartu.fa'
        [2024-06-12T14:31:50+0100] [MainThread] [D] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job files/for-job/kind-RedMaskJob/instance-5hl88ec2/cleanup/file-054baea8b68e484db3faf3dc400885b4/stream used 37.21% disk (9.1 GiB [9792798720B] used, 24.5 GiB [26319257600B] requested).
        [2024-06-12T14:31:50+0100] [MainThread] [D] [toil.deferred] Running own deferred functions
        [2024-06-12T14:31:50+0100] [MainThread] [D] [toil.deferred] Out of deferred functions!
        [2024-06-12T14:31:50+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
        [2024-06-12T14:31:50+0100] [MainThread] [D] [toil.deferred] Ran orphaned deferred functions from 0 abandoned state files
        Traceback (most recent call last):
          File "/home/cactus/cactus_env/lib/python3.8/site-packages/toil/worker.py", line 409, in workerScript
            job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
          File "/home/cactus/cactus_env/lib/python3.8/site-packages/cactus/shared/common.py", line 979, in _runner
            super(RoundedJob, self)._runner(*args, jobStore=jobStore,
          File "/home/cactus/cactus_env/lib/python3.8/site-packages/toil/job.py", line 2845, in _runner
            returnValues = self._run(jobGraph=None, fileStore=fileStore)
          File "/home/cactus/cactus_env/lib/python3.8/site-packages/toil/job.py", line 2761, in _run
            return self.run(fileStore)
          File "/home/cactus/cactus_env/lib/python3.8/site-packages/cactus/preprocessor/redMasking.py", line 67, in run
            pre_mask_size = int(awkres) if awkres else 0
        ValueError: invalid literal for int() with base 10: '4.31768e+09'
        [2024-06-12T14:31:50+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host codon-gpu-008.ebi.ac.uk
<=========
thiagogenez commented 2 weeks ago

This is a similar error message from issue https://github.com/ComparativeGenomicsToolkit/cactus/issues/1327

thiagogenez commented 2 weeks ago

Proposed solution is here: https://github.com/ComparativeGenomicsToolkit/cactus/pull/1410