OverflowError: int too large to convert to float

[antonkulaga]

Atropos crashes with "OverflowError: int too large to convert to float"

2017-10-06 22:44:50,847 ERROR: Unexpected error in Worker process 2
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/multicore.py", line 210, in run
    self.pipeline.process_batch(batch)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/multicore.py", line 146, in process_batch
    super().process_batch(batch)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/base.py", line 58, in process_batch
    self.handle_records(context, records)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/trim/__init__.py", line 48, in handle_records
    super().handle_records(context, records)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/base.py", line 73, in handle_records
    self.handle_record(context, record)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/base.py", line 122, in handle_record
    return self.handle_reads(context, read1, read2)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/trim/__init__.py", line 52, in handle_reads
    return self.record_handler.handle_record(context, read1, read2)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/trim/__init__.py", line 70, in handle_record
    reads = self.modifiers.modify(read1, read2)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/trim/modifiers.py", line 1054, in modify
    read1, read2 = mods(read1, read2)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/trim/modifiers.py", line 363, in __call__
    match = self.aligner.match_insert(read1.sequence, read2.sequence)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/align/__init__.py", line 359, in match_insert
    prob = self.match_probability(insert_match[4], insert_match_size, **self.base_probs)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/util/__init__.py", line 95, in __call__
    self.factorial(i) /
OverflowError: int too large to convert to float
2017-10-06 22:46:01,125 ERROR: Atropos error
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/util/__init__.py", line 673, in run_interruptible
    func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/multicore.py", line 291, in __call__
    self.ensure_alive)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/multicore.py", line 512, in enqueue_all
    fail_callback=fail_callback)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/multicore.py", line 426, in wait_on
    fail_callback()
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/trim/__init__.py", line 686, in ensure_alive
    super().ensure_alive()
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/multicore.py", line 249, in ensure_alive
    ensure_processes(self.worker_processes)
  File "/usr/local/lib/python3.6/site-packages/atropos-0+unknown-py3.6-linux-x86_64.egg/atropos/commands/multicore.py", line 394, in ensure_processes
    str(i) for i, a in enumerate(is_alive) if a != alive)))
atropos.commands.multicore.MulticoreError: One or more process exited: 0,1,2,3,4,5,6
2017-10-06 22:46:05,863 ERROR: Result process waiting on result for 70.1 seconds
2017-10-06 22:46:10,868 ERROR: Result process waiting on result for 75.1 seconds
2017-10-06 22:46:15,874 ERROR: Result process waiting on result for 80.1 seconds
2017-10-06 22:46:20,879 ERROR: Result process waiting on result for 85.1 seconds
2017-10-06 22:46:25,880 ERROR: Result process waiting on result for 90.1 seconds
2017-10-06 22:46:30,886 ERROR: Result process waiting on result for 95.1 seconds
2017-10-06 22:46:35,889 ERROR: Result process waiting on result for 100.1 seconds
2017-10-06 22:46:40,890 ERROR: Result process waiting on result for 105.1 seconds
2017-10-06 22:46:45,896 ERROR: Result process waiting on result for 110.1 seconds
2017-10-06 22:46:50,901 ERROR: Result process waiting on result for 115.1 seconds
2017-10-06 22:46:55,903 ERROR: Result process waiting on result for 120.1 seconds
2017-10-06 22:47:00,903 ERROR: Result process waiting on result for 125.1 seconds
2017-10-06 22:47:01,184 ERROR: Waiting on Result process to terminate for 60.1 seconds

the commands that I run with latest atropos docker container were:

cd /cromwell-executions/quality_de_novo/0cd5264f-4d18-4639-95c7-6b4f34cdd992/call-atropos_illumina_pe/execution
atropos trim \
  --aligner insert \
  -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCACACAGTGATCTCGTATGCCGTCTTCTGCTTG \
  -A AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTAGATCTCGGTGGTCGCCGTATCATT \
  -pe1 /cromwell-executions/quality_de_novo/0cd5264f-4d18-4639-95c7-6b4f34cdd992/call-atropos_illumina_pe/inputs/pipelines/whales/graywhale/GWliver_S1_L001_R1_001.fastq.gz \
  -pe2 /cromwell-executions/quality_de_novo/0cd5264f-4d18-4639-95c7-6b4f34cdd992/call-atropos_illumina_pe/inputs/pipelines/whales/graywhale/GWliver_S1_L001_R2_001.fastq.gz \
  -o GWliver_S1_L001_R1_001_trimmed.fastq.gz \
  -p GWliver_S1_L001_R2_001_trimmed.fastq.gz \
  --threads 8 \
  --correct-mismatches liberal

I can give the input files if needed but they are quite large (several gb-s)

[jdidion]

This is a duplicate of #37 and should be fixed in 1.1.14. Please reopen if you are using 1.1.14+ and still seeing the error.

[antonkulaga]

@jdidion yes, probably the problem was that I used your later atropos docker container without checking that it is outdated. I've build my own container based on master, this error dissapear, now I have a new one https://github.com/jdidion/atropos/issues/48

[jdidion]

Thanks, I'll take a look.

On Oct 7, 2017, at 9:33 AM, Anton Kulaga notifications@github.com wrote:

@jdidion yes, probably the problem was that I used your later atropos docker container without checking that it is outdated. I've build my own container based on master, this error dissapear, now I have a new one #48

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[antonkulaga]

I will also check on my side. Sometimes collaborators send us crappy data. In my case I have 22442394 in both files, so the files seem to be ok

[flpsantos]

No updates in docker hub? Still get same error

[jdidion]

You are right, I need to push a new version to DockerHub. Will open a separate issue for that. Please test on the latest version and open a new issue if you still get the error.