search

DataBiosphere / toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
http://toil.ucsc-cgl.org/.
Apache License 2.0
894 stars 241 forks source link

resource parsing can't handle nums in scientific notation #1063

Closed arkal closed 8 years ago

arkal commented 8 years ago

A collaborator of mine is seeing this problem.

WARNING: toil.leader: R/N/jobzPnr5J:    Traceback (most recent call last):
WARNING: toil.leader: R/N/jobzPnr5J:      File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/worker.py", line 330, in main
WARNING: toil.leader: R/N/jobzPnr5J:        job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
WARNING: toil.leader: R/N/jobzPnr5J:      File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 1043, in _runner
WARNING: toil.leader: R/N/jobzPnr5J:        returnValues = self._run(jobWrapper, fileStore)
WARNING: toil.leader: R/N/jobzPnr5J:      File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 991, in _run
WARNING: toil.leader: R/N/jobzPnr5J:        return self.run(fileStore)
WARNING: toil.leader: R/N/jobzPnr5J:      File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 1189, in run
WARNING: toil.leader: R/N/jobzPnr5J:        return self.addChildJobFn(userFunction, *self._args, **self._promisedKwargs).rv()
WARNING: toil.leader: R/N/jobzPnr5J:      File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 229, in addChildJobFn
WARNING: toil.leader: R/N/jobzPnr5J:        return self.addChild(JobFunctionWrappingJob(fn, *args, **kwargs))
WARNING: toil.leader: R/N/jobzPnr5J:      File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 1095, in __init__
WARNING: toil.leader: R/N/jobzPnr5J:        checkpoint=kwargs.pop("checkpoint") if "checkpoint" in kwargs else False)
WARNING: toil.leader: R/N/jobzPnr5J:      File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 69, in __init__
WARNING: toil.leader: R/N/jobzPnr5J:        self.disk = parse(disk)
WARNING: toil.leader: R/N/jobzPnr5J:      File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/toil/job.py", line 67, in <lambda>
WARNING: toil.leader: R/N/jobzPnr5J:        parse = lambda x : x if x is None else human2bytes(str(x))
WARNING: toil.leader: R/N/jobzPnr5J:      File "/neoantigen_prediction/protect/venv/lib/python2.7/site-packages/bd2k/util/humanize.py", line 122, in human2bytes
WARNING: toil.leader: R/N/jobzPnr5J:        raise ValueError("can't interpret %r" % init)
WARNING: toil.leader: R/N/jobzPnr5J:    ValueError: can't interpret '1.15102817192e+11'

It arises from a resource request (in this case, disk) being passed a float in the scientific format. The fix lies in modifying and updating bd2k-python-lib.

This PR should fix it: https://github.com/BD2KGenomics/bd2k-python-lib/pull/9

arkal commented 8 years ago

@hannes-ucsc could you look at this please?

hannes-ucsc commented 8 years ago

What's the user script that leads to this? Stuffing float literals down human2bytes' throat is probably just treating a symptom.

arkal commented 8 years ago

This comes from ProTECT. The result of dynamic disk resource requests can be floats.

hannes-ucsc commented 8 years ago

The result of dynamic disk resource requests can be floats.

This should have been in the issue description.

arkal commented 8 years ago

d'oh!