adamnovak
commented
3 years ago It looks like you might have multiple problems happening.
Here's a line from your log:
andra-ThinkPad-E550 2021-01-05 11:48:38,780 MainThread INFO toil: Overriding docker appliance of quay.io/ucsc_cgl/toil:3.24.0-de586251cb579bcb80eef435825cb3cedc202f52-py2.7 with quay.io/ucsc_cgl/toil:latest from TOIL_APPLIANCE_SELF.That looks like you're running on Python 2.7. Toil 3.24 still shipped a Python 2.7 build, and quay.io/ucsc_cgl/toil:3.24.0-de586251cb579bcb80eef435825cb3cedc202f52-py2.7 ought to exist and even work, but quay.io/ucsc_cgl/toil:latest is going to point to a random container for a random Toil version on a random minor version of Python 3 (depending on the order they upload in), and is almost certainly not going to work with Toil 3.24 on Python 2.7. I just pulled quay.io/ucsc_cgl/toil:3.24.0-de586251cb579bcb80eef435825cb3cedc202f52-py2.7 and it pulled fine, so you should leave TOIL_APPLIANCE_SELF unset or use TOIL_APPLIANCE_SELF=quay.io/ucsc_cgl/toil:3.24.0-de586251cb579bcb80eef435825cb3cedc202f52-py2.7. Or, even better, upgrade to the latest Toil release on Python 3. Toil 3.24's Kubernetes code is almost a year old, and we've fixed some pretty big bugs since then, although none that I can recall that look quite like your issue.
What problem did you run into pulling quay.io/ucsc_cgl/toil:3.24.0-de586251cb579bcb80eef435825cb3cedc202f52-py2.7?
However, even with the wrong appliance container, the leader should run through and observe all its jobs failing when they start up and can't understand the commands it sent them. Instead, the jobs aren't even starting, and the leader is crashing.
The stack trace looks like you might have a Kubernetes cluster that doesn't have a working metrics service. Can you run kubectl top nodes and see information about the load on the nodes? Toil relies on the metrics service to work around some conditions we see on our infrastructure where pods get stuck and aren't killed when they run out of memory.
I have some code in https://github.com/DataBiosphere/toil/pull/3357/files#diff-9703248ad4bc5dd9a5dde18161239f4759a2e0e079094d5cdbab7d99b7c592abR639 to make the metrics service optional, but it hasn't been merged in yet.
The part about not being able to mount the "tmp" volume is weird. We did have a volume called "tmp" in Toil 3.24, but as far as I can tell we attached it properly.
Can you do kubectl get -o yaml pod YOUR_POD and/or kubectl get -o yaml job YOUR_JOB, so we can check over the YAML descriptions of the broken pods/jobs? Then we could see if the problem is that Toil is asking for impossible things, or that your Kubernetes cluster somehow can't deliver what it is asking for.
alexiswl
I am trying to deploy a cwl workflow ona kubernetes cluster. I have followed the officially provided instructions and focused on the Option 2: Running the Leader Outside Kubernetes. The workflow I tried is very straightforward:
I ran the workflow using the command: toil-cwl-runner --jobStore aws:us-west-2:toil-cwl-example --batchSystem kubernetes --realTimeLogging --logInfo workflow.cwl inputs.yaml
AWS works okay, meaning the job store is created but, with executing crashing it never deletes.
After creating a pod, the container is never successfuly created. First there seemed to be an error pulling the quay.io/ucsc_cgl/toil image, but I even pulled the image manualy, but the problem still persists.
Here is an error log from toil-cwl-runner:
Also what I find wierd is that multiple successive executions of the workflow produces different events on kubernetes: First run:
Second run:
┆Issue is synchronized with this Jira Task ┆Issue Number: TOIL-771