search

Sage-Bionetworks / SynapseWorkflowHook

Code for linking a workflow engine to a Synapse evaluation queue
Apache License 2.0
4 stars 1 forks source link

Workflow ReadTimeout error #34

Open thomasyu888 opened 5 years ago

thomasyu888 commented 5 years ago 
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

I researched a bit and found these: https://github.com/docker/compose/issues/1045 https://github.com/docker/compose/issues/812

Unsure of definite solutions.

brucehoff commented 5 years ago

Can you run with debuging on: docker-compose --verbose up 'to see which API call is hanging' (as per https://github.com/docker/compose/issues/812) and/or to get other debug info that might help?

thomasyu888 commented 5 years ago

I can try. But unsure if I can reproduce. I think I should always have --verbose on just so I can send errors.

thomasyu888 commented 5 years ago

Similar to https://github.com/Sage-Bionetworks/SynapseWorkflowHook/issues/33. I am going to change the ec2 instance type to something with more memory and CPUs, and enforce a memory limit on the participant submissions docker run command.

I will attempt to run through ~100 submissions without failure.

brucehoff commented 5 years ago

If you determine that memory is the issue then I can address this by changing the memory that the 'hook' uses (in docker-compose.yaml) and that the Toil container uses.

thomasyu888 commented 5 years ago

@brucehoff

After a bit more testing, I think that issue #33 and this one is caused by submissions eating up the memory of the machine. The solution is to pass in the mem_limit parameter to the docker run command in my CWL tool.

The one thing to investigate is if the toil cli commands passed into the hook in the .env file is propagated to the internal workings of TOIL calling docker in a CWL tool step that uses Docker as a hint. For now, the immediate solution is to document my challenge working templates to make sure that people use the mem_limit parameter and perhaps the hook as well.