search

data2health / DREAM-Challenge

EHR DREAM Challenge
7 stars 2 forks source link

Timed out docker containers are not being stopped #49

Closed trberg closed 4 years ago

trberg commented 4 years ago

When containers run for more than the allotted time (10 hrs.) the toil workflow hook that is running the container is stopped but the submitted docker container is not. It will continue until the administrator manually kills it or it stops on its own. This could be a problem if the workflow hook thinks that the still running container has been stopped and it pulls in a new submission leading to a memory overflow. The other issue is the workflow hook is not being stopped gracefully, so logs are not being saved after the time quota.

tschaffter commented 4 years ago

@trberg I believe that this issue has been fixed a long time ago. Can you provide a brief description of the solution for future references before closing this ticket? Thanks!

thomasyu888 commented 4 years ago

I implemented a tool that looks at currently running docker containers-(I name the running containers with submission id). If the submission is INVALID, then stop and remove the container.