common-workflow-language / cwltool

Common Workflow Language reference implementation
https://cwltool.readthedocs.io/
Apache License 2.0
335 stars 230 forks source link

jobs exceeding max allocation of a resource are not limited or failed #339

Open qiukunlong opened 7 years ago

qiukunlong commented 7 years ago

In CWL CommandLineTool Document, it is described that "If a job attempts to exceed its "max" resource allocation, an implementation may deny additional resources, which may result in job failure".

I set ramMax: 2000 in my CWL, and use cwltool to run the CWL, the job exceed the max memory I set and achieve 6G memory. In my opinion, the job memory used should limited at a upbound of 2G.

mr-c commented 7 years ago

Hello @qkl871118

Thank you for raising this issue, I appreciate it.

cwltool, the reference implementation of the CWL standards, doesn't currently implement this feature as it doesn't schedule jobs on remote resources, it just runs them locally.

This could be achieved for jobs that are run in a Docker container: https://docs.docker.com/engine/admin/resource_constraints/#memory PRs that achieve this are very welcome.

For local running without Docker containers then that is still possible, but more difficult and less portable.