search

broadinstitute / viral-ngs

Viral genomics analysis pipelines
Other
190 stars 67 forks source link

When determining available memory and cores, take into account cgroup limits #905

Closed notestaff closed 5 years ago

notestaff commented 5 years ago

When determining available memory and cores, take into account cgroup limits, which are used e.g. by docker run's --cpus and --memory options. Currently we use memory and cores values for the host machine, not the container.

dpark01 commented 5 years ago

This is so much better, and thanks for deleting the hacky one-off scripts that kept reproducing themselves.

BTW, it's much more correct to compute for container shape than host shape, and we should definitely merge this in. But out of curiosity, what's the use case you're addressing, kubernetes clusters? (most dnax and papi use cases have container shape = host shape)

notestaff commented 5 years ago

The use case is running large benchmarks on a single high-end instance with Cromwell's Local backend. E.g. running 200 rabies assemble_denovos this way costs under $8, vs. $238 on DNAnexus.