search

OpenNeuroOrg / openneuro

A free and open platform for analyzing and sharing neuroimaging data
https://openneuro.org/
MIT License
114 stars 40 forks source link

Jobs don't seem to isolate memory (RAM) #608

Closed chrisgorgo closed 6 years ago

chrisgorgo commented 6 years ago

An app definition includes requirement for 15Gb of RAM, but from the point of view of the app all memory seems to be accessible:

2018-04-30 02:28:42,961 workflow:INFO [MultiProc] Running 1 tasks, and 0 jobs ready. Free memory (GB): 26.75/26.95, Free processors: 3/4.

nellh commented 6 years ago

This is a limitation of Docker, the memory limit is available inside a container at /sys/fs/cgroup/memory/memory.limit_in_bytes but cgroups do not isolate /proc/meminfo / sysinfo() calls / etc.

nellh commented 6 years ago

Here's a good post that covers this in detail. As far as I know the userland interface for this doesn't exist in a general way still, so I'm not sure what the best workaround for bids-apps would be other than reading the limits from the cgroup values.

https://fabiokung.com/2014/03/13/memory-inside-linux-containers/