Worker memory determination relies on psutils, may fail in certain docker configurations

[tazlin]

Notably, Docker resource limits are only available by cgroups. The worker's reliance of psutil apparently gives the same/similar results as running free -m would, rather than the Docker-configured memory limits, if they exist.

• It should be possible to incorporate some shell calls into the docker entrypoint.sh to dump/store the system memory statistics and use those values instead.

[CIB]

Since comfyui is using psutil as well, this issue is relevant for us:

https://github.com/giampaolo/psutil/issues/490

Unfortunately the system (free -m) reports memory usage inside of docker that includes processes outside the current container. I verified it by using ps (which will only list processes inside the same container) and summing the memory usage.

# ps aux --sort=-%mem | awk '{sum += $6} END {print sum / 1024 " MB"}' && echo "" && free -m
85884.6 MB

               total        used        free      shared  buff/cache   available
Mem:          128722       94672        1079         318       32970       32512
Swap:              0           0           0

More evidence this is how it works: https://stackoverflow.com/questions/74052985/how-to-get-containter-memory-limit-in-python https://forums.docker.com/t/how-to-limit-the-cpu-and-memory-of-a-docker-container/108417

So to get the currently used memory inside docker:

ps aux --sort=-%mem | awk '{sum += $6} END {print sum / 1024 " MB"}'

To get the total available memory inside docker:

cat /sys/fs/cgroup/memory.max