Open frizwi opened 5 years ago
audetto
commented
1 year ago I have a very similar issue. I run it inside a container which has very strict memory limits.
In particular the current memory usage is available in a file /sys/fs/cgroup/memory/memory.usage_in_bytes and one can get limit_in_bytes too.
How is cronicle determining this?
Problem is that if it goes above, the container is rebooted quickly. Can we communicate to cronicle how to get a better snapshot of memory usage?
jhuckaby
commented
1 year ago Cronicle tracks memory per job by using /bin/ps -eo "ppid pid %cpu rss" output and building a tree of PID / PPID relationships. All of the job process rss numbers are added up (resident memory). It does not track things like virtual memory.
https://github.com/jhuckaby/Cronicle/blob/master/lib/job.js#L1701-L1830
If you are launching jobs that then spawn their own "detached" processes that don't have a correct PPID (parent PID) set, or if your processes are using virtual rather than resident memory, then it may not track memory correctly.
🤷🏻♂️
audetto
commented
1 year ago Understood thanks.
Summary
Hi, exactly how is the memory usage tracked? Presently its grossly under reporting but that's probably due to the shell script running jobs in the background. Is there a way to JSON output the pid of the actual job so that its more reflective of the actual usage?
In case you're interested, it ends up being a Docker container that's invoked
Thanks! -Farhan
Steps to reproduce the problem
Your Setup
Operating system and version?
Node.js version?
Cronicle software version?
Are you using a multi-server setup, or just a single server?
Are you using the filesystem as back-end storage, or S3/Couchbase?
Can you reproduce the crash consistently?
Log Excerpts