marieai / marie-ai

Integrate AI-powered Document Analysis Pipelines
MIT License
57 stars 3 forks source link

Slow memory leak #74

Open gregbugaj opened 1 year ago

gregbugaj commented 1 year ago

Inspecting sudo dmesg | grep 'oom-kill' we find that the process has a memory leak.

[595453.487193] ERAAgent invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[595453.487457] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=eraagent.service,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-ceaea40258ae16cdce82902830385c73299ca6a69cec4d88fb2bf69b4891f3b9.scope,task=marie,pid=2740,uid=0
[2237411.969010] marie invoked oom-killer: gfp_mask=0x1100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
[2237411.969250] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=docker-ba84390d0c740112e46190cddd446b8e9c051053e94feb3774fd7a4d976cdca8.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-ba84390d0c740112e46190cddd446b8e9c051053e94feb3774fd7a4d976cdca8.scope,task=marie,pid=1540445,uid=0
[2245192.069801] marie invoked oom-killer: gfp_mask=0x1100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
[2245192.069980] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=docker-314aaae63582f70dd22b5a5095193ad51b601f31d0fc495c0268accca139419f.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-314aaae63582f70dd22b5a5095193ad51b601f31d0fc495c0268accca139419f.scope,task=marie,pid=3650725,uid=0

This could be replicated by sending large amount of volume to the service to trigger out-of-memory Will use memray to track this down.