search

dceejay / tfjs-coco-ssd

Node-RED node for tensorflowjs coco ssd
Apache License 2.0
14 stars 10 forks source link

Memory leak. When restarting flow memory is not released - revisited #14

Open krambriw opened 3 years ago

krambriw commented 3 years ago

Unfortunately, it did not help

I put 12 copies of the node on a flow (nothing else). After each flow restart, memory consumption increases with around 5% on a RPi4 w 4G. When memory consumption increases it takes longer and longer for the models to be loaded into all 12 nodes. When it reaches around 69%, it is not able to load all nodes with the models. After another flow restart, NR is no longer able to load the model into any of the 12 nodes. Node is now occupying 72% of the memory

On a RPi3 it has difficulties to load the models in all 12 nodes already from start, it takes long time, I think memory is not enough so continuing the test on a RPi3 is meaningless

dceejay commented 3 years ago

Well yes 12 x 5% = 60% plus the base Node-RED memory use I can well believe it gets up to that. I don't know how the tensorflowjs is handling the models directly - I do try to release them - but maybe some link to underlying native process isn't. I know there are newer releases of tensorflowjs that may address this (or may not) - but at the time I looked they didn't have a version that "just worked" on Pi and was a painful manual install. I guess that may also have been fixed but I haven't had a chance to look. Also the instances don't share a model - so loading 12 will be loading 12 large models... if you can somehow re-use the one instance it would probably be better.

Sadly I don't currently have any spare cycles to investigate this right now... maybe over Xmas... or maybe not... so will need to rely on the community to have a look and see what can be found.

krambriw commented 3 years ago

OK understood, it is working fine from functional point, that is not the issue. The issue is only if you restart the flow, memory is not released. After "several" flow restarts you will face a problem, depending on how much total ram is available. A complete restart of NR releases the memory. Could it be the binaries are not released?? Full respect regarding your workload, there is no panic, we just have to be aware that this is the situation and the workaround is to restart NR