search

codeproject / CodeProject.AI-Server

CodeProject.AI Server is a self contained service that software developers can include in, and distribute with, their applications in order to augment their apps with the power of AI.
https://codeproject.github.io/codeproject.ai
Other
732 stars 160 forks source link

Out of memory crash #94

Closed tbclark3 closed 7 months ago

tbclark3 commented 8 months ago

I just installed code project ai-server under Docker and I'm using it with Frigate and Double-Take. I am getting frequent crashes with an "out of memory" error. In fact, it crashes on almost every detection. (See logs below) The Docker memory is unlimited, and the system had 182 GB available at the time of the most recent crash. I don't think it's out of memory. However, I wonder if it's because I am sending as many as 15 high resolution images at a delay of 0.1 seconds each time Frigate detects a person.

Trace Client request 'recognize' in queue 'faceprocessing_queue' (#reqid 28f7694a-13b4-4aba-8b99-08f4085950ef) Trace Request 'recognize' dequeued from 'faceprocessing_queue' (#reqid 28f7694a-13b4-4aba-8b99-08f4085950ef) Debug Face Processing: Retrieved faceprocessing_queue command 'recognize' Infor Response rec'd from Face Processing command 'recognize' (#reqid 28f7694a-13b4-4aba-8b99-08f4085950ef) ['A face was recognised'] took 142ms Trace Client request 'recognize' in queue 'faceprocessing_queue' (#reqid 644bb80d-c1f9-4f06-ba79-07a47cf652a8) Trace Request 'recognize' dequeued from 'faceprocessing_queue' (#reqid 644bb80d-c1f9-4f06-ba79-07a47cf652a8) Debug Face Processing: Retrieved faceprocessing_queue command 'recognize' Infor Response rec'd from Face Processing command 'recognize' (#reqid 644bb80d-c1f9-4f06-ba79-07a47cf652a8) ['A face was recognised'] took 120ms Trace Client request 'recognize' in queue 'faceprocessing_queue' (#reqid 4a9cde28-b895-4410-b81b-192bf7dd1f8e) Trace Request 'recognize' dequeued from 'faceprocessing_queue' (#reqid 4a9cde28-b895-4410-b81b-192bf7dd1f8e) Debug Face Processing: Retrieved faceprocessing_queue command 'recognize' Infor Response rec'd from Face Processing command 'recognize' (#reqid 4a9cde28-b895-4410-b81b-192bf7dd1f8e) ['A face was recognised'] took 153ms Out of memory.

ChrisMaunder commented 7 months ago

Sorry but by the very nature of AI inferencing and Python, memory use is high.

Try shutting down unneeded modules, using smaller models, or increasing available memory