search

codeproject / CodeProject.AI-Server

CodeProject.AI Server is a self contained service that software developers can include in, and distribute with, their applications in order to augment their apps with the power of AI.
Other
565 stars 135 forks source link

What is the load balancing strategy? #73

Closed laurentopia closed 8 months ago

laurentopia commented 8 months ago

As a test I spun up a few object detection and it seems that CP-AI uses all of them regardless of past efficiency, meaning that yolo-cpu will be used even though yolo-GPU is available (and 3x faster). Is this a simple idle-first queue? Random? image Here we should see 3x more GPU yolo than CPU if scheduling prioritized speed.

ChrisMaunder commented 8 months ago

Each module polls the server to see if there are jobs available. It's first in, best dressed, since the server isn't pushing, the modules are polling / pulling.

laurentopia commented 8 months ago

Ok that explains what I'm seeing. I guess that's fine when all devices have roughly the same speed. What happens in the case with multiple Coral TPU? They're cheap again on Mouser...