search

OPHoperHPO / image-background-remove-tool

✂️ Automated high-quality background removal framework for an image using neural networks. ✂️
https://carve.photos
Apache License 2.0
1.39k stars 272 forks source link

API Concurrent processing with CPU & CUDA #121

Closed mariuszbeltowski closed 1 year ago

mariuszbeltowski commented 1 year ago

Hey @OPHoperHPO - Great work! I have one question. How I can solve the issue with slow concurrent requests processing via API? If you make 5 requests in a second, every next request is slower than previous: Screenshot 2023-01-17 at 18 24 42

I am using EC2 p3.2xlarge - Tesla V100, 8 vCPU, GPU Mem 16GB, CPU Mem 61GB The CPU usage is below 20%. GPU usage is below 30%. RAM usage is below 5% for the whole time.

OPHoperHPO commented 1 year ago

Hi @mariuszbeltowski

The server processes 1 image per handler call, so it is slower than if it were to process those images through a different interface. The CarveKit web API has a simple task queue. This queue blocks all images passed through the API from being processed at the same time, as they can cause some random instability in the API due to unregulated resource consumption. You need to rewrite it into a task queue that submits for processing a whole batch of images that have accumulated for processing, or use an external task queue system such as RabbitMQ, Celery, etc. You can also remove it, but this may crash the API due to the simultaneous processing of many images

mariuszbeltowski commented 1 year ago

Thank you @OPHoperHPO . AWS machines with GPU are quite big, having only 1 image processing with them is not cost effective. Would be great if we could control the max amount of concurrent images being processed with ENV variable. Or at least enable/disable queue single image blocking.