search

jina-ai / clip-as-service

🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP
https://clip-as-service.jina.ai
Other
12.34k stars 2.06k forks source link

bert-as-a-service memory consumption #412

Open boxabirds opened 5 years ago

boxabirds commented 5 years ago

Hi all I want to profile the memory usage of bert-as-a-service so I can do some budget calculations. A single worker on my Mac when starting up seemed to take upwards of peak 20GB and stable of 6GB.

Has anyone done any more exhaustive analysis on the memory footprint of this?

More concretely, when people have deployed this to say, AWS, what sized EC2 instance worked well?

thanks in advance!

kevin-rowney commented 5 years ago

I had trouble getting a stable running system with RAM as high as 26GB on a default instance out on GCP. Once I upgraded to 52GB, it seemed to run fine.

boxabirds commented 5 years ago

I had trouble getting a stable running system with RAM as high as 26GB on a default instance out on GCP. Once I upgraded to 52GB, it seemed to run fine.

@kevin-rowney thank you! Is this an inference-only setup or was it also doing fine-tuning? I'm thinking it makes sense to separate inference-only as it seems at least on my local machine to work happily in "only a few GB") with any additional fine-tuning.

Have you had much experience with resource consumption with multiple workers?

bigrig2212 commented 4 years ago

I was having a heck of a time getting this working on GCP. Tried every option they've got. Was also working fine on my local machine (32g ram, 3.5 GHz Intel Core i5, 4 cores). Thanks to the suggestion above, it seems to be working reliably now:

GCP: Compute Engine: VM n1-highmem-8 (8 vCPUs, 52 GB memory)

Wish I could make it work in the various container services... but no luck there. Can't remember if i tried exactly the same config there though... will need to give that a try. Don't think i went up as high as 52gb.