Make the backend scale in K again - Githubissues

rom1504 / clip-retrieval

Easily compute clip embeddings and build a clip retrieval system with them

https://rom1504.github.io/clip-retrieval/

MIT License

2.41k stars 210 forks source link

Make the backend scale in K again #160

Open rom1504 opened 2 years ago

rom1504 commented 2 years ago

make dedup faster using a smarter hash
Reorder metadata per cluster
Make safety filtering faster / do things by batch
Implement a queue system

rom1504 commented 1 year ago

implementation of batching.

choices:

let the user provide batches
have a batch function, eg every item is sent independently and received independently by the user, but internally a combination of timeout; batch sizes and thread implement batching

implementation of batching itself:

clip model should support batching
safety and aesthetic model as well
dedup should support batching

how to test thing:

implement tests for batching for every individual component
implement test for batching for the whole flow
implement test for the batch function if implemented

metrics to check:

speed
memory usage