It looks like batching was added in #437 - thank you for implementing this, it's very helpful.
I notice that batching, as defined here, depends on a fixed batch size. This can be suboptimal for clients submitting a large number of smaller documents, as we cannot configure the ThreadPoolExecutor size to parallelize a large number of small data payloads. As a result a client might end up blocking while waiting for small network responses.
Would it be possible to allow clients to configure either the ThreadPoolExecutor size or the embed_batch_size setting when calling embed?
It looks like batching was added in #437 - thank you for implementing this, it's very helpful.
I notice that batching, as defined here, depends on a fixed batch size. This can be suboptimal for clients submitting a large number of smaller documents, as we cannot configure the ThreadPoolExecutor size to parallelize a large number of small data payloads. As a result a client might end up blocking while waiting for small network responses.
Would it be possible to allow clients to configure either the
ThreadPoolExecutor
size or theembed_batch_size
setting when callingembed
?