Endpoint for batch tokenization in core

[flvndvd]

We should have a batch end point for tokenization that we can use here:

https://github.com/dust-tt/dust/blob/656576bff4220b84a68099c5e926ec42949c83e6/front/lib/api/assistant/generation.ts#L172-L194

[Duncid]

@flvndvd why is this in Product Framing?

[flvndvd]

Moved to Olympus!

[fontanierh]

So after digging a bit, all our tokenizers are singletons behind a RW lock. So unless we accept to change that, we wouldn't get much from a batch endpoint since everything will continue to be sequential.

@spolu Is there a strong reason to have a lock on the tokenizers ?

[spolu]

The RW lock is used in read mode for tokenization and therefore allows parallelization within core, additionally this would save the N connections + Promise.all which would overall be much more efficient :+1:

[fontanierh]

Thanks @spolu