Example of batch/throttle of large lists of inputs

[julien-c]

This is for a customer to the paid version of the Inference API, who is looking to send bulk volumes of sentences (e.g 1,000), in short spikes.

The API works with batches out of the box but obviously not batches of arbitrarily large size :)

We also don't want them to send all requests in parallel, so this is an example of how they can sequentially batch their documents

[julien-c]

we don't necessarily have to merge this if it's too localized