deislabs / bindle

Bindle: Object Storage for Collections
Apache License 2.0
263 stars 37 forks source link

fix: address "too many open files" error when pushing large bindles #337

Closed dicej closed 2 years ago

dicej commented 2 years ago

Pushing a bindle that contains many assets using StandaloneRead::push can result in a "too many open files" error on POSIX systems with a default file descriptor limit of 1024. This is due to the unlimited concurrency of future::join_all.

The alternative is to use futures::StreamExt::buffer_unordered instead, which accepts a parameter to limit maximum concurrency.

This fixes https://github.com/deislabs/bindle/issues/336.

Signed-off-by: Joel Dice joel.dice@gmail.com