Threads used for CPU bound tasks (when not using Rayon)
This allows us to control the resource usage of the server and helps to
avoid exceeding open file limits, overloading upstream object stores,
and running out of memory. Currently these limits must be specified
manually.
The memory limit is fairly rough, and currently only accounts for the
size of the data returned from the S3 object
store. It does not account for decompressed data which may be larger
than the compressed size, nor does it account for operations such as
deshuffling which may require a second buffer of the same size.
Adds CLI arguments to specify limits for:
This allows us to control the resource usage of the server and helps to avoid exceeding open file limits, overloading upstream object stores, and running out of memory. Currently these limits must be specified manually.
The memory limit is fairly rough, and currently only accounts for the size of the data returned from the S3 object store. It does not account for decompressed data which may be larger than the compressed size, nor does it account for operations such as deshuffling which may require a second buffer of the same size.