Proposal:
We should mitigate problems caused by low ulimits. Ideas:
Reduce the number of shards by default (so fewer files are created by default)
Add ulimits to /metrics endpoint
Warn in the logs (and UI?) when file ulimits are low
Current behavior:
Users start up a bucket with a 30 day retention policy and get daily shards. Many linux distros set the max ulimit -n to 1024 . 30 days * ~20 files per shard when TSI is enabled creates ~600 files, so with their first bucket users are right up against their ulimits.
Desired behavior:
TBD
Alternatives considered:
Document that users should increase their ulimits as part of setup
Use case:
Many users get inscrutable "too many files" errors in the logs, which causes much frustration and also data issues as new files cannot be written.
lesam
commented
2 years ago
Proposal: We should mitigate problems caused by low ulimits. Ideas:
/metricsendpointCurrent behavior: Users start up a bucket with a 30 day retention policy and get daily shards. Many linux distros set the max
ulimit -nto 1024 . 30 days * ~20 files per shard when TSI is enabled creates ~600 files, so with their first bucket users are right up against their ulimits.Desired behavior: TBD
Alternatives considered: Document that users should increase their ulimits as part of setup
Use case: Many users get inscrutable "too many files" errors in the logs, which causes much frustration and also data issues as new files cannot be written.