search

foxglove / mcap

MCAP is a modular, performant, and serialization-agnostic container file format, useful for pub/sub and robotics applications.
https://mcap.dev
MIT License
523 stars 96 forks source link

CLI: doctor warning on large chunks #731

Open wkalt opened 1 year ago

wkalt commented 1 year ago

It would be useful if the doctor subcommand printed a warning on large chunks. Going from memory, I think we have used 4MB, 8MB, and 10MB as chunk sizes in various contexts. The warning should be well above those numbers since single oversized messages can create oversized chunks.

50MB would probably be a sufficiently large threshold to trigger primarily in cases where mis-configuration is to blame.

james-rms commented 1 year ago

People can have much larger messages than 50MB, i think it makes sense to only warn if the chunk size gets to some size too big to allocate a single buffer for on certain platforms, say 2**30.