Our internal ingestion product noticed that it takes a significant amount of time for sweep to kick in, since they're doing small, but lots of writes. This lead to the presence of a large row when sweep thought there was nothing to sweep.
The current thresholds for flushing to the KVS the stats about number of writes are:
private static final int WRITE_THRESHOLD = 1 << 16; (number of cells written)
private static final long WRITE_SIZE_THRESHOLD = 1 << 30; (number of bytes)
in SweepStatsKVS.
We should consider lowering them, or making these thresholds configurable.
fsamuel-bs
commented
6 years ago
Our internal ingestion product noticed that it takes a significant amount of time for sweep to kick in, since they're doing small, but lots of writes. This lead to the presence of a large row when sweep thought there was nothing to sweep.
The current thresholds for flushing to the KVS the stats about number of writes are:
in SweepStatsKVS.
We should consider lowering them, or making these thresholds configurable.