search

logv / sybil

columnar storage + NoSQL OLAP engine | https://logv.org
https://logv.org
Other
305 stars 26 forks source link

Is digest invoked automatically at some point? #117

Closed wang502 closed 4 years ago

wang502 commented 4 years ago

In this intro, it says:

If more than a certain amount of data is in the ingestion log, a digest is initiated

Does this mean digestion will be invoked after amount of data in ingest log reach certain threshold?

okayzed commented 4 years ago

Yes, that's correct.

The threshold is either 256 files sitting in the ingest log (1 ingest = 1 file) or 2MB of data.

The function is ShouldCompactRowStore in table_ingest.go: https://github.com/logv/sybil/blob/master/src/lib/table_ingest.go#L82

You can also manually force a digest if you want.