Allowing empty segments with no offset advancing

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore

https://pinot.apache.org/

Apache License 2.0

5.26k stars 1.23k forks source link

Allowing empty segments with no offset advancing #12703

Open jadami10 opened 3 months ago

jadami10 commented 3 months ago

This is a followup from https://github.com/apache/pinot/issues/8929, but in the case of 0 data being consumed. We've since found a poor interaction between Pinot and our s3 lifecycling.

we have a few partitions that have 0 data all the time (not filtered out, literally 0 events)
Pinot keeps that consuming segment open indefinitely (in this case we noticed it had been 1 year)
Pinot also keeps the last completed segment for each partition (in this case, it's from 2023)
We wanted to use s3 lifecycling to delete all data < 30 days old (our tables had 10 day retention) rather than just rely on Pinot's mechanisms, but this sent segments into error state since they had been around for 1 year

Is there any reason we can't seal a segment where the offset hasn't advanced? In this case, we would have had N segments for this partition all with 0 records and the same start/end offset.

cc @Jackie-Jiang @priyen-stripe

Jackie-Jiang commented 3 months ago

I think we don't seal them right now because we don't support empty segment before. Since we can support empty segment now, we should be able to seal them. We want to revisit the timestamp used for empty segment (using current time should work) so that retention manager can remove them properly.

tibrewalpratik17 commented 4 weeks ago

Hey @Jackie-Jiang any updates on this?

Is this as straight-forward as removing these lines? https://github.com/apache/pinot/blob/0f92742b45aa9ac57439b68a63e4a9ef8d9d9d8e/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java#L334-L338

Saw this in one of our tables today. The table existed for more than a year and we enabled snapshot on it recently. Now since the last consuming segment (which was like months old) never received any message, the snapshot flow didn't run and that entire partition was knocked out of upsert-compaction flow. [Update] I have tried to fix this issue for partial-upsert tables here - https://github.com/apache/pinot/pull/13285#discussion_r1631296201

Jackie-Jiang commented 3 weeks ago

Is this as straight-forward as removing these lines?

I think so.

The drawback is that we might end up with a lot of segments that are all empty, but I guess we can make retention manager keep cleaning up empty segments (it might already do) as long as they are not the last completed segment.