Currently in AbstractFileStoreWrite, we use writerContainer.lastModifiedCommitIdentifier <= latestCommittedIdentifier to check that there is no more record waiting to be committed for this writer.
However, each commit identifier may have multiple snapshots (for example, one APPEND snapshot and one COMPACT snapshot). If the commit is slow, writer might be closed and re-opened between the APPEND snapshot and the COMPACT snapshot, thus causing commit conflicts in the future.
To ensure that all snapshots of this identifier are committed, the correct condition should be writerContainer.lastModifiedCommitIdentifier < latestCommittedIdentifier.
Purpose
Currently in
AbstractFileStoreWrite, we usewriterContainer.lastModifiedCommitIdentifier <= latestCommittedIdentifierto check that there is no more record waiting to be committed for this writer.However, each commit identifier may have multiple snapshots (for example, one APPEND snapshot and one COMPACT snapshot). If the commit is slow, writer might be closed and re-opened between the APPEND snapshot and the COMPACT snapshot, thus causing commit conflicts in the future.
To ensure that all snapshots of this identifier are committed, the correct condition should be
writerContainer.lastModifiedCommitIdentifier < latestCommittedIdentifier.Tests
TableWriteTest#testWaitAllSnapshotsOfSpecificIdentifierAPI and Format
Not affected.
Documentation
No new feature.