Open otherhy opened 1 month ago
If my logic is correct, then I believe this issue is caused by the PartitionExpire instance not being reused. It can be resolved by recording the instance object in the upper layer AbstractFileStoreTable and reusing this instance object each time a TableCommitImpl instance is created.
I look forward to your response and feedback.
in a stream env, StoreCommitter holds TableCommitImpl, so expiration of partition works as expected。But in a batch env, if you create TableCommitImpl every time, sure there will be a new PartitionExpire every time and PartitionExpire#expire() will not be executed。but in both stream env and batch env, paimon uses TableCommitImpl to commit changes, there's no other way to differ it's a batch env or stream env in TableCommitImpl. But it does have a isStreamingMode
which is used in table.newWrite() but not used in table.newCommit(), see withExecutionMode
# BatchWriteBuilderImpl
@Override
public BatchTableWrite newWrite() {
return table.newWrite(commitUser)
.withIgnorePreviousFiles(staticPartition != null)
.withExecutionMode(false);
}
@Override
public BatchTableCommit newCommit() {
InnerTableCommit commit = table.newCommit(commitUser).withOverwrite(staticPartition);
commit.ignoreEmptyCommit(true);
return commit;
}
maybe isStreamingMode
should be set when create a TableCommitImpl so a TableCommitImpl can differ stream and batch env and do diffrent things when expire a partition.
Overall, Paimon is a stream-first computing engine, and some designs do not take batch processing into consideration. or maybe they want you to use a seperate Partition Expiration to doExpire. just like what they do in PartitionExpireTest, to new a Expire to do Expiration. And maybe there should be a PartitionExpirationProcedure for spark engine too.
Just need add parameter isEndInput
for committer to distinguish batch or streaming mode.
Search before asking
Paimon version
0.8
Compute Engine
JavaAPI
Minimal reproduce step
According to the examples in the documentation, this is how we perform the commit. Taking BatchWrite as an example, after writing the data, we need to create a new instance of the BatchTableCommit class each time to execute the commit.
Based on the following call chain, we can infer that each time a BatchTableCommit instance is created, a TableCommitImpl instance is ultimately created within it. In its constructor, a PartitionExpire instance is passed as a parameter.
Based on the following code, we can infer that each time a TableCommitImpl instance is created, the org.apache.paimon.AbstractFileStore#newPartitionExpire method is called to create a new PartitionExpire instance.
Based on the constructor of the PartitionExpire class, we can infer that when the instance is initialized, lastCheck is set to the current time.
After BatchTableCommit is created, based on the example, we immediately start the commit. When the commit is completed, the org.apache.paimon.operation.PartitionExpire#expire(long) method of the PartitionExpire instance is called, as shown in the following code, to check for partition expiration.
But at this time, lastCheck is set to now because it was just initialized. Using the default value checkInterval=1h as an example, lastCheck.plus(checkInterval) would be one hour later. Therefore, now.isAfter(lastCheck.plus(checkInterval)) always results in false, causing the partition expiration to be skipped. And because the BatchTableCommit can only perform a single commit, the next time we execute a commit, we will use a brand new PartitionExpire instance. This causes our commits to always fail to trigger the partition expiration check.
Please help me check if my logic is correct or if there is an issue with my usage.
What doesn't meet your expectations?
The partition expiration parameters set on the table did not take effect because they were not correctly triggered during the commit.
Anything else?
No response
Are you willing to submit a PR?