Closed JosepSampe closed 4 weeks ago
I've found where the ConcurrentAppendException problem is. There is a flag in Delta's AddFile
called dataChange
, which can be true or false.
In the new version, both Table.write()
and Table.optimize()
call RollupDataWriter.write()
, which returns AddFile(data_change=true).
In the old version, Table.write()
called RollupDataWriter.write()
, which returned AddFile(data_change=true)
, while Table.optimize()
called RollupDataWriter.optimize()
, which returned AddFile(data_change=false)
.
Therefore, in the current code, since this flag is always set to true, the exception is triggered.
I have performed several tests, keeping the flags as before: true
for write
and false
for optimize
, and I haven't encountered the exception again.
Since PR Issue #405: DataWriter refactory #402, a ConcurrentAppendException started appearing during a write() and a optimize()