Implements the BEST_EFFORT write strategy for Single Instance. All executors insert into a user specified table directly. Write to table is not transactional and may results in duplicates in executor restart scenarios.
I'm not sure I understand when we could end up with duplicates when using BEST_EFFORT reliabilityLevel.
In which cases could that happen? When an SQLException gets thrown when committing the transation? Are we thinking of a connection reset that causes Spark to handle an SQLException even though the transaction has been committed in Sql Server? Are there other situations where this could happen?
Hi,
From here:
I'm not sure I understand when we could end up with duplicates when using BEST_EFFORT reliabilityLevel.
For each Spark's dataframe partition we call savePartition and savePartition only commits the transaction after SQLServerBulkCopy.writeToServer has succeded.
Thanks.