Assessing the risk of duplicated entries for BEST_EFFORT reliabilityLevel

[jcblancomartinez]

Hi,

From here:

Implements the BEST_EFFORT write strategy for Single Instance. All executors insert into a user specified table directly. Write to table is not transactional and may results in duplicates in executor restart scenarios.

I'm not sure I understand when we could end up with duplicates when using BEST_EFFORT reliabilityLevel.

For each Spark's dataframe partition we call savePartition and savePartition only commits the transaction after SQLServerBulkCopy.writeToServer has succeded.

• Are we worried that executor restarts after committing the transaction?
• In which cases could that happen? When an SQLException gets thrown when committing the transation? Are we thinking of a connection reset that causes Spark to handle an SQLException even though the transaction has been committed in Sql Server? Are there other situations where this could happen?

Thanks.

[jcblancomartinez]

@shivsood @luxu1-ms could you please help here?

Thanks.