Key duplication occurs when embulk-output-mysql retries - Githubissues

embulk / embulk-output-jdbc

MySQL, PostgreSQL, Redshift and generic JDBC output plugins for Embulk

Other

88 stars 86 forks source link

Key duplication occurs when embulk-output-mysql retries #252

Closed hito4t closed 5 years ago

hito4t commented 5 years ago

embulk-output-mysql will retry when deadlock occurs. But sometimes key duplication error will occur. If number of records to load is large, some records will be committed even if deadlock occurs. So key duplication will occur when embulk-output-mysql retries to load all records.

This is the bug of the PR #251 .

hito4t commented 5 years ago

Values of Statement#executeBatch when succeeded :

number of records to load : 4
length of value : 4
count of Statement.SUCCESS_NO_INFO : 4

Values of BatchUpdateException#getUpdateCounts when deadlock occurs :

number of records to load : 4
length of value : 3 (deadlock occurred at the 3rd record)
count of Statement.EXECUTE_FAILED : 3
count of committed records : 0

Values of Statement#executeBatch when succeeded :

number of records to load : 4
length of value : 4
count of Statement.SUCCESS_NO_INFO : 4

Values of BatchUpdateException#getUpdateCounts when deadlock occurs (many records) :

number of records to load : 250,000
length of value : 249,999
count of 0 : 174,760
count of Statement.EXECUTE_FAILED : 75,239
count of committed records : 174,760