Deadlock found when trying to get lock; try restarting transaction

[dinithihj]

Some instances (so far 1 instance from each environment) were suspended after getting the following error. This occurred in the dev.app, test.app, and test.mod environments.

Process Instances

• https://dev.app.spiff.status.im/i/3498
• https://test.app.spiff.status.im/i/19347
• https://test.mod.spiff.status.im/i/8893

When checking the events tab, noticed that the error occurred after the "Set Xero Currencies Enum List"

FYI: with_hint(MyModel, 'WITH (nolock)')

[madhurrya]

Noticed the same in test.mod. https://test.mod.spiff.status.im/i/866

[burnettk]

Maybe just stop doing the fancy “on duplicate key update” stuff: https://medium.com/@mingwho/mysql-deadlock-caused-by-insert-on-duplicate-key-update-893ef30993a8. We can probably just rescue the duplicate key in code and ignore it, since if it is already written, we’re good (everyone is trying to insert the exact same row in this case, since it is just data and its corresponding hash).

[madhurrya]

Noticing this error time to time

[sashayar13]

Due to this issue, five process instances were errored in prod yesterday. I was able to restore four of them because they errored at the beginning of the process (no task data was captured), but not the one that errored closer to the end of the process.

[dinithihj]

Noticed deadlock errors in 2 instances on the test.mod(version 64/66) while release testing.

• https://test.mod.spiff.status.im/i/6496
• https://test.mod.spiff.status.im/i/6501