kvdb: concurrency issue tracker

[guggero]

After merging https://github.com/lightningnetwork/lnd/pull/5484 we'll need to make sure that any DB code is aware of the reset/retry mechanism.

This issue is meant to keep track of all the possible problems that should be addressed to handle DB errors and retries more gracefully.

• [ ] failIdx in batch/batch.go (https://github.com/lightningnetwork/lnd/pull/5484#discussion_r674900458)
• [ ] assert that the retry mechanism is working in all cases by artificially running the code in the callback twice (comment 1, https://github.com/lightningnetwork/lnd/issues/5557#issuecomment-902724058)
• [ ] assert that query functions with a callback can handle duplicates if the read transaction is retried (comment 2, https://github.com/lightningnetwork/lnd/issues/5557#issuecomment-902932810)

[joostjager]

Linking https://github.com/lightningnetwork/lnd/pull/5643#discussion_r692703535

"I had an idea that could help with that: modify the code to artificially always run the code inside each transaction twice. It is then as if everything needs a retry. Itests, bench, etc, should still work."

[joostjager]

The other issue that came up in #5643 is that callers of query functions with a callback such as ChannelGraph.ForEachChannel (and probably several others) should be prepared to receive duplicates if the read transaction is retried.

Linking https://github.com/lightningnetwork/lnd/pull/5484#discussion_r682943762 about retries of a read-only tx.