add retry for db transactions

[jrwbabylonlab]

Our service utilise mongodb transaction in a few places. For example https://github.com/babylonchain/staking-api-service/blob/main/internal/db/stats.go#L70

The issue with transaction is that it could quite easily having write conflict during huge load on updating the same field in a collection. We currently not retrying if this happens which leads to the failed event being push to the delayed queue for processing later. A quick small improvement we can do to help the system being more effective is to add a exponential backoff retry on all db transactions and make sure the num of retry is controlled by a config value (3 is a good starting number).

In the babylon-api we already doing it, just not exponentially. So maybe you can use some of the code from there, but just make sure we make it exponentially this time. https://github.com/babylonchain/babylon-api/blob/dev/internal/store/dbclient/common.go#L22

[jeremy-babylonlabs]

Proposed solution Implement a generic function transactionWithRetries that wraps mongoDB tx method. Allow a configurable backoff setting and specific error type that wil trigger retry, such as write conflict or connectivity issue.

Implementation function signature

func (db *Database) txWithRetries(ctx context.Context, txnFunc func(sessCtx mongo.SessionContext) (interface{}, error)) (interface{}, error)

input params

• ctx: The context for managing cancellation and timeouts.
• txnFunc: The tx function to execute.

function detail

1. move the execute transaction function into txWithRetries
2. set default params in txWithRetries
   1. DefaultMaxAttempts 3 times
   2. DefaultInitialBackoff lets say 100ms
   3. DefaultBackoffFactor maybe set to 2.0 for now
3. implement retry with a for loop and use DefaultBackoffFactor to calculate
4. retry based on error from the txError during with tx, the should retry funcion will accept the error and return true / false accordingly (https://pkg.go.dev/go.mongodb.org/mongo-driver/mongo#hdr-Error_Handling)

unit test

1. test exponential backoff logic
2. test max retry
3. test retryable error
4. test non retryable error