Server processes concurrent txns against stale commit/t values

[aaj3f]

Description

I made a load test script to send concurrent txns to Nexus & server to test under load scenarios. I found that when sending concurrent txns to the same ledger (either on server or on Nexus's use of server) one of the concurrent txns is accepted, and the rest are rejected for attempting to update the ledger at a stale commit/t value:

db_name: fl1
response: {"ledger":"fl1","commit":"fluree:file://fl1/main/commit/36527581f5b0f22e4db6a979b5d7558f96672c55fe529ea56d49421877efdd87.json","t":245,"tx-id":"65dfa107f187141c35b8ff94e80e1e6d6117b5198f44f5ab698fd7a863fd6b53"}

db_name: fl1
response: {"error":"db/invalid-commit","message":"Commit failed, latest committed db is 245 and you are trying to commit at db at t value of: 245. These should be one apart. Likely db was updated by another user or process."}

db_name: fl1
response: {"error":"db/invalid-commit","message":"Commit failed, latest committed db is 245 and you are trying to commit at db at t value of: 245. These should be one apart. Likely db was updated by another user or process."}

request_count: 0
request_number: 27
Elapsed time: 1.065494708s

Reproduction Steps

You can try this with the load test app here: https://github.com/fluree/nexus-load-script

Use the following .env and run the app per README instructions (e.g. w/ cargo run)

# This should be a comma-separated list of the db names, e.g. fluree-jld/387028092977707,fluree-jld/387028092977714
db_names=fl1

# This should be a comma-separated list of API Keys (where each corresponds to the db name at the same position in the list above), e.g. TvbMo...5dXUQ,mhXNe...JA9RA
authorization_keys=

# This should be the number of total requests to be made (if 0, then the script will run indefinitely)
request_number=27

# This should be the number of threads to use to concurrently run the total number of requests
concurrency_num=3

# This should be either "query" or "transaction"
request_type=transaction

# This should be a value, in milliseconds, for an optional delay between successive requests
request_interval=0

# This should be either 'prod', 'staging', or a valid self-hosted Fluree URL (e.g. 'http://localhost:58090/fluree/')
nexus_target=http://localhost:58090/fluree/

It assumes that server is running on localhost:58090 and that you've created a ledger called fl1

It will send 27 requests over 3 threads, reclaiming the threads after each request, so you should see 9 log outputs. In each stdout report, you'll see that one of the threads saw a successful transaction and the other two failed with the error above

[cap10morgan]

Fix PR: https://github.com/fluree/server/pull/22