Prior to this commit, Shovel would open a PG transaction for each call to Converge. Converge queries for the local and remote latest and decides what is next to index. Shovel then proceeds to download the remote blockchain data and saves the indexed results to various PG tables -- all within the same PG transaction.
The problem with this approach is that a PG transaction is open while downloading blockchain data. If downloading is fast this is no big deal. However, if downloading is slow (ie lots of trace data, or any other reason) then the transaction is open for a while and because of MVCC overall PG performance can degrade. It is generally best practice to minimize the use PG transactions for a minimal amount of time.
This commit splits up the loading from the inserting. This was mostly straightforward except for dealing with reorgs. Without reorgs, Converge would not need to query the local state using a PG TX. We use a PG TX for querying the latest local state and reorg pruning because Shovel will automatically prune (ie delete) data during a reorg. We want to have a limit on how much data we are willing to delete in case there is a bug that causes Shovel to delete its entire database (bad).
The solution is to use (possibly) 2 PG TXs for converge. The 1st transaction is optional. Specifically: we query the local state and then download the next set of data to index. If we detect a reorg, we will open a PG TX and prune and try again. If we succeed in pruning we commit the transaction. If we exhaust the reorg limit we rollback the transaction (therefore not deleting any data).
The 2nd transaction happens after the download step. We use a transaction to insert the indexed data and to update the shovel.task_updates table to indicate that we have successfully indexed a particular block(s).
Prior to this commit, Shovel would open a PG transaction for each call to Converge. Converge queries for the local and remote latest and decides what is next to index. Shovel then proceeds to download the remote blockchain data and saves the indexed results to various PG tables -- all within the same PG transaction.
The problem with this approach is that a PG transaction is open while downloading blockchain data. If downloading is fast this is no big deal. However, if downloading is slow (ie lots of trace data, or any other reason) then the transaction is open for a while and because of MVCC overall PG performance can degrade. It is generally best practice to minimize the use PG transactions for a minimal amount of time.
This commit splits up the loading from the inserting. This was mostly straightforward except for dealing with reorgs. Without reorgs, Converge would not need to query the local state using a PG TX. We use a PG TX for querying the latest local state and reorg pruning because Shovel will automatically prune (ie delete) data during a reorg. We want to have a limit on how much data we are willing to delete in case there is a bug that causes Shovel to delete its entire database (bad).
The solution is to use (possibly) 2 PG TXs for converge. The 1st transaction is optional. Specifically: we query the local state and then download the next set of data to index. If we detect a reorg, we will open a PG TX and prune and try again. If we succeed in pruning we commit the transaction. If we exhaust the reorg limit we rollback the transaction (therefore not deleting any data).
The 2nd transaction happens after the download step. We use a transaction to insert the indexed data and to update the shovel.task_updates table to indicate that we have successfully indexed a particular block(s).