bonustrack
commented
2 days ago Did more testing and it seem that writing is what's take most time in the writers. If this can be done once per block along with the db update of the last indexed block this would also solve the issue where there can be some duplicate if you restart a Checkpoint instance in middle of a block, currently it's possible that some data get ingested in middle of a block and server restart and reindex it.
Locally running a writer function with Checkpoint can take about 6sec, this must come from the MySQL queries that write, update, delete or read data. We should over time move these queries to happen in same time for multiple event within the same block using a SQL transaction. The most challenging would be to move read queries, we could do write update and delete first.