Breakout Incoming Transactions into its own Unprocessed Transactions

[achiurizo]

Following up a discussion with @InoMurko , we should consider breaking up our API/Transaction acceptance into two parts:

• Incoming transactions are validated by the protocol, and inserted into a separate table/database. API can return a "ACCEPTED"/202 response. Client needs to ask again to find out if it's been successful.
• Secondary process goes through that table, and does stateful checks. This will flip used outputs as spent or whatever and then fully insert this Transaction into the main Transaction table, awaiting to be submitted as a block

[boolafish]

What is the problem we are trying to solve? We don’t have enough performance to do the stateful check?

[InoMurko]

the idea with v2 API would be to support async transaction processing. where each transaction submitted to childchain is first put into a simple queue, the queue is then processed one by one. this would give a perceived higher TPS.

You could also expand on that and take reasonable chunks of transactions in the queue and process them in bulk, that would give an actual increase of TPS (because you don't have to acquire locks for every transaction).

But these are merely ideas ... nothing written in stone yet.

[boolafish]

My feeling is this might be a bit premature optimising and with quite a high trade off of API user experience doing this (I guess they would need to continuously pull...). I would think it is better to back this change with some performance data and probably having some insight from @Pongch on the user impact.

But this topic itself is fun 😄, I think it would be interesting to discuss how do we implement our back pressure of the queue, especially now we got 2 queues, one for stateful check and another to collect into block.

[InoMurko]

Not doing it.