[BUG]: Removal of bad transactions from the mempool

[sgerbino]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current behavior

It is possible for a transaction that will never succeed to end up in the mempool with no way for it to be removed.

Expected behavior

I expected block production to be robust enough to never result in an unrecoverable failure.

Steps to reproduce

No response

Environment

- OS: Any

Anything else?

No response

[sgerbino]

• Apply new transactions to a pending transaction state in koinos-chain so that rejecting incompatible transactions occur as soon as possible
• Modify apply_block such that it returns a failing transaction in the RPC response if it occurs (in structured json format)
• Modify koinos-block-producer so that if a block fails because of a particular transactions, we remove that transaction from the block and all others that proceed it
• In koinos-chain upon reapplication of pending transaction state, when a transaction that previous succeeded now fails, we broadcast a message about it allowing the koinos-mempool to remove the transaction from the pending list
• Handle the newly created transaction_failed broadcast within koinos-mempool

Note: In koinos-chain, all writes to the pending transaction state needs to be locked in order to ensure matching ordering of pending transactions between koinos-chain and koinos-mempool.

[sgerbino]

This bug will be handled by koinos/koinos-chain#540.