Open glesaint opened 2 years ago
I think there is a significant architectural point to consider, when you take a few of the above requirements into account.
Currently Ethconnect functions in a single "nonce assigned at end" mode. It performs streaming based nonce management as close as possible to a target node, and for any given from
address it makes sure all nonce management is performed using the transaction pool of one node at any given time.
A stream of transactions flows in order from parallel submitting REST API calls that add unsigned transaction JSON payloads to a queue, through to the "processor", which delivers batches of transactions into the chain by assigns nonces as part of the last phase in the pipeline. This means signing (internal and/or external) occurs at this last phase of the pipeline - as this is the point at which nonces are known.
The signing itself can be internal to the ethconnect code, or external. Both have been performed with the code as it is today, and extending the plug-point for new consumers of the codebase to plug in signing in new ways would be simple.
This model is great for allowing the applications to be highly parallel, submitting REST API requests to the ethconnect tier. By plugging in the optional Kafka layer, it provides a very reliable stream of transactions, that allows applications to submit bursts of transactions very quickly.
With Kafka included, it also allows the nodes to be parallel. The "consumer group" concept in Kafka allows for multiple nodes to be placed down-stream of ethconnect, while still maintaining the integrity of the architecture where there is a single stream of ordered transactions. Kafka partition management (using the from address, or a custom ordering context) allows multiple separately ordered queues to exist.
So how does all this relate to the above...
Well there are two things hinted at in these requirements that might mean introducing a fundamentally new mode of nonce management to the technology, or some other significant creative architectural thinking:
It wasn't clear this is absolutely required from the comment, but it was implied when mentioning a couple of technologies like metamask
etc.
Because nonces are part of the signed transaction payload, this means any change to the nonce assigned to the transaction requires a re-signing. So the HTTP contract with applications at the front side of ethconnect would be radically changed by introducing a fully-fledged "nonce assigned at start" approach.
Here it would be much more like a feedback loop with the submitting applications, where it would say something like:
Also implied in the requirements, is that individual transactions have a life that's policy-driven.
Rather than being just another transaction in an ordered queue, they are unique transactions that might need to be guided onto the chain, over periods of main minute/hours, and other transactions might be going ahead/behind them.
This again would imply a much richer feedback loop, based on a stateful store, with a UI/API management tier to see what state various transactions are in.
For this layer, I'm not sure that the ethconnect connector itself is the right home within the FireFly stack. It might be better to pull this back out into a plugin within FireFly that takes the operation
construct that exists at that layer, and performs policy based decisions around re-submission and management.
e.g. FireFly core (with plugins) becomes the application involved in the feedback cycle mentioned in 1-4 above.
The picture you've painted for me through these requirements is something like:
|--------------| |----------------| --- submission ---> |--------------------------|
| business app | --- fire&forget --> | FireFly core | loop | ethconnect (/fabconnect) |
|--------------| |----------------| <--- feedback ----- |--------------------------|
|
----------------
| policy engine |
----------------
Background
EthConnect was initially designed as a bridge into Ethereum permissioned chains. These chains usually are running a PoA/IBFT consensus algorithm, where the finality of transactions is guaranteed. These chains are usually configured with a gas price of 0, so that transaction fees are not required. Lastly, the permissioned chain provider usually offers a integrated signing service. Users of public blockchains have similar needs for an ESB/EAI integration in many cases, which EthConnect could fulfill provided that features are built to support other scenarios than those outlined above.
Motivation
It's beneficial for the community to have a product that addresses the needs of public, and private chain users, so that developers working on the Ethereum platform have a set of tools available to them that can address any requirement for integration. At first sight, the work required to enable this in EthConnect does not seem to have any roadblock.
Requirements
Gas Price
Finality
External Signing
Events Detection
Outline
To be discussed with the community on how the above can be implemented.