Part 1 is the first part of the Event Streaming expansion set of work. Its goal is to expand event streaming within the Flow protocol. The current grant encompasses tasks that address investigation steps, community and team discussions, API definition for the whole scope of work, and FLIP's engagement. The part is divided into two separate milestones.
Part 2 focuses on 6 open issues that address technical debt for Access and Observer nodes, as well as 2 tasks to maintain the issues and usability of Access node which the core Flow team would like to resolve but currently cannot prioritize.
The part is divided into three milestones.
Proposed solution
Part 1 - Event Streaming enhancement design
1 - [Access] Make a proposal for the community in Discord to expand streaming for blocks, headers and transaction results. Discuss with community the idea overall and other possible needed subscriptions, get a feedback about current streaming approach for events and execution data, and possible ways for improvement.
2 - [Access] Investigate community applications/products to base streaming expansion on users needs.
3 - [Access] Investigate subscribing API on other platforms like Infura, Alchemy, QuickNode etc. to discover api templates and most usable endpoints.
4 - [Access] Revise the current streaming approach based on feedback from community and researches, create a basic design.
5 - [Access] Write a FLIP document based on discussions with community and investigation results.
6 - [Access] Share the FLIP document with the community for further polishing and improvements.
Introduce two new methods for access API: GetProtocolStateSnapshotByBlockID and GetProtocolStateSnapshotByHeight that return the snapshot from the specified block if that block is finalized.
8 - [Access] Handle inconsistent protocol state in access rpc engine #4028 ( https://github.com/onflow/flow-go/issues/4028 ):
Document the expected errors from all access API endpoints and implement handlers for each of them. Any unexpected errors should be forwarded and managed by irrecoverable.SignalerContext as irrecoverable.
9 - [Access] Add endpoints to Execution nodes to support getting Transaction Result error messages #4754 ( https://github.com/onflow/flow-go/issues/4754 ):
For execution node add next API calls:
Get error messages for all failed txns for block and specific collection: (blockID, collectionID)
Get error messages for all failed txns in block and all collections: (blockID)
Get error messages for a specific failed txn in block: (blockID, txID)
Add tests
For access node: update GRPC API, add supporting code for AN to call new EN methods.
Adopt current reconnectingClient function implementation of access Forwarder to use existing GetAccessAPIClient functionality from access ConnectionFactory to retrieve clients.
11 - [Access] Cache Transaction Result error messages #4850 ( https://github.com/onflow/flow-go/issues/4850 ):
Add support for AN to cache error messages in a separate storage in lazy fashion. Whenever we sync execution data and depending on user's query we will request messages for failed transaction and cache it in local db. Next queries for the same transaction will be served from local DB. Add tests.
12 - [Access] Get Block endpoint is missing the system collection #4584 ( https://github.com/onflow/flow-go/issues/4584 ):
Implement a separate API calls to return system transaction and system transaction result. Remove special workarounds for retrieving system tx that are present in AN API calls right now. Update existing tests and add new tests.
13 - [Access] Prepare documentation for existing streaming API. Make a publication on onflow/docs portal.
14 - [Access] Maintenance of existing code.
Extra effort for possible issues that may arise on the functionality related to previously developed features.
Impact
Proposed fixes will improve the streaming capabilities of the Flow project and will have a beneficial impact on community needs.
Milestones and funding
Milestone 1
Milestone
Deliverables
Timeline
Risks
USD proposal
1 - [Access] Community proposal
Make a proposal to the community to expand streaming. Discuss the idea, get the feedback.
24h
-
1560$
2 - [Access] Analyze streaming expansion
Investigate community applications/products to base streaming expansion on users needs.
24h
-
1560$
3 - [Access] Investigate subscribing API on other platforms.
4 platforms analyzed to get the best approaches ideas.
24h
-
1560$
4 - [Access] Revise the current streaming approach based on feedback from community and researches, create a basic design.
Draft design ready
80h
-
5200$
Milestone 2
Milestone
Deliverables
Timeline
Risks
USD proposal
5 - [Access] Write a FLIP document based on discussions with community and investigation results.
FLIP ready and properly documented
120h
-
8580$
6 - [Access] Share the FLIP document with the community for further polishing and improvements.
Final version ready, discussed and published into repo
80h
-
5200$
Milestone 3
Milestone
Deliverables
Timeline
Risks
USD proposal
7 - [Access] Add endpoint to get protocol snapshot from any block #4869
Grant category
Description
This grant proposal consists of two parts:
Part 1 is the first part of the Event Streaming expansion set of work. Its goal is to expand event streaming within the Flow protocol. The current grant encompasses tasks that address investigation steps, community and team discussions, API definition for the whole scope of work, and FLIP's engagement. The part is divided into two separate milestones.
Part 2 focuses on 6 open issues that address technical debt for Access and Observer nodes, as well as 2 tasks to maintain the issues and usability of Access node which the core Flow team would like to resolve but currently cannot prioritize. The part is divided into three milestones.
Proposed solution
Part 1 - Event Streaming enhancement design
1 - [Access] Make a proposal for the community in Discord to expand streaming for blocks, headers and transaction results. Discuss with community the idea overall and other possible needed subscriptions, get a feedback about current streaming approach for events and execution data, and possible ways for improvement. 2 - [Access] Investigate community applications/products to base streaming expansion on users needs. 3 - [Access] Investigate subscribing API on other platforms like Infura, Alchemy, QuickNode etc. to discover api templates and most usable endpoints. 4 - [Access] Revise the current streaming approach based on feedback from community and researches, create a basic design. 5 - [Access] Write a FLIP document based on discussions with community and investigation results. 6 - [Access] Share the FLIP document with the community for further polishing and improvements.
Part 2: Access node open issues tech debt
7 - [Access] Add endpoint to get protocol snapshot from any block #4869 ( https://github.com/onflow/flow-go/issues/4869 )
Introduce two new methods for access API:
GetProtocolStateSnapshotByBlockIDandGetProtocolStateSnapshotByHeightthat return the snapshot from the specified block if that block is finalized.8 - [Access] Handle inconsistent protocol state in access rpc engine #4028 ( https://github.com/onflow/flow-go/issues/4028 ): Document the expected errors from all access API endpoints and implement handlers for each of them. Any unexpected errors should be forwarded and managed by
irrecoverable.SignalerContextasirrecoverable.9 - [Access] Add endpoints to Execution nodes to support getting Transaction Result error messages #4754 ( https://github.com/onflow/flow-go/issues/4754 ): For execution node add next API calls:
Add tests
For access node: update GRPC API, add supporting code for AN to call new EN methods.
10 - [Observer] Improve upstream gRPC failovers with AN connection pooling #2955 ( https://github.com/onflow/flow-go/issues/2955 )
Adopt current
reconnectingClientfunction implementation of accessForwarderto use existingGetAccessAPIClientfunctionality from accessConnectionFactoryto retrieve clients.11 - [Access] Cache Transaction Result error messages #4850 ( https://github.com/onflow/flow-go/issues/4850 ): Add support for AN to cache error messages in a separate storage in lazy fashion. Whenever we sync execution data and depending on user's query we will request messages for failed transaction and cache it in local db. Next queries for the same transaction will be served from local DB. Add tests.
12 - [Access] Get Block endpoint is missing the system collection #4584 ( https://github.com/onflow/flow-go/issues/4584 ): Implement a separate API calls to return system transaction and system transaction result. Remove special workarounds for retrieving system tx that are present in AN API calls right now. Update existing tests and add new tests.
13 - [Access] Prepare documentation for existing streaming API. Make a publication on onflow/docs portal.
14 - [Access] Maintenance of existing code. Extra effort for possible issues that may arise on the functionality related to previously developed features.
Impact
Proposed fixes will improve the streaming capabilities of the Flow project and will have a beneficial impact on community needs.
Milestones and funding
Milestone 1
Milestone 2
Milestone 3
Milestone 4
Milestone 5
Total funding proposed: 42900 USD
Team
Primary skills include C++(Standards 11,14,17), Rust, and Golang.
Primary skills include C++(Standards 11,14,17), Rust, and Golang.
Primary skills include C++(Standards 11,14,17), Rust, and Golang.