New Tool: Events indexing service

[psiemens]

👋   If you are interested in working on this issue, please check out the Getting Started guide on HackerEarth!

Description (Problem Statement)

Most Flow applications, and especially dapps, need to respond to on-chain events emitted by Cadence smart contracts. Typically, an application will periodically query the Flow Access API for a specific set of events that they care about. When an event occurs, the application will update its internal state (e.g. a database) or notify its users of the on-chain activity. Many teams have integrated event tracking systems directly into their application infrastructure, but this is a common enough problem that it warrants a simple and general-purpose solution that can serve all teams.

Experience Required

• Familiarity with the Flow Access API and its event querying functions
• Good understanding of the Cadence event model
• Good understanding of design principles and UI/UX for developer tools
• (Optional) Experience with message queueing services such as RabbitMQ and Kafka

Minimum Feature Set (Acceptance Criteria)

Event Indexing

• A user can configure the events they wish to watch using a list of Cadence event IDs. The service will continuously monitor all events in this list. The event list can only be changed upon startup — the service does not need to support dynamic configuration (e.g. adding event IDs using an API call).
• The service should watch for events in every block. It can (and should!) use the GetEventsByHeight range to query multiple blocks at once.
• The service must not miss any events; if a failure occurs in a block, the event indexer should not proceed to the next block until all events have been fetched.

Event Delivery

• The service should deliver each event to an external consumer (i.e. a dapp backend) as a JSON payload.
  • At a minimum, the service should support event delivery via HTTP webhooks. In this scenario, the consuming application will provide an HTTP endpoint as part of the startup configuration. The consumer should respond with appropriate status consumes to indicate a successful or unsuccessful delivery — these status codes are defined by the indexer.
  • The service only needs to support delivery to a single consumer.
  • All events should be delivered in the same order they were emitted on-chain.
  • Each event should be delivered exactly once. The indexer will need to implement logic to ensure idempotency of events.

Extension (Optional) Feature Set

Message Queueing

• Some applications may benefit from consuming events via an event queue such as Apache Kafka or RabbitMQ.

Multiple Consumers

• Some applications may wish to set up multiple consumers and route specific events to each consumer. This is an optional feature because it should always be possible to start multiple instances of the event indexer, each configured with a unique set of event IDs and a unique consumer.

Milestone Requirements

1. Create wireframes and an architectural software diagram on your potential implementation of the minimum feature set.
2. Implement the minimum viable product (MVP) for the event indexing logic.
3. Implement the remaining minimum feature set and the event delivery (consumer) mechanism.
4. Prepare the final version of the application as a deployable artifact (e.g. Docker image), along with a basic example consumer application and accompanying documentation.

Software Requirements

Infrastructure

• The tool is runnable as a standalone deployment, ideally as a containerized image (e.g. Docker)
• The service adheres to the 12 Factor App methodology. In particular, the service should be configured by the environment (i.e. using environment variables).

Maintainability

• The tools or libraries used to construct the solution should be well-vetted and well-maintained.
• Code should be written in a way where it's easily extensible to new functionality and semantic enough for open-source developers to contribute to.

Testing

• All core logic should have unit tests.
• E2E tests for each feature implemented.

Other Requirements

Documentation

• The following pieces of documentation need to be completed alongside the code for a successful submission:
  • Installation Guide: How to get this application up and running.
  • Usage Guide: A highlight of all the features available and how to use them.
  • Contribution Guide: A thorough explanation of the codebase, where features are located, a diagram explaining complex interactions, etc. This is intended to be a primer for new contributors that would like to add or modify features.

Code standards or guidelines

• Follow the general contribution guidelines (borrowed from the emulator project).

Judging Criteria

• For all projects, technical & high-level judging criteria apply.
• Fault tolerance and reliability (i.e. does the service reliability deliver all events to the consumer?)

Resources

• General Resource Guide
• Flow Access API
• Cadence Events
• Example of a Flow event listener implemented in Go

[psiemens]

Hey! 👋

I'm an engineer from the core Flow team who will be the go-to person for this task (the point-of-contact).

Feel free to hit me up with any questions about this issue. Once you start, I'll also be around to help with your implementation and review code.

You can comment here or find me on Discord (I'm pete#0003). Join the Flow Discord server if you're not there already!

[nvdtf]

Hello!

I was a backend engineer on Dapper and I've created a similar service for Dapper/TopShot. Would be happy to help on this issue and provide feedback/reviews/guidance. You can reach out to me on Discord navid#3642.

[prpatel05]

Your team name with a link to the HackerEarth team

pratik1719_a2eb - FLIP Fest - Team | HackerEarth Hackathons

Tag all team members with their Github usernames that are working on the project

https://github.com/prpatel05

When you plan on completing the first milestone.

1. Create wireframes and an architectural software diagram on your potential implementation of the minimum feature set.

ECD: 2021-09-24

Excited to work on this! We built a very similar system which continuously watches for events on the Flow Blockchain and triggers custom logic based on them. It was built to be reusable and configurable such that additional hooks can be installed for certain event types. It leverages an AWS infrastructure for its compute/message queuing. We’ll iterate on the design we have and have a draft ready for 2021-09-24.

[marisming]

Your team name with a link to the HackerEarth team hackerearth.com/challenges/hackathon/flip-fest/dashboard/dc22c4b/team/

Tag all team members with their Github usernames that are working on the project： https://github.com/cata-network

When you plan on completing the first milestone. 2021.10.10

[prpatel05]

Milestone 1 PR - https://github.com/onflow/flip-fest/pull/45

[chriswayoub]

Your team name with a link to the HackerEarth team https://www.hackerearth.com/challenges/hackathon/flip-fest/dashboard/dad2979/team/

Tag all team members with their Github usernames that are working on the project https://github.com/chriswayoub @chriswayoub

When you plan on completing the first milestone. 10/29/2021

[kimcodeashian]

Good day @prpatel05, @chriswayoub!

Thanks so much for all your hardwork & participation. In order to finalize winners & prepare for prize payout, we'll need the following actions from your end.

Please provide the following information by Nov 17, 2021, (in this GH Issue is fine):

1. Team Information

• Team Members Information - Github Username + Email Contact + Percentage of prize allocation (total should = 100%)
• All mentioned members MUST react to the post with a 👍 which will act as confirmation that the information is correct, or a 👎 to indicate that the information is not correct.
• We will be reaching out via e-mail

🎖IMPORTANT: We will only proceed with prize payouts once all members have confirmed with 👍 on the post.

2. Video Demo (optional)

• Please provide a 5-minute video demo to be featured & showcased in the FLIP Fest Closing Ceremonies
• Link format & Downloadable (eg. Google Drive, Vimeo)
• Content Format (Problem Statement, your work / how you solved it, final outcome)

We will be hosting Closing Ceremonies on November 23rd, 8AM PT where we'll having closing remarks from Dete & will be announcing the winners! I'll share the details here before Nov 17.

[prpatel05]

1. Team Information
   • Team Members Information - prpatel05 + prpatel05@gmail.com + 100%
2. Video Demo (optional)
   • ECD 2021-11-16, ill edit this post with the video

[chriswayoub]

Thank you!

Team Members: @chriswayoub - chriswayoub@gmail.com - 100%

Video Demo: Are you looking for more of an instructional format or a demo?

[kimcodeashian]

Thank you!

Team Members: @chriswayoub - chriswayoub@gmail.com - 100%

Video Demo: Are you looking for more of an instructional format or a demo?

A video demo to showcase your work. Preferable a demo since we might highlight it in the Closing Ceremonies (for context)

[kimcodeashian]

Hey folks,

We've received and reviewed over 82 submissions! What an amazing community on Flow! To commemorate all the hard work done, we have finalized winners and will be announcing them during our Closing Ceremony on Nov 23rd, 8AM PT. Be sure to join us - there may be some attendance prizes & a keynote from our CTO, Dete 😉!

RSVP here so you don't miss out! See you then!