search

onflow / developer-grants

Grants for developers that contribute to the broader developer ecosystem
Apache License 2.0
48 stars 17 forks source link

Flow On-Chain Analytics Dashboard #25

Open Guisong-Fu opened 2 years ago

Guisong-Fu commented 2 years ago

Grant category

Developer service

Description

Similar to Dune. A general analytics tool for Flow blockchain.

Problem statement

Dune does not support Flow blockchain yet, and there is no other tool that provides similar services for Flow blockchain.

Proposed solution

Impact

Milestones and funding

Milestone Deliverables Timeline Risks USD requested
1 - Data Analysis P1; Initial Setup Determine what data should be collected and how the data is preferred to be visualized by surveying the community; Setup the domain and the landing page 1 week May need some support from Flow Team 3000
2 - Data Analysis P2 Determine the best way to collect/index data 1 week May need to consult Flow Team and set up an access node 3000
3 - Data Storage Determine the suitable table structure 2 weeks Time 3000
4 - Job Implementation Build up cloud infrastructure; Implement the corresponding ETL jobs and data pipeline 4 weeks Time 30000
5 - Standalone Historical Event Indexer Opensourced Refine the indexer job and make it opensource 1 week Time 2000
6 - Visualization Tool Set up Configure SuperSet 3 weeks May need to tweak SuperSet a bit; may involve some visual desgin work 20000
7 - Integration Test Dashboard being connected to data warehouse 1 week 3000
8 - Pre-build some dashboards A few dashboards can be seen 1 week Some bugs may pop out and some rework maybe needed 3000
9 - Open for testing Platform are ready to be tested; bounty program is released 2 weeks May encounter performance issue thus need to scale up the infra 6000
10 - Documentation Docmentation is ready 1 week 3000
11 - Maintainence Maintain Cloud Service; Support 3 months 18000

Team

Name Role Bio Contact
Guisong Fu Data Architect/Engineer CTO of SoulMade; muti-year experience as Data Architect guisongfu@gmail.com
SkyOne Senior Flow Blockchain Developer Founder of 7DayReFundNFT; core developer of SoulMade skyone.nanshan@gmail.com

Milestone completion

alxflw commented 2 years ago

thanks for the submission @Guisong-Fu! we'll review this application soon

alxflw commented 2 years ago

@Guisong-Fu we were just chatting about this and we want to see how you're thinking about using an existing indexing service for flow (as opposed to building your own in milestones 2-4). Currently, there's: flowscan, graffle, bitquery and flip side.

Maybe, you could explore with these team if they could offer access to their data warehouse? If not, we suggest you refine your application to account for the projected maintenance cost (given that the data collection will be large).

Guisong-Fu commented 2 years ago

Hi @alxflw , thanks a lot for your suggestion! We do have thought about collaborating with other teams as well. But according to our knowledge and experience, we may still have to set up our own indexer. It's mainly because that if we want to have complete data to analyse, we would need to index all the historical events by crawling all the available blocks, and none of the existing services seems to have done such kind of job, so we have to do it by ourselves. And we had done some tests, it seems that it won't be much more complicated to index the events from the newly coming blocks on our own compared with reading data from for example Flowscan's database and parse data from there. But indeed, maintaining the entire data collection pipeline maybe very costly. I have adjusted our application! Thanks for pointing it out! We have already kicked off this project. And we just learned that for different ranges of blocks, we have to use different access node~ 😄 We will try to crawl all the blocks and extract all the data related to TopShot, and launch the bare bone platform so users could use to build dashboards for TopShot in the up coming weeks. What a milestone TopShot has achieved! 1 billion sales! 👍 I know many have been long for some more thorough analysis on Topshot. It would be fantastic if we could help! We can always have a call at any time and have a deeper discussion. 😃

Guisong-Fu commented 2 years ago

Hi @alxflw , we have developed a proof of concept. Please let me know if you would like to try our prototype and have a further discussion. 😃

alxflw commented 2 years ago

Hey @Guisong-Fu, would love to check it out. You can find me on Discord: alx-flw.find#6198.

One thing that is coming up quite frequently is the need to index the state. I was wondering if you could build your indexer as a standalone opensource project, so other developers could re-use/improve it. Happy to chat more!

alxflw commented 2 years ago

Hi @Guisong-Fu, thanks for the demo. We have a few follow up questions:

1) It would be great if the indexing project could a standalone opensource project that any other developer could use for their own need. Do you think you could refine the milestone to account for that? Essentially, milestone 1-4 would lead up to that deliverable

2) Can you confirm that SuperSet can be set up in a way that allows anyone without an account to access a shared dashboard? We'd consider this viz UI "public infrastructure" for Flow, so it should have the least friction for anyone to view

3) Could you extend milestone 1 to include surveying the community to see what data and visualization needs they have?

4) Could you specify the time you'd be able to maintain the project in milestone 10? I'd suggest we start with 3/6 months. Follow up grants for maintenance are possible.

Guisong-Fu commented 2 years ago

Hi @alxflw , thanks a lot for all the great suggestions! 😄 We now have updated the proposal accordingly!

  1. We have added this "opening up the indexer job as a standalone opensource project" as milestone 5, as it will take a bit of refinement work. The idea is that everyone could use this repo as a template so to crawl/index the target events. We are also thinking about setting up a central place(e.g. a shared cloud storage) where users could upload/download/share the scanned data for other use cases. We can take this as our next step.

  2. Yes, it can be open to the public audience(anonymous users, users who do not login) if the dashboard builders permit so. Authentication and authorisation are big topics thus we didn't have time to take them in with our demo. But it can definitely be configured in whatever ways we want.

  3. Sure. And the idea is that everyone can build the dashboard in whatever ways they want, so we probably do not need to worry too much about how the data should be visualised. But sure, we can pre-build some dashboards in the ways that community would like to see as examples and references for other builders.

  4. Thanks for pointing it out! We now have set it to 3 months. The main cost would be coming from the cloud infrastructure, according to our rough estimation, the infra would cost around 1,500 USD per month. But it may go up if there are more needs.

alxflw commented 2 years ago

Fantastic, thanks for the updates and confirmations!

WRT 2: It was pointed out to me that there was pre-existing work done on the indexing. Maybe you could see how that would impact your milestone 5? If the projects below already cover what is needed for indexing, maybe you could use them directly?

Guisong-Fu commented 2 years ago

Hi @alxflw , maybe there is a bit of misunderstanding here 😃

In terms of milestone 5, I'm actually more thinking of "an indexer job that indexes all the historical events" rather than the new events. The reason why I think of it in this way is that when we did the proof of concept, we found it actually a bit tricky to crawl all the historical blocks and index the target events. We want to make it easy so everyone could easily crawl all the historical blocks and retrieve the data wanted. I do believe there will be needs on the historical data for other use cases as well besides analysis.

And yes! We know Flow-scanner, it's a pretty cool project! And there are also a few other projects that do the similar thing. Have you seen this one? It's very thorough. 👍 https://github.com/ph0ph0/Get-The-Flow-Down#event-listenersmonitorsindexers

For indexing new events, we will look into all of these options and pick the one that suits us most. And maybe we could work with Graffle as well. 😄

alxflw commented 2 years ago

wonderful, that makes a lot of sense! thanks for clarifying

Guisong-Fu commented 2 years ago

Hi @alxflw , any update on this grant? 🙂 Thanks!

alxflw commented 2 years ago

hi @Guisong-Fu, this proposal was just accepted! 🚀

I'll follow up with you with next steps on discord.

flipsidejack commented 1 year ago

Hi, just dropping in here to spread some awareness - I just found this developer request in Flow's year-end grant summary and list of requests. It's mentioned that there are no tools like Dune that support Flow yet, but we actually do at Flipside. We have the complete chain indexed back to mainnet launch, with curated tables for easy blockchain querying and analysis.

You can see the tables by trying out our query editor at: flipside.new

Here are some recent dashboards to give you a gist of the analysis possible:

Regarding the milestone steps, Flipside has completed every single one. This is not a request for grant, just wanted to make you aware that we've been working on this

bluesign commented 1 year ago

@flipsidejack I just tried your app yesterday, I can honestly say, I am beyond amazed.