algorand / conduit

Algorand's data pipeline framework.
MIT License
37 stars 26 forks source link

DISCUSSION: MAPPING POTENTIAL USE-CASES TO CONDUIT'S DEVELOPMENT GOALS #30

Closed kcelestinomaria closed 1 year ago

kcelestinomaria commented 1 year ago

Problem

Hi Algorand Conduit Dev Team. I lead the team at jasiriprotocol.org , an Algorand Foundation Grantee project, and we've been having some technical issues and constraints, since last year, trying to push a full mainnet solution.

One of the issues was the lack of a robust data layer that is native to the Algorand network.

For our dAPP(which bridges Web2 data to Web3 and vice versa), we couldn't avoid using AWS or Azure DBs, in as much as we tried to 'decentralize' our solution. This limits our goal of putting data and assets in 100% control of the users on our protocol, despite it being a design principle.

Storage on Algorand is expensive, and in most cases Algorand is simply a state machine, just timestamping data on the go and then storing it in a local DB.

Solution

An iteration on Conduit that optimizes it for almost-free event streaming, crucial for data-centric applications. Simply think of it as decentralized Kafka, natively on Algorand, at web scale.

Someone could then use Conduit, in this scenario, for:

  1. Distributed compute pipelines
  2. Open, decentralized, and composable databases on Algorand... very helpful for applications that rely on NFTs or just any use case that is reliant on data, or solving a data problem at scale.

Dependencies

This could potentially be merged with DIDs on Algorand, mapping user auth to the data. But that is just an idea.

I think one-click, downloadable relay nodes are needed so as to make the high-level idea a reality.

IPFS might be needed as well, but this one can come as an option for the developer leveraging conduit for their data solution

Urgency

This will help further scale Algorand, from the data storage standpoint, by enabling:

winder commented 1 year ago

Hi @kcelestinomaria, conduit is designed to be extended by builders like you for accessing on chain data. You can build an exporter for your application that stores just the data you want in a format convenient for your use case.

I would be interested in hearing more about what you're trying to do, could you message me on discord? My handle is will#0709.

winder commented 1 year ago

Conversation moved to discord.