Polkadot <> Arweave

[cedriking]

Polkadot <> Arweave

Archival Bridge

Introduction

Polkadot is an ecosystem of connected blockchains (‘parachains’), designed to handle huge numbers of transactions. Like all blockchain systems, these parachains create a significant amount of data that, in most cases, needs to be stored indefinitely. Arweave is a network built to provide scalable and economically sustainable permanent data storage. The network achieves this using a Succinct Proof of Access mining mechanism (enforcing that miners replicate data), as well as a storage endowment that yields interest which is employed to reward miners perpetually.

This jointly funded bounty is to build an archival bridge of data from Polkadot and its parachains, into the Arweave network. Once built, anyone will be able to spin up a version of the solution and begin to archive specific chains from the Polkadot ecosystem into the Arweave network. By tagging each piece of data and employing bundled data entries (ANS-102) on Arweave, the resulting dataset will also be queryable by any dApp’s front-end using a GraphQL interface.

Application requirements

The application shall facilitate a bridge to connect Polkadot data to Arweave storage, watching the Polkadot chains for new blocks and transactions, packaging them into bundles, and dispatching them to the Arweave network.

Given the availability of tooling and similar implementation work with SKALE and Solana, a JavaScript/TypeScript-based solution will likely be fastest to engineer.

The solution must fulfil the following criteria:

1. The archival bridge must store each of the following from Polkadot chains selected by the deployer:

• Block Data
• Transactions Data
• Transaction Receipts
• Event Data

2. The data must be packaged into bundles using the Arweave’s ANS-102 standard. The arweave-bundles JavaScript/TypeScript library may be useful in creating these bundles. This will allow you to minimise the AR fee requirements of your solution.

3. All entry types must be appropriately tagged, using a schema either completely compatible, or as closely compatible as possible, with prior Arweave archival bridge projects for Solana and SKALE.

4. Acceptable solutions must include detailed documentation, capable of allowing an engineer unfamiliar with the project to deploy it easily, as well set up a dev environment.

Rules

Successful submissions will meet the following criteria:

• Submissions must be your own original work.
• Submissions must be open source, with the full source code available on Gitopia, GitHub or another open code hosting repository. Feel free to use whatever OSS license you prefer.
• Your submission must include a link to a live, functioning, and running instance of the app itself, as well as a link to the repository it is stored within.
• All submissions must include a brief description of the application and functionality in the GitHub repo.

Resources

Need to learn more about Polkadot or Arweave technology? Use the resources below, or reach out to the relative dev communities on Discord:

Arweave Dev Discord Polkadot Discord

Existing Archival Bridges

SKALE Solana

Arweave

Get an Arweave Wallet & Free Tokens Arweave Web Extension Wallet Developer Documentation & Getting Started Guides Arweave JS Arweave Bundles

Polkadot

Polkadot Builder's Portal Developer Documentation & Getting Started Guides

Submission Deadline

The hackathon will take place from the 28th of January 2021 to the 25th of February 2021. The submission deadline is the 25th of February 2021 @ 23:59 PM PST.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 7500.0 USDC (7500.0 USD @ $1.0/USDC) attached to it as part of the ArweaveTeam fund.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
  • Want to chip in? Add your own contribution here.
  • Questions? Checkout Gitcoin Help or the Gitcoin's Discord
  • $1,054,406.57 more funded OSS Work available on the Gitcoin Issue Explorer

[CB002]

@cedriking May I know the difference between Transactions Data and Transactions Receipts; My initial understanding was the first one refers to the wallet transfers between accounts, but I am not sure about the second. Glad if you can provide some examples. Or are you referring to tx hash correlation to balance events?

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Workers have applied to start work.

These users each claimed they can complete the work by 1 month ago. Please review their action plans below:

1) jarcodallo has applied to start work _(Funders only: approve worker | reject worker)_.

Description

The goal of Polkadot <> Arweave Archival Bridge is to allow users to listen to new blocks and extrinsics using polkadot-js/api, packaging them into bundles using arweave-bundles, and dispatching them to the Arweave network through arweave-js.

Architecture approach

Using the Substrate RPC interfaces through polkadot-js/api for listening to new blocks (api.rpc.chain.subscribeNewHeads) and make a call to retrieve the current network head. The head contains the block number with this we can query for the block hash and then get the block data. Transactions are included in a signed block as part of the extrinsics, some of these will be unsigned and generated by the block author and some of these may be submitted from external sources and be signed.

NodeJS backend will be implemented using the polkadot-js/api and arweave-js. Polkadot-JS will allow us to interact with any substrate-based node (Polkadot relay chain, parachains, Kusama, testnets, etc).

Deliverables

• Open-source repositories licensed under APACHE2.0.
• Docker Compose with a NodeJS service, MongoDB or Redis instance to store everything related to blocks and transactions, Polkadot Node (connected to a one of the relay chains) and Arweave Node. All this for local development.
• Integration Test.
• Documentation

Questions

What do you mean with Transaction Receipts on Polkadot? 2) fabianriewe has applied to start work _(Funders only: approve worker | reject worker)_.

Me and @johnletey have a lot experience in the Arweave Ecosystem. Our Approach: Build a NodeJS Backend which connects to a configureable Polkadot chain using PolkadotJS (api.rpc.chain.subscribeNewHeads). We then extract all necessary data via PolkadotJS (api.rpc.chain.getBlock, api.query.system.events etc) and bundle the data into an ANS-102 scheme via the Arweave library. To compensate the speed differences between the two chains we are using a queuing system to create an asynchronous architecture. Additionally we are building a library to make the accessibility of the archived data easier. By this developers with zero-knowledge of the backend structure can make straight-forward library requests to access the data stored on Arweave. 3) heruard has applied to start work _(Funders only: approve worker | reject worker)_.

Ethereum bitcoin indodax binance stake usdt dogecoin token swap 4) 1-macros has applied to start work _(Funders only: approve worker | reject worker)_.

Typescript for the client/service layer, Mongodb as a shared queue and with the TTL. With this approach worker layer can be scaled independently to overcome the impedance mismatch between two chains.

Approach: Live subscriptions will be created for new blocks, and system events. These will be pushed down to different mongodb queues; indexed on some meta info (like blocknumber, parentHash) for easy retrieval. Workers will listen to the queues and will try to fetch the extra information (like block data) and will index them onto Arwave .

Questions: What's is mean by Transaction Receipts? Are you referring to getting receipts of one or more wallets?

Thanks 5) kaaid has applied to start work _(Funders only: approve worker | reject worker)_.

I look forward to working together 6) mccrearyd has applied to start work _(Funders only: approve worker | reject worker)_.

problems to address (I have a machine learning-y approach due to my background).

1. division of work:

• if we have 100 nodes all targeting some chain, how do we divide the work between them? well, if we do want redundancy, then the best way to do this would probably be stochastic sampling. inspiration: stochastic gradient descent. you don't need to go and explicitly copy every single one one time exactly, but rather if enough workers randomly traversed a target chain, they would eventually converge to cover the whole surface (in other words, bridge from Polkadot to Arweave. A method that may be more applicable here is fractal monte carlo, an implementation of fragile AI. traversal of the "state space" would be contextualized to be "chain space". as the chain increases in complexity going backwards, recruiting more nodes to that specific location could be done by exploring the "virtual reward" landscape.

• if redundancy is prohibited, division of work becomes a bit more costly, but not impossible. live synchronizing between nodes can be done in real time by broadcasting a "target chain interval" to all listening nodes. if I am a node, and I want to join efforts, first I listen to the broadcasts from other nodes for a while and try to gather an idea of what blocks each one is working on and determine a (small) interval to work on. the reason it should be small is in case I hear another node randomly picked an overlapping interval, I can still continue my work and it's not a terrible redundancy loss and I can move on to the next interval.

2. node deployment/runtime

• docker container that act as cellular automata exploring the "chain space". 7) stephenalaba has applied to start work _(Funders only: approve worker | reject worker)_.

I will do all the required tasks. 8) itsmajestix has applied to start work _(Funders only: approve worker | reject worker)_.

I have created a program called Loom that I believe satisfies the requirements for this project. It can be found at https://github.com/ItsMajestiX/loom. It works by connecting to any Polkadot node running in archive mode and getting block data from it, turning that block data into a JSON format, and uploading the blocks one at a time as transactions queryable using ArQL when the program is in library mode. It can also bundle many of these transactions up using Arweave bundles, optionally compressing the bundle, and then sending it to the network when the program is in archive mode. I also plan on adding a library for others to be able to access the stored blocks later.

Learn more on the Gitcoin Issue Details page.

[aahutsal]

Is that bounty still actual? Or you should cancel/close it, @cedriking ?

[cedriking]

Hey @aahutsal thanks for the message. This bounty is closed.