Possible Opportunity for Magda to Solve Boarder Sense `Open Data` Issue With Merkle Tree Related P2P Technologies

[t83714]

Further our topic on re-position Magda from the technology innovation perspective, I think we might want to have a look at the possible opportunity for Magda to solve the in a broad sense Open Data issue (i.e. not just the public sector or government-related ) with Merkle Tree Related P2P Technologies.

The main idea is to help open/public dataset to be published, distributed, utilized, recreated & authorized over a distributed network and managed & governed by the network itself. Therefore, the data publisher doesn't have to take the data custodian role or worry about the dataset distribution infrastructure.

The benefits will be:

• Truly Open Data. The data will not be managed & governed by a particular entity. Instead, it would be the network who managed & governed based on a suite of predefined policy/contracts of the dataset. The ultimate encourage collaboration on open data and improve data quality.
• There is no central node of the infrastructure that physically distributes the data, nor a central authority for governance. Thus, it will be a more reliable system for everyone and any dataset won't become inaccessible when the original publisher's gone.

Here are two closely related ideas:

1. Leverage blockchain technology (probably a private version in order to be free) to:
   • allow a dataset to be created with policy & contract of how the data can be improved and what's the (quality) standard of the dataset.
   • let network consensus to execute & manage the future collaboration of the dataset 2 Leverage P2P content sharing technology e.g. IPFS to solve the problem of the dataset distribution problem. Therefore, the data publisher doesn't have to serve the whole world's traffic of the dataset. And the content will be hosted by everyone who gets involved.

[maxious]

"Dat is a protocol for sharing data between computers. Dat’s strengths are that data is hosted and distributed by many computers on the network, that it can work offline or with poor connectivity, that the original uploader can add or modify data while keeping a full history and that it can handle large amounts of data." https://datprotocol.github.io/how-dat-works/ https://dat.foundation/

There is an opportunity to have a data catalog - just like GitHub, where is DatHub? Even if you have the network, you still need to discover data and additionally know about forks of the data so you can use them or merge them back in to the original dataset - Pull Requests?

[t83714]

Thanks, @maxious Dat is a very good one~ Yeah~ As you said, I am thinking about a Dathub like Github but even further the management/governance of a particular dataset should be decentralized & not rely on the data publisher as well (have been a data consumer previously, it seems not every publisher is interested in managing the dataset ).

Currently, the open data publishing process for a publisher is more like a make data available process. After a dataset is published, the data publisher is still reliable for:

• running a good infrastructure to serve the data file downloading request from the whole world (this could be solved by Dat, IPFS or even Academic Torrents kind solution)
• management/governance of the published dataset & collaborate with the general public for the dataset improvement (e.g. merge/process the pull request).

A P2P data sharing protocol only solves the first issue. But We do need a data catalog for the second issue (data discovery, management & governance).

I think we probably need a truly de-centralized data catalog to free the publisher from many obligations and also enable the general public to get involved at the same time.

To achieve that, I think we probably can:

• Store the followings to a block chain public ledger
  • partial dataset metadata for discovery (full metadata still store at Dat or IPFS networks)
  • An end user facing searching portal should store a cached version full ledger in a different store (e.g. elasticsearch) for better searching experience
  • a dataset collaboration policy that defined data quality requirements & dataset pull request management policy etc.
  • Those policies probably can be defined as programmable smart contract (e.g. a Solidity program ) to allow some of the changes to be validated by different nodes automatically
• Anyone can attempt to update the dataset (dataset metatada, actual data file or even policy )
• Those update (or we call pull requests) are required to be validated by a certain number of nodes in a predefined way (all defined by the original policy as the part of the dataset)
  • Some of the updates might be able to be validated without human if the original policy was defined well (e.g. assume it's a simple CSV file, we have simple requirements of the data type of each column)
  • Those require to be validated by a human need to be voted by a number of participants from the general public as defined by the dataset policy

Only some rough ideas here~ 😄

The other problem is most current blockchain / public ledger system use Proof-of-Work type consensus protocol. It may imply the financial cost (around 1 USD per dataset, according to here ) for any data to be stored on the public ledger.

We might want to try a different type consensus protocol (e.g. a mix of Proof-of-Authority with other types --- no clear idea on this yet) to offer a coin-free (no compulsory cost) network.