search

hashgraph / guardian

The Guardian is an innovative open-source platform that streamlines the creation, management, and verification of digital environmental assets. It leverages a customizable Policy Workflow Engine and Web3 technology to ensure transparent and fraud-proof operations, making it a key tool for transforming sustainability practices and carbon markets.
Apache License 2.0
97 stars 129 forks source link

Public Policies and Schemas #47

Closed danielnorkin closed 2 years ago

danielnorkin commented 2 years ago

Problem description

As a user, I would like for my Policies and Schemas to be discoverable beyond just my own business infrastructure silo. Currently, all schemas are stored in an internal database, therefore, we need to add the option to store schemas in a publicly accessible database or distributed storeage.

Requirements

The action of creation a schema must be recorded in a specific topic in Hedera, associated with the specific Authority. The internal database currently used needs to be synced with an IPFS node[s] for storing all system artefacts in the following way:

There will be another ticket to do with the 'discoverability' of assets (Policies, Schemas) and the 'import' of these artefacts into a Guardian instance for DR and other reasons.

There is a broader ticket #324 that covers all the work we are going to do to decentralise Guardian, beyond to just publishing schemas/policies on IPFS.

Please note that published data need to be verifiably immutable as per #162.

Definition of Done

Acceptance Criteria

blockchain-biopharma commented 2 years ago

As per @sergmetelin the Mindset we need to have is that Hedera is the Source of Truth. Every single record needs to go to Hedera first. Everything needs to be attested by the Root Authority. All of this needs to be recorded on the Hedera Consensus Service.

As per @sergmetelin , we should use IPFS for now. Everything is public and all SDK’s are available.

Definition Of Done is in the Schemas, there needs to be post a link to the Schema in the topic after we post the Schema on IPFS, we also post a Topic Message. Topic needs to be an auditable log of everything that the Root Authority does. Currently, there is no access to the linked Schema, therefore it is not auditable. Root Authority needs to create a schema, the schema is posted to IPFS, then the ID is posted to the topic. Every Root Authority action needs to be recorded under that topic, every schema, every credential. Policy has its own topic, and Root Authority has its own topic. From the moment the Root Authority came in, the Guardian should be tracking ALL of the events. Its important to have Hedera as a Source of Truth in disaster recovery (DR) situations (Shawn needs to add this as another Issue)

DOD – A Hedera Improvement Proposal, new Features, Frameworks, all in a Public manner needs to be issued describing how this works. This may be an addition to HIP 28 or a HIP 28 Updated. A flag describing how the Policy Engine works. Creating a HIP will be an entirely new Issue, @envisionblockchainpm to create as an Action Item.

blockchain-biopharma commented 2 years ago

We need to gather more requirements from Hedera on this ticket during todays Grooming.

blockchain-biopharma commented 2 years ago

Next step is to have a discussion on this as per this mornings Scrum.

blockchain-biopharma commented 2 years ago

Next Step for this is to have a meeting with developers, Serg and for Andrey to have a Review Document completed prior to this meeting. Shawn to get Serg's timing on meeting next week for another Discovery Session,

blockchain-biopharma commented 2 years ago

Next step is on Andrey to reach out to Serg for a one on one Tech Talk

anvabr commented 2 years ago

@sergmetelin Upon deeper analysis I have concerns that using IPFS here provides any real advantages. I'll summarise them here, and hopefully we can discuss them either in a meeting of asynchronously in the ticket.

There are three options of enabling storage of information on IPFS:

To host data on IPFS it requires running you own server, and saving content onto it. By default this information would be only available on this one server (not replicated), and only for the duration while the application/node is running. There is a way of making the data replicate which involve costs (Filecoin), or the data only gets replicated when someone else from elsewhere needs to access the file, and I believe for a limited time.

Instantiating and operating an IPFS node is not a trivial task, and is likely to provide a barrier to entry comparable to, if not more severe, using an unfamiliar API. Introducing IPFS technology will complicate Guardian solution architecture. It is much easier and cheaper to achieve geo-replicated and backed-up storage using commercial cloud providers services.

For demo/test environment running your own node seems to be the only option since costs and account setup etc for just with storing test/demo data seems to be unacceptable.

The alternative is to use Pinata service offering for IPFS access. This is simpler since no technical operations are required, and simpler to implement because Pinata offers an API for storage and retrieval. However it involves costs (if the usage goes beyond the very small volume of 'free tier') and complexities of setting up and maintaining accounts. It also introduces dependency on the external commercial service provider.

I propose we reassess the benefits of using IPFS, and consider alternatives such as for example introducing a solution based on a new set of Guardian APIs for discovering and accessing published schemas and policies on the instance. Each each instance of the Guardian would have a well-known API access point which can be accessed in various ways.

blockchain-biopharma commented 2 years ago

As of today, Andrey is discussing this with @sergmetelin .

anvabr commented 2 years ago

The outcome of the conversation is the following design which is an MVP for the public availability of schemas and policies only:

There will be another ticket to do with the 'discoverability' of assets (Policies, Schemas) and their 'import' into a Guardian instance for DR and other reasons.

There is a broader ticket #324 that covers all the work we are going to do to decentralise Guardian, beyond to just publishing schemas/policies on IPFS.

Please note that published data need to be verifiably immutable as per #162.

anvabr commented 2 years ago

Next step:

blockchain-biopharma commented 2 years ago

Alex has this been Designed and have you reviewed with the Development Team?