Decentralized storage (IPFS/IPNS or similar) as DDO metadata storage (including media files)

[MBadea17]

Feature # 66. Later edit by @alexcos20 :

Status-quo: Whenever an asset is published, the coresponding DDO is published at the smartcontract level (using setMetaData or updateMetaData - where an event is emited which contains the DDO). DDO can be unecrypted or encrypted (see flags option). DDO (which is a JSON Object) has a lot of fields (see https://docs.oceanprotocol.com/developers/ddo-specification)

Request: add additional types of storages (beside on-chain), so publisher can have more control on the DDO

How to:

• Define how the DDO should be published: Proposal: DDO should be a JSON object, with only ONE key, called "remote". Value of that field is a match of supported storages (see https://docs.oceanprotocol.com/developers/storage#static-urls, etc). If object has multiple keys or 'remote' key is missing, then is should be not considered as remote DDO and handled like status-quo (this way we ensure backwards compatibility) Pro: no changes required to ocean.js Cons: none? (TBD)

Example :

{
   "remote":  {
        "type": "url",
        "url": "https://url.com/ddo_x.json",
        "method": "GET",
    }
}

Encryption:
We should add two extra fields on the Storage spec (for all types of storages):

• encryptedBy: public node address who encrypted the file
  • encryptMethod: file is encrypted by encryptedBy, using method ("ecies" or "aes")

So, a fully encrypted DDO stored on IPFS will look like:

{
   "remote":  {
        "type": "ipfs",
        "hash": "12334343",
        "encryptedBy": "16Uiu2HAmUWwsSj39eAfi3GG9U2niNKi3FVxh3eTwyRxbs8cwCq72"
        "encryptMethod": "ecies"
    }
}

The indexing flow should be:

• get data from metadataCreated/Updated event
• if flags & 2 (encrypted on chain) , call endpoint decrypt on that remote node and index if it returns a DDO, then exit It's not encrypted, so we move forward:
• convert JSON string to object
• check no of keys, should == 1
• check "remote" key exists
• if no_of_keys==1 and 'remote' in object:
  • if encryptedBy is present:
    • if encryptedBy === our_node_id, decrypt the file (using storage classes) and index it
    • else, call endpoint decrypt on that remote node, index if it returns a DDO, then exit
  • if not, fetch file using storage classes and index it
• else:
  • index JSON.decoded DDO

The decrypt endpoint flow should be:

• get data from metadataCreated/Updated event
• if flags==0 (unencrypted on chain): convert JSON string to object
• else decrypt from chain Now, that we have the raw DDO:
  • check no of keys, should == 1
  • check "remote" key exists
  • if no_of_keys==1 and 'remote' in object:
    • if encryptedBy is present:
      • if encryptedBy === our_node_id, decrypt the file (using storage classes) and index it
      • else, fail with error, because we cannot decrypt
        • else, read the ddo using storage classes and return it
  • if not, return the DDO

Tasks:

• [ ] extend Storage classes: add function decrypt(stream,method=['ecies' or 'aes'], returns stream
• [ ] update ocean-node to parse remote ddo when indexing (simple fetch of remote ddo, using already existing storage clases)
• [ ] update ocean-node to parse remote ddo when decrypting