pablomendezroyo commented 3 years ago

Similar to the fullnode/remote ETH client option there is for DAppNode right now, implement a similar feature for IPFS:

[x] Change DAPPMANAGER IPFS consumer code to be able to fetch all necessary data from a Gateway instead of an IPFS HTTP API. The biggest challenge will be with directories. Can we ask IPFS devs to add a header and return directories as JSON
[x] Must verify all data downloaded from Gateways (cryptographically) ensure the hashes match
[x] Deploy an IPFS node in a server taking as reference the ETH remote node OR use a remote IPFS node mantainned by a third party (Pinata)
[ ] DAPPMANAGER UX. Add a new "feature" popup similar the eth multiclient. Ask new users. For old users we should ask too. Offer 3 options:
1. Independent: full node of geth and full node of ipfs
2. Not independent: remote node of get and ipfs
3. Advanced: select custom options for each node (full, light, remote...)
[ ] Do same procedure as geth to grab content hash in case the remote ipfs node is down so the user is not blind
[ ] Make IPFS a "core not mandatory" package, similiar to the https and wireguard packages
[ ] Test it

Advantages:

Fix the well know issue of IPFS breaking the local network os dappnode users

dapplion commented 3 years ago

IPFS v9.0.0 exposes the DAG endpoint in the gateways so it should be feasible to validate data as we download it.

IPFS v9.0.0 release: https://github.com/ipfs/go-ipfs/releases/tag/v0.9.0
The downloaded format is CAR, learn about it here https://ipld.io/specs/transport/car/carv2/ - to check a JS implementation of the format, see https://github.com/ipld/js-car
To validate the download stream we could use this package https://github.com/web3-storage/ipfs-car

pablomendezroyo commented 3 years ago

UPDATE

The main blocker to this issue is how to verify the content from an IPFS directory to consume trusted data from a remote IPFS gateway.

IPFS forum discussion: https://discuss.ipfs.io/t/verify-content-with-api-dag-export/11813/3
Main libraries to be used: https://github.com/olizilla/ipfs-get https://github.com/web3-storage/ipfs-car https://github.com/ipld/js-car
Proof of concept: https://gist.github.com/pablomendezroyo/90a57f742dbe13a6db4e7b702ba823c2
- Fetch /dag/expor and consume CAR content
- Hashes CAR content and compare it to the original CID
- Write files if CIDs are equal

TODO

[x] Simulate and attack and test it works
[x] Improve and clean code
[x] Add integration test and startup an IPFS node to test all IPFS functionality
[x] Research for how to not download all the content from dag/export to verify is trusted data

3alpha commented 2 years ago

@pablomendezroyo I think this is done. Right?

pablomendezroyo commented 2 years ago

ups, right

closed by https://github.com/dappnode/DNP_DAPPMANAGER/pull/901

dapplion commented 2 years ago

@pablomendezroyo is this done?

DAPPMANAGER UX. Add a new "feature" popup similar the eth multiclient. Ask new users. For old users we should ask too. Offer 3 options: Independent: full node of geth and full node of ipfs Not independent: remote node of get and ipfs Advanced: select custom options for each node (full, light, remote...)

dapplion commented 2 years ago

Still part of core package this is not complete

https://github.com/dappnode/DNP_CORE/blob/6fa54e7551fcc1f4fcbeff244d817c5ed390ddff/dappnode_package.json#L23

  "dependencies": {
    "bind.dnp.dappnode.eth": "0.2.6",
    "ipfs.dnp.dappnode.eth": "0.2.15",
    "dappmanager.dnp.dappnode.eth": "0.2.45",
    "wifi.dnp.dappnode.eth": "0.2.8"
  },

dappnode / DNP_DAPPMANAGER

Make IPFS as optional #833

UPDATE

TODO