rvagg / js-ipld-hashmap

An associative array Map-type data structure for very large, distributed data sets built on IPLD
Other
23 stars 4 forks source link

How to connect to IPFS node #17

Open eduardonunesp opened 3 years ago

eduardonunesp commented 3 years ago

How to connect to a store that connects to an IPFS node, for instance localhost:5001, could you please give an example?

rvagg commented 3 years ago

You'll probably want to use the ipfs-http-client https://github.com/ipfs/js-ipfs/blob/master/packages/ipfs-http-client/README.md to talk to the node and use block put and block get APIs https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/BLOCK.md - or the dag put and dag get APIs https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/DAG.md, although you probably don't need the overhead of re-encoding blocks since this library uses the bytes and CIDs directly.

Your store would then have to use these APIs to get and put by CIDs:

const store = {
  async get (cid) { /* ... ipfs-http-client call to get block bytes by CID ... */ },
  async put (cid, bytes) { /* ... ipfs-http-client to store these bytes with this CID ... */ }
}

The ideal is that there would be an intermediate layer such that you don't store every block that gets created during a mutation of a HAMT because it's going to create a lot of garbage, but you could lean on IPFS' GC to handle some of that, it's just unfortunate that you have HTTP calls in between and the cruft building up. A big TODO for our JS IPLD stack is some kind of easy intermediate store that allows for more efficient GC of intermediate blocks so that you get to say "I have a full HAMT in a state I want to save now, push all of the blocks from this root into IPFS and discard any cruft that isn't linked to that root". So unless you want to try and build that yourself then you will just have to lean on IPFS to do its GC thing and use the root of the HAMT for pinning or other management of the DAG in the state you care about.

eduardonunesp commented 3 years ago

Thanks for the guidance @rvagg, I started a simple example nothing fancy, just to see the stuff working together and later trying to improve as you described above, for instance the following example, executes as expected, but when I try to check the dag on local IPFS it never returns the result.

import IpfsHttpClient from 'ipfs-http-client'
import { create } from 'ipld-hashmap'
import { sha256 as blockHasher } from 'multiformats/hashes/sha2'
import * as blockCodec from '@ipld/dag-cbor'

async function main() {
  const ipfsAPI = IpfsHttpClient.create('http://localhost:5001')

  // IPFS basic store
  const store = {
    async get (k) {
      return await ipfsAPI.block.get(k)
    },
    async put (k, v) {
      const encodedBlock = blockCodec.encode(v)
      return await ipfsAPI.dag.put(encodedBlock, k)
    }
  }

  const map = await create(store, { blockHasher, blockCodec })
  await map.set('foo', 'bar')
  return map.cid
}

main()
  .then(result => console.log(result))
  .catch(err => console.error(err))

It returns the CID(bafyreihzbbfyhewbvku3pmqmkrwqmv5chkroff33kzu5padzw5minyubeq)

And the following command get stuck on terminal

ipfs dag get /ipld/bafyreihzbbfyhewbvku3pmqmkrwqmv5chkroff33kzu5padzw5minyubeq