search

monadic-xyz / ipfs

Haskell libraries for interacting with IPFS
https://ipfs.io
BSD 3-Clause "New" or "Revised" License
26 stars 6 forks source link

Support for IPLD Data Model and Codecs #59

Open tysonzero opened 3 years ago

tysonzero commented 3 years ago

I was looking to manipulate and serialize/deserialize IPLD objects, and I came across a few of the libraries listed here in the process.

It looks like the current set of libraries does not include the above, so I figured I'd make an issue in case you thought it was in scope.

I may end up making the libraries myself in my free time, likely on top of some of your libraries, in which case I'll open source them and we can discuss if it's worth merging them into your repo later.

tysonzero commented 3 years ago

Basic ipld-data-model and dag-cbor libraries pushed here

kim commented 3 years ago

I must admit that I haven’t followed up with how libraries/applications are supposed to handle IPLD. Afaics, the IPFS API is still fairly untyped as far as the data payloads are concerned. Is the idea to have some higher-level wrappers to encode/decode from/to the data the IPFS API provides?

tysonzero commented 3 years ago

All data stored in IPFS is stored in IPLD DAG form, which is reasonably structured but not super typed (think JSON), the ipfs dag api interacts more directly with this form.

The ipfs files api is a layer on top of IPLD that converts files and folders to and from this DAG form. In some cases (like my own) it is nicer to skip this abstraction and interact directly with the DAG instead.

kim commented 3 years ago

All data stored in IPFS is stored in IPLD DAG form, which is reasonably structured but not super typed (think JSON), the ipfs dag api interacts more directly with this form.

Yes, that doesn't seem to have changed since last time I worked with it. My question is more, if you wanted to contribute the types you defined to one of the packages in this repo, where would it go?

The ipfs-api's servant types are very close to what go-ipfs exposes (they were, in fact, generated from that). So I suppose one would use those types in a layer on top of ipfs-api?

tysonzero commented 3 years ago

From a spec standpoint they are a layer underneath IPFS, and you can see this in practice with Filecoin, which builds on top of IPLD but not on top of IPFS (common misconception).

There will be other situations where you want to use IPLD but not IPFS, for example most crypto blockchains can be pulled into IPLD (using a different codec then dag-cbor one I provided), but they know nothing about IPFS.

For that reason they should likely be towards the bottom of the dependency graph.

I think that my approach is reasonable: ipld-data-model package which depends on ipld-cid, dag-cbor which depends on ipld-data-model.

I could also see it being reasonable to merge ipld-cid and ipld-data-model into a single package. Although I do think certain extensions to ipld like schemas are complex enough to justify a separate package built on top.

Due to the sizable amount of codecs I don't think they should all be in the same package, but with that said dag-cbor is an important one, so it could also feasibly be merged in to ipld-data-model.

As for how to best handle ipfs-api, it seems like in an ideal world it would just directly use those types, as it'd be more type safe that way. However due to the code generation that might be a pain. I'm actually rather surprised that go-ipfs doesn't use more precise types, then again it is go. So I suppose something like ipfs-dag-api might be needed, although the existing block api is likely fine, with the end user converting to and from bytestrings.