search

ipfs / notes

IPFS Collaborative Notebook for Research
MIT License
401 stars 30 forks source link

Support non-recursive links #230

Open mitar opened 7 years ago

mitar commented 7 years ago

Currently, objects can have only one type of links, which, when I pin the object recursively, they are also traversed. I would like to propose that there is another type of links, where this does not happen (or happen with some "force full recursion" switch or command).

I am trying to store blockchain on the IPFS. And I have two types of links in each block: to the parent block, and to transactions. Now, I would like to mark transactions as non-recursive links. So that I can easily pin the whole chain into local storage by pinning recursively the top block. But I do not want also all transactions ("data") to get pinned, because transactions could point to huge data.

Currently, I have to manually traverse the whole chain, but this means that I have to read the whole chain to the client. It would be great if this could be simply done inside the IPFS daemon and I could just do one command to pin (or unpin) the chain.

jbenet commented 7 years ago

@mitar would this be solved by IPLD selectors?

The goal is to be able to do a pin that pins with a selector, like:

# pin all blocks, but not the txns
($root)(/parent)+

that may work.

note the IPLD selectors language is unspecified, because we need to define the syntax. we are very close though-- if you need this asap, we could (you, @nicola, and I) finalize this in a couple of weeks.

mitar commented 7 years ago

Yea, some type of selectors could also work. Not sure how good abstraction this is, because it requires then the caller to understand/anticipate the DAG structure being pinned (imagine that somebody makes a link to a block in my blockchain, and pins that object with that link, I might not want the whole blockchain to get pinned and all other objects linked from it, but just the blockchain itself). So in some way, the person creating an object (blocks in my case) probably knows better which linked objects are reasonable to be pinned when pinning recursively, and which are not. And not the piner/caller itself.

I do not need this urgently. I solved it with some caching and changing of my data structure to make it easier to traverse and not having to store everything. I have to do traverse for validating the chain anyway, before pinning. I think it works well for my current goal which is a blockchain demo, but for real production grade blockchain probably something like this described in this issue would be helpful.

nicola commented 7 years ago

@mitar in my experimental repo, you can use the regex strategy to achieve that (it is not really the optimal implementation) and needs a lot of re-design and help - but it's a playground for this type of experimentation.

I would also link to this: https://github.com/ipld/specs/issues/35 (look for the blockchain headers case)