search

rs-ipfs / rust-ipfs

The InterPlanetary File System (IPFS), implemented in Rust.
Apache License 2.0
1.26k stars 163 forks source link

ipfs-unixfs: other layouts, other chunkers, more options #222

Open koivunej opened 4 years ago

koivunej commented 4 years ago

The obvious missing features:

In addition to this there could be other conditions and parameters even to the balanced collector implemented in #220. If you are interested working on any of these, just comment down below; new issue can be created or multiple PRs could be done against this one.

Writing this right after #220 I think the trickle layout will be the biggest effort since I am sure it will cause at least some changes to the block creation (at least the type: File => Raw for trickle items). At least some mentoring can be given, and there are existing rust projects related to content sensitive chunking which can be leveraged. Integrating those might bring other changes to the current very naive chunker "api".

nuke-web3 commented 4 years ago

:wave: @koivunej - so this is a bit of a looong shot, but I am participating in HackFS and am really interested in creating a wasm stand-alone CID generating library/package as a tool for cli and packaged for use in JS programs as a library. I am brand new to rust, so I will be navigating learning and trying to get this done, but I see this issue being really related, as I hope to have this tool generate a CID based on various chunkers / options.

If you would be willing to help guide me a bit, I would be keen on working on implementing these as I move forward.

Happy to move this convo to another channel as well, lmk.

aphelionz commented 4 years ago

Hey @NukeManDan, we definitely want to help HackFS folks out. Can you describe a little bit more about what kind of CID generation you want to do? What would the inputs be (bytes?) and I assume the output would be a CID?

koivunej commented 4 years ago

The ability to no_std the ipfs-unixfs crate is something we've tried to keep an eye out for. I am not an expert in any things wasm, but with no_std + alloc we should be able to use quick_protobuf as is and the rest of the complications would be very welcome optimizations. With just the optimizations we could end up with WASM thing which takes file bytes and outputs the blocks (or CAR file), possibly hosted in a JS app.

If the above sounds something you'd like to pursue, we should create a new issue on that.

whalelephant commented 4 years ago

Hi, I sort of have similar idea as @NukeManDan for HackFS, but was going to use neon-bindings. But it would be great to see if no_std would be possible. So the idea is to have the input date in bytes and can output a version of CID.

koivunej commented 4 years ago

Hi @whalelephant and @NukeManDan, I created #247 for the no_std topic, lets continue discussion over there.