search

ipfs / js-ipfs

IPFS implementation in JavaScript
https://js.ipfs.tech
Other
7.44k stars 1.25k forks source link

Add support for Rabin fingerprinting to js-ipfs #1283

Closed daviddias closed 6 years ago

daviddias commented 6 years ago

For feature parity with go-ipfs

https://github.com/ipfs/notes/issues/1

gitcoinbot commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 0.4 ETH (247.87 USD @ $619.68/ETH) attached to it.

vs77bb commented 6 years ago

Hi @diasdavid, Vivek from Gitcoin here. We posted a small bounty on this on Gitcoin to see if it brings some attention. If you'd like any other bounties, let us know - we'd love to support!

gitcoinbot commented 6 years ago

@rafaelboyero Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

timxor commented 6 years ago

Still need help on this?

haraldrudell commented 6 years ago

The ECMAScript client is in need of a rewrite: it uses some pre-2015 non-standard streams implementation, closures and dictionaries of functions when today's answer would be async iterators. Not a class in sight

with the go client, one can do ipfs add --chunker=rabin-48-96-192 LICENSE added QmaJfZamtgZAPgYfPe5HbjEzbgf3YHgCUCu7yfDadwNXSd LICENSE 1.06 KiB / 1.06 KiB [================================================] ipfs object links QmaJfZamtgZAPgYfPe5HbjEzbgf3YHgCUCu7yfDadwNXSd QmYcQE8seMKSqWq2GZfcpnfK3WgPXLqhtryNEfRX6TXqsy 185
QmdH3L1zu7hWu7nrpoCe33zfn3hDm6baYpTSENbRitpcFu 140
Qmaw7roFAKSj3V3QxmFUfpqU3dqAEMPMqHynqXxJmhhGhB 64
Qmewm77gB4V7cgnZLsuikXgrG8Q38M1TX8qPRBcKrw4yJ4 90
QmRd6EoLAyp6TXZmSRTByaAgEn5632J8u9NwFSiHfXurLx 95
QmeLhozoTP1z3RiLRBRUuLJ7xpmDh5oi1ShN91B4UPjRwA 172
QmdxhoXpK74gGbm2G8bAoCiuqDfbq44rN6RZKefCAFEhoZ 99
QmZNEhqU6uVjMc7p7xfqfwQK5YGxe1bdUNdG6Hq81149iw 139
Qmd1yFj2ZuNd28r5MaQaWkr3vcUwPF61i8oBdfRjaUC4V5 64
QmeJhcwipWbjXA5DdXVGe6Sh8kwBzADNnw3Zd3HKcPtPkm 82
Qme1G8Mk1PLThzCr11NzChQVy1FJoHCQkjK7rB2iTWKkaJ 58

spm32 commented 6 years ago

Hey @tcsiwula - yes! If you'd still like to take it on let me know.

mikeal commented 6 years ago

https://www.npmjs.com/package/rabin

On Fri, Jul 13, 2018, 9:41 AM Scott Moore notifications@github.com wrote:

Hey @tcsiwula https://github.com/tcsiwula - yes! If you'd still like to take it on let me know.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ipfs/js-ipfs/issues/1283#issuecomment-404767964, or mute the thread https://github.com/notifications/unsubscribe-auth/AAACQxuqhUciU8BCkcaplXmDWsFi06_Rks5uGF1RgaJpZM4S9D0X .

dordille commented 6 years ago

Looking for help on this? Took a look at the rabin module linked by @mikeal and was wondering if there is any concern about compatibility with the browserify build of js-ipfs, not too familiar with that process and whether modules that have c dependencies are included elsewhere.

mikeal commented 6 years ago

While there isn't a browser version we can just set the browser field in package.json to{'rabin': null}. That way, in the browser, the require() statement will just return null and we can check for it and opt out of these feature later in the code.

dordille commented 6 years ago

Submitted datproject/rabin/pull/24 to allow for configuring the window size and polynomial used by the chunker. This will allow chunking used by js-ipfs to match go-ipfs-chunker.

Will have PRs up shortly for ipfs/js-ipfs-unixfs-engine and ipfs/js-ipfs

mikeal commented 6 years ago

Should we consider surfacing these chunker settings in unixfs so that, when files are changed, they can be re-chunked to more closely match the prior chunking?

dordille commented 6 years ago

I'd have to defer to someone that knows more about how often settings like window and the polynomial will be changed. Looking at the equivalent go source, it doesn't seem like there is a practical way to alter the window size or the polynomial used when chunking.

Polynomial: https://github.com/ipfs/go-ipfs-chunker/blob/master/rabin.go#L11 Window: https://github.com/whyrusleeping/chunker/blob/master/chunker.go#L16

It would be pretty straightforward though to update the chunkers in unixfs to export default values used in addition to the stream handler.

gitcoinbot commented 6 years ago

⚡️ A tip worth 0.40000 ETH (106.31 USD @ $265.77/ETH) has been granted to @dordille for this issue from @vs77bb. ⚡️

Nice work @dordille! To redeem your tip, login to Gitcoin at https://gitcoin.co/explorer and select 'Claim Tip' from dropdown menu in the top right, or check your email for a link to the tip redemption page.

gitcoinbot commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

The funding of 0.4 ETH (106.31 USD @ $265.77/ETH) attached to this issue has been approved & issued.

vs77bb commented 6 years ago

Hi @dordille great work here. We tipped you for your work via the Gitcoin bounty which was posted on the issue 🙂