`jsipfs add` has defaults differing from `ipfs add`

[ribasushi]

In go-ipfs raw-leaves defaults to true when cid-version=1 is specified. Additional the defaults for rabin are identical to rabin-262144 which translates to rabin-87381-262144-393216

However js-ipfs does otherwise, observe:

go  chunker:rabin   rl: bafybeifgfx3nkx6lfyrzivtqmeqxuitds7t7k6t2anudptymhsfbmpsvzu
js  chunker:rabin   rl: bafybeictpvoociikd5pm4vnqtbedccef5gohdjya6ly726csvwm6jy3xea
go  chunker:rabin   rl:--raw-leaves=true    bafybeifgfx3nkx6lfyrzivtqmeqxuitds7t7k6t2anudptymhsfbmpsvzu
js  chunker:rabin   rl:--raw-leaves=true    bafybeicos3bc7li5wahssh2qs2gsdokcjmce3wymusdm6jrmm6wvw6vf2i
go  chunker:rabin   rl:--raw-leaves=false   bafybeierguac6kvgj2plzgsqkpwkg2czmwwhcoccrkdsjgc5loucwjawje
js  chunker:rabin   rl:--raw-leaves=false   bafybeictpvoociikd5pm4vnqtbedccef5gohdjya6ly726csvwm6jy3xea
go  chunker:rabin-262144    rl: bafybeifgfx3nkx6lfyrzivtqmeqxuitds7t7k6t2anudptymhsfbmpsvzu
js  chunker:rabin-262144    rl: bafybeictpvoociikd5pm4vnqtbedccef5gohdjya6ly726csvwm6jy3xea
go  chunker:rabin-262144    rl:--raw-leaves=true    bafybeifgfx3nkx6lfyrzivtqmeqxuitds7t7k6t2anudptymhsfbmpsvzu
js  chunker:rabin-262144    rl:--raw-leaves=true    bafybeicos3bc7li5wahssh2qs2gsdokcjmce3wymusdm6jrmm6wvw6vf2i
go  chunker:rabin-262144    rl:--raw-leaves=false   bafybeierguac6kvgj2plzgsqkpwkg2czmwwhcoccrkdsjgc5loucwjawje
js  chunker:rabin-262144    rl:--raw-leaves=false   bafybeictpvoociikd5pm4vnqtbedccef5gohdjya6ly726csvwm6jy3xea
go  chunker:rabin-87381-262144-393216   rl: bafybeifgfx3nkx6lfyrzivtqmeqxuitds7t7k6t2anudptymhsfbmpsvzu
js  chunker:rabin-87381-262144-393216   rl: bafybeierguac6kvgj2plzgsqkpwkg2czmwwhcoccrkdsjgc5loucwjawje
go  chunker:rabin-87381-262144-393216   rl:--raw-leaves=true    bafybeifgfx3nkx6lfyrzivtqmeqxuitds7t7k6t2anudptymhsfbmpsvzu
js  chunker:rabin-87381-262144-393216   rl:--raw-leaves=true    bafybeifgfx3nkx6lfyrzivtqmeqxuitds7t7k6t2anudptymhsfbmpsvzu
go  chunker:rabin-87381-262144-393216   rl:--raw-leaves=false   bafybeierguac6kvgj2plzgsqkpwkg2czmwwhcoccrkdsjgc5loucwjawje
js  chunker:rabin-87381-262144-393216   rl:--raw-leaves=false   bafybeierguac6kvgj2plzgsqkpwkg2czmwwhcoccrkdsjgc5loucwjawje

Produced via:

#!/bin/bash

for ch in \
  "rabin" \
  "rabin-262144" \
  "rabin-87381-262144-393216" \
  ; do \
    for rl in \
      "" \
      "--raw-leaves=true" \
      "--raw-leaves=false"; do \
        echo -en "go\tchunker:$ch\trl:$rl\t"; dd if=/dev/zero bs=$((1024*1024)) count=16 2>/dev/null | cmd/ipfs/ipfs add -nq --cid-version=1 --chunker=$ch $rl
        echo -en "js\tchunker:$ch\trl:$rl\t"; dd if=/dev/zero bs=$((1024*1024)) count=16 2>/dev/null | jsipfs add -nq --cid-version=1 --chunker=$ch $rl
    done
done

[ribasushi]

Additional info requested by @hugomrdias to potentially incorporate a slimmed-down version of the rabin chunker into the wasm version

@hugomrdias aside from that: yes: the rabin chunker as written here is complete, and there isn't much more left to squeeze out note that my "intake" interface is slightly different due to being a radically changed framework, so it will take a little massaging to retrofit instead of readers, it expects a buffer and an indication whether "more is coming" or not chunkCB is where it passes each chunk as it finds them https://github.com/ipfs/DAGger/blob/19fc1fd0e8/dagger/chunker/ipfs/rabin/rabin.go#L28-L89

@hugomrdias note I have not looked at the wasm version, so I can't guarantee this will be better, but it likely can be ( I do away with the window buffer entirely )

[achingbrain]

Something that may cause a problem here is that the default polynomial value of 17437180132763653 used by go-IPFS with the rabin chunker is a bigger number than you can represent in JS:

It needs to use long or bignumber.js BigInt to represent it instead.