CID doesn't match ipfs-desktop CID

[IanPhilips]

Hi, thanks for the library! I've heard a generated CID differs from node to node based on parameters so this question may be too difficult to answer, but do you have any idea why the CID your library generates for a file would be different from the CID generated via adding the same file to ipfs-desktop with the default settings? I've been messing around with your options parameter but still can't produce the same CID.

[olizilla]

How doo, I help maintain IPFS Desktop... under the hood IPFS Desktop calls ipfs.add. This library aims to generate the same CID as you would get from any ipfs add. It also lets you vary the parameters to control the chunking and hashing, but any change to those would lead to a different CID, as you mentioned.

To get a file with the same CID as created in IPFS Desktop you would want to leave all the options to ipfs add or to this hash function of this lib at their default values.

If I add a file via the command like ipfs add /path/to/file or i drop it on IPFS Desktop, the CID I get is the same. If I run that same file through this library, I get the same CID. Can you share a link to your code and the files you are using to test with?

[IanPhilips]

Thanks for the response, here's the repo to see the different CIDs being produced: https://github.com/IanPhilips/jst-cids-test

I load my avatar image via fetch and get one CID, then from the import statement and get another, and then manually via a fresh install of ipfs-desktop I get a third.

[IanPhilips]

Here's the necessary code: if you want to add it to your own project:

  // download from the url below
  import ian from "./23196210.jpg";
  const Hash = require('ipfs-only-hash');

   async testHashes() {
    const url = "https://avatars0.githubusercontent.com/u/23196210?s=460&v=4";
    const toDataURL = fetch(url)
        .then(response => response.blob())
        .then(blob => new Promise((resolve, reject) => {
            const reader = new FileReader();
            reader.onloadend = () => resolve(reader.result);
            reader.readAsDataURL(blob)

        })).then(async dataUrl =>{

            const strData = dataUrl as string;
            const data = Buffer.from(strData);
            const hash = await Hash.of(data);
            console.log("fetch data hash: " + hash); // QmQAawPBfrhUsMMN7bCAVDHgm47NdpTsxGjMiVvtddoFo1

        });

    const data = Buffer.from(ian);
    const hash = await Hash.of(data);
    console.log("import image data hash: " + hash); // QmWFFQkkxxg4WgSJG6zvJR5wD4uM8jbPCWhfoEsppDPKjw
    console.log("ipfs-desktop hash: " + "QmYHzA8euDgUpNy3fh7JRwpPwt6jCgF35YTutYkyGGyr8f"); // QmWFFQkkxxg4WgSJG6zvJR5wD4uM8jbPCWhfoEsppDPKjw
}

[IanPhilips]

The Buffer.from(ian) shouldn't work as that's actually an image location, but I'm not sure why the fetched image as a string doesn't work, maybe it's encoded differently than the file?

[IanPhilips]

I'm thinking this is just a difference in data encoding, currently trying to figure out how I can make the data types match

[olizilla]

FileReader.readAsDataURL will return a base64 encoded string that is prefixed with a data url declaration data:*/*;base64, see: https://developer.mozilla.org/en-US/docs/Web/API/FileReader/readAsDataURL

You can get an arrayBuffer from the fetch response, and you should be able to pass that yo Hash.of. You may still need to call Buffer.from on it first.

[olizilla]

also note that there has been a significant update to this lib as of yesterday https://github.com/alanshaw/ipfs-only-hash/commit/78189a4a0ba9e5367e62b5dfe9c93b469792199a

[IanPhilips]

Hoooooray!!!! Thanks for the help @olizilla! In order to properly decode the image's base64string ya have to pass "base64" to the Buffer.from() call. Not sure if Buffer would've caught the encoding if I hadn't cleaned the string.

OK this is how to match an image's CID downloaded via fetch to the CID generated from ipfs-desktop for the same image (added as a file from the local drive):

async testHashes() {
    const url = "https://raw.githubusercontent.com/IanPhilips/jst-cids-test/master/src/23196210.jpg";
    fetch(url)
        .then(response => response.blob())
        .then(blob => new Promise((resolve, reject) => {
            const reader = new FileReader();
            reader.onloadend = () => resolve(reader.result);
            reader.readAsDataURL(blob)

        })).then(async dataUrl =>{

            const strData = dataUrl as string;
            // remove "data:*/*;base64," from dataUrl
            const endOfPrefix = strData.indexOf(",");
            const cleanStrData = strData.slice(endOfPrefix+1);
            const data = Buffer.from(cleanStrData, "base64");
            const hash = await Hash.of(data);
            console.log("fetch data CID: " + hash); // QmYHzA8euDgUpNy3fh7JRwpPwt6jCgF35YTutYkyGGyr8f
        });

    console.log("ipfs-desktop CID: QmYHzA8euDgUpNy3fh7JRwpPwt6jCgF35YTutYkyGGyr8f");
}