search

ipfs / js-ipfs

IPFS implementation in JavaScript
https://js.ipfs.tech
Other
7.44k stars 1.25k forks source link

determine the files to be different sizes #3209

Closed bluelovers closed 1 year ago

bluelovers commented 4 years ago

Severity:

Description:

https://discuss.ipfs.io/t/why-has-same-sub-cids-but-not-same-root-cid/8810/3

Steps to reproduce the error:

upload file by ipds-desktop and https://share.ipfs.io/

aschmahmann commented 4 years ago

@achingbrain @olizilla

I'm not sure if this is because share.ipfs.io is out of date or not. However, I did try adding the same files from the link above into go and js-ipfs ipfs add -r theDirectory and got the same result which is good, and indicates it may be a share.ipfs.io issue (it hasn't been updated in a while).

aschmahmann commented 4 years ago

Perhaps an unrelated issue, is that it looks like jsipfs files stat does not seem to be calculating cumulative size correctly for directories. If it's an unrelated issue feel free to delete this comment and put it in a new issue.

PS /home/adin/workspace/tmp/full/QmbHK4rKG1p87DTnemXrhh9QSxZk6xBCSHg5E985MEH1Zu> jsipfs files ls /
hof.zip
php-5.6.32-nts-Win32-VC11-x64.zip
PS /home/adin/workspace/tmp/full/QmbHK4rKG1p87DTnemXrhh9QSxZk6xBCSHg5E985MEH1Zu> jsipfs files stat /       
QmPq9t1UbjT51Y2bo4dc84KKXtHqD5KnD4vDNMqeAertvi
Size: 0
CumulativeSize: 8070
ChildBlocks: 2
Type: directory
Mode: drwxr-xr-x
Mtime: -
PS /home/adin/workspace/tmp/full/QmbHK4rKG1p87DTnemXrhh9QSxZk6xBCSHg5E985MEH1Zu> jsipfs files stat /hof.zip
QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH
Size: 20010278
CumulativeSize: 20015062
ChildBlocks: 77
Type: file
Mode: -rw-r--r--
Mtime: -
bluelovers commented 4 years ago

temp/111.png => https://explore.ipld.io/#/explore/QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9

import { outputFile } from 'fs-extra';
import fetchIPFS from 'fetch-ipfs';
import { statSync } from 'fs';
import { publishToIPFSAll } from 'fetch-ipfs/lib/put/all';
import { filterList } from 'ipfs-server-list';

console.log(`fs.stat`, statSync('./temp/111.png').size)

fetchIPFS(`QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9`)
    .then(async (buf) => {

        console.log(`fetch ipfs cid`, buf.length, `QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9`)

        await publishToIPFSAll(buf, [
            ...filterList('API'),
        ])
            .then(result => {
                console.log(`add to ipfs again`, result[0].value[0].size, result[0].value[0].cid.toString());
            })
        ;

        return outputFile('./temp/111.png', buf)
    })
    .catch(e => console.trace(e))
;

will see file size is not same 98734 vs 98748

fs.stat 98734
fetch ipfs cid 98734 QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9
add to ipfs again 98748 QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9
bluelovers commented 4 years ago

@aschmahmann i think not only dir, also happen on file

achingbrain commented 4 years ago

Here is what I see:

Bare filesystem

$ ls -la dir
total 84992
drwxr-xr-x  4 alex  staff       128  5 Aug 11:58 .
drwxr-xr-x  4 alex  staff       128  5 Aug 13:25 ..
-rw-r--r--  1 alex  staff  20010278  5 Aug 11:58 hof.zip
-rw-r--r--  1 alex  staff  23016923  5 Aug 11:58 php-5.6.32-nts-Win32-VC11-x64.zip

go-IPFS

Add the directory

$ ipfs add -r dir
added QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH dir/hof.zip
added QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 dir/php-5.6.32-nts-Win32-VC11-x64.zip
added QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn dir

List the root dag

$ ipfs ls /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH 20010278 hof.zip
QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 23016923 php-5.6.32-nts-Win32-VC11-x64.zip

Stat the root of the dag

$ ipfs files stat /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
Size: 0
CumulativeSize: 43037585
ChildBlocks: 2
Type: directory

js-IPFS

Add the directory

$ jsipfs add -r dir
added QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH dir/hof.zip
added QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 dir/php-5.6.32-nts-Win32-VC11-x64.zip
added QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn dir

List the root dag

$ jsipfs ls /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
-rw-r--r-- - QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH 20010278 hof.zip
-rw-r--r-- - QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 23016923 php-5.6.32-nts-Win32-VC11-x64.zip

Stat the root of the dag

$ jsipfs files stat /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
Size: 0
CumulativeSize: 43037585
ChildBlocks: 2
Type: directory
Mode: drwxr-xr-x
Mtime: -

Everything seems in alignment.

@aschmahmann what is QmPq9t1UbjT51Y2bo4dc84KKXtHqD5KnD4vDNMqeAertvi?

@bluelovers the reason the sizes are different is the first is the size of the file, the second is the size of the dag that represents the file:

$ ipfs files stat /ipfs/QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9
QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9
Size: 98734
CumulativeSize: 98748
ChildBlocks: 0
Type: file

See Size vs CumulativeSize.

I'm not familiar with the modules you are using in your example. If you can put an example together that only uses the ipfs module that shows the problem you're seeing that would be helpful.

aschmahmann commented 4 years ago

@aschmahmann what is QmPq9t1UbjT51Y2bo4dc84KKXtHqD5KnD4vDNMqeAertvi?

@achingbrain it's the hash of the MFS root directory after manually copying in the two files into it. See the below walkthrough.

Script of commands

ipfs files ls / # show that it's empty
ipfs files add -r . # add all files (and the directory object) to IPFS noting that hash Qmdvb...
ipfs files cp QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/hof.zip /hof.zip # add the first file to the MFS root (could've used the direct hash instead, but this shows we're definitely adding the right file with the right name)
ipfs files cp /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/php-5.6.32-nts-Win32-VC11-x64.zip /php-5.6.32-nts-Win32-VC11-x64.zip # add the second file to the MFS root
ipfs files ls / # show the directory has the two files now
ipfs files stat / # shows what the MFS root looks like... in go-ipfs it matches the result of ipfs add -r . but in js-ipfs it does not.

Go-IPFS

(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs files ls /
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs add -r .
added QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/hof.zip
added QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/php-5.6.32-nts-Win32-VC11-x64.zip
added QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
 41.03 MiB / 41.03 MiB [====================================================================] 100.00%(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs files cp /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/hof.zip /hof.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs files cp /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/php-5.6.32-nts-Win32-VC11-x64.zip /php-5.6.32-nts-Win32-VC11-x64.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs files ls
hof.zip
php-5.6.32-nts-Win32-VC11-x64.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs files stat /
QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
Size: 0
CumulativeSize: 43037585
ChildBlocks: 2
Type: directory
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ ipfs version
ipfs version 0.6.0

JS-IPFS

(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs files ls /
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs add -r .
added QmYtpgcqDgsxrHhGN2f3xuD8Xw4TeXaVoseQbKJmMa3CBH QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/hof.zip
added QmPRnUadS9kozY1QrCjcYbg5smeAYrruniWQVgstNrv8a4 QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/php-5.6.32-nts-Win32-VC11-x64.zip
added QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs files cp /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/hof.zip /hof.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs files cp /ipfs/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn/php-5.6.32-nts-Win32-VC11-x64.zip /php-5.6.32-nts-Win32-VC11-x64.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs files ls
hof.zip
php-5.6.32-nts-Win32-VC11-x64.zip
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs files stat /
QmPq9t1UbjT51Y2bo4dc84KKXtHqD5KnD4vDNMqeAertvi
Size: 0
CumulativeSize: 8070
ChildBlocks: 2
Type: directory
Mode: drwxr-xr-x
Mtime: -
(base) adin@adin-ubuntu:~/workspace/tmp/full/QmdvbTgdaoqYV2rmh6raYT5pWZ7Lujhz73M6xtaRcC5xXn$ jsipfs version
js-ipfs version: 0.48.1
bluelovers commented 4 years ago 
import ipfsClient from 'ipfs-http-client';
import IPFS from 'ipfs';

const cid = `QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9`;

(async () =>
{

    let ipfs = await ipfsClient({
        port: 5001,
    });

    let ipfs2 = await IPFS.create();

    await doTest(ipfs);

    console.log(`--------------------`)

    await doTest(ipfs2);

})();

async function doTest(ipfs)
{
    let chunks: Buffer[] = [];
    for await (let chunk of ipfs.cat(cid))
    {
        chunks.push(chunk)
    }
    let buf = Buffer.concat(chunks)

    console.dir(buf.length)

    for await (let ret of ipfs.addAll(buf))
    {
        console.log(ret)
    }
}
98734
{
  path: 'QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9',
  cid: CID(QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9),
  size: 98748
}
--------------------
98734
{
  path: 'QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9',
  cid: CID(QmdPAhQRxrDKqkGPvQzBvjYe3kU8kiEEAd2J6ETEamKAD9),
  size: 98748,
  mode: 420,
  mtime: undefined
}
olizilla commented 4 years ago

Related https://github.com/ipfs-shipyard/ipld-explorer/issues/40 - ipld explorer always reports the size for dag-pb nodes as the value of node.size, it doesn't currently try to match the logic of ipfs files stat

SgtPooki commented 2 years ago

Per @olizilla in https://github.com/ipld/explore.ipld.io/issues/40#issuecomment-669488903

Looking at the latest code for js-ipfs that seems like it should be updated to take into account the unixfs data, and either return 0 for directories or the value of unixfs.fileSize()

Next actions:

  1. Ensure sizes are consistently reported, regardless of node/encoding type.
  2. Ensure FAQ/documentation is updated to indicate variances such as these, and why they are not able to be resolved.
SgtPooki commented 1 year ago

js-ipfs is being deprecated in favor of Helia. You can learn more about this deprecation and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterwards (see https://github.com/ipfs/js-ipfs/issues/4336).

FYI that share.ipfs.io will be migrating to Helia soon: https://github.com/ipfs-shipyard/ipfs-share-files/issues/136

ipld-explorer-components and explore.ipld.io are already migrating off of js-ipfs: https://github.com/ipfs/ipld-explorer-components/pull/360