HTTP API reports incorrect size after adding via /api/v0/add

iostat commented 6 years ago

Version information:

$ ipfs version --all
go-ipfs version: 0.4.15-
Repo version: 6
System version: amd64/darwin
Golang version: go1.10

Type:

Bug

Description:

The HTTP API reports an incorrect size for a file added to IPFS via the /api/v0/add endpoint.

To reproduce: 0) Clean slate: rm -rf ~/.ipfs && ipfs init && ipfs daemon & 1) Create a file named testfile containing just the contents:Plz add me!<LF>. This is exactly 12 bytes. 2) POST to /api/v0/add to add the file. 3) Observe that the response says the Size is 20. 4) Read it back via ipfs cat Qma4hjFTnCasJ8PVp3mZbZK5g2vGDT4LByLJ7m8ciyRFZP | wc -c and observe it was correctly stored with its 12 byte length.

Sample session follows:

$ rm -rf ~/.ipfs && ipfs init && ipfs daemon &
initializing IPFS node at /Users/io/.ipfs
generating 2048-bit RSA keypair...done
peer identity: Qmc7dLt8afHiDtoE3VabiGUUiq637X8i8CPCkx936C6URM
to get started, enter:

    ipfs cat /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv/readme

[1] 12968
Initializing daemon...
Swarm listening on /ip4/10.105.46.40/tcp/4001
Swarm listening on /ip4/10.140.0.2/tcp/4001
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
Swarm listening on /p2p-circuit/ipfs/Qmc7dLt8afHiDtoE3VabiGUUiq637X8i8CPCkx936C6URM
Swarm announcing /ip4/10.105.46.40/tcp/4001
Swarm announcing /ip4/10.140.0.2/tcp/4001
Swarm announcing /ip4/127.0.0.1/tcp/4001
Swarm announcing /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
$ cat testfile
Plz add me!
$ cat testfile | wc -c
      12
$ cat testfile | wc -l
       1
$ cat testfile | xxd
00000000: 506c 7a20 6164 6420 6d65 210a            Plz add me!.
$ ipfs add testfile
added Qma4hjFTnCasJ8PVp3mZbZK5g2vGDT4LByLJ7m8ciyRFZP testfile
$ curl -F arg=@testfile "http://localhost:5001/api/v0/add"
{"Name":"testfile","Hash":"Qma4hjFTnCasJ8PVp3mZbZK5g2vGDT4LByLJ7m8ciyRFZP","Size":"20"}
$ ipfs cat Qma4hjFTnCasJ8PVp3mZbZK5g2vGDT4LByLJ7m8ciyRFZP | wc -c
      12
$ ipfs cat Qma4hjFTnCasJ8PVp3mZbZK5g2vGDT4LByLJ7m8ciyRFZP | xxd
00000000: 506c 7a20 6164 6420 6d65 210a            Plz add me!.

Note that building a 0.4.16 master correctly shows 12 B in the ipfs add command, but still reports "Size": "20" over the HTTP API.

ivan386 commented 6 years ago

Size is the size of all data behind this hash. Your message packed in dag node and size of dag node with your message is 20b. You can check it by: ipfs block get Qma4hjFTnCasJ8PVp3mZbZK5g2vGDT4LByLJ7m8ciyRFZP

You can use /api/v0/add?raw-leaves=true

With raw-leaves data less than 256kb Size will be same as data size.

iostat commented 6 years ago

Thanks for the reply! I figured after opening this issue that that's what's happening here. Seems very counterintuitive for anyone who doesn't know much of IPFS internals, though. Is this perhaps something that is/should be clarified in docs somewhere?

ipfs / kubo