search

fission-codes / fission

Fission CLI & server
https://runfission.com/docs
119 stars 14 forks source link

Update endpoint for dag size result on app/user updates #603

Open walkah opened 2 years ago

walkah commented 2 years ago

Summary

Problem

The fission-server is currently using a deprecated IPFS API to size usage apps and user file systems.

Impact

This means there is incorrect reporting in the database for how much space an app or user is using.

Solution

We should replace the call to the (deprectaed) /api/v0/object/stat endpoint (https://docs.ipfs.io/reference/http/api/#api-v0-object-stat) with a version that is "DAG" aware (that isn't re-counting common blocks in the DAG). The documentation recommends /api/v0/files/stat (but /api/v0/dag/stat might be more appropriate?).

See also: https://github.com/fission-suite/go-ipfs/issues/56

justindotpub commented 2 years ago

Looks like /api/v0/dag/stat is the one we want, though it has performance implications since it traverses the DAG, skipping duplicates.

# one file in a directory, therefore no dupe files
»  random-files --dirs 0 --files 1 --filesize 4096 blah
blah/rnlq6zl7wr11

»  ipfs add -r blah
added QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2 blah/rnlq6zl7wr11
added Qma5pU4piKaN84xzafFK8QGEZYThcDSqpQLptGLod5qVFK blah
 3.49 KiB / 3.49 KiB [==========================================================================================================] 100.00%

# file
»  ipfs dag stat QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2                              
Size: 3585, NumBlocks: 1

»  ipfs files stat /ipfs/QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2
QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2
Size: 3574
CumulativeSize: 3585
ChildBlocks: 0
Type: file

# directory
»  ipfs dag stat Qma5pU4piKaN84xzafFK8QGEZYThcDSqpQLptGLod5qVFK                                              
Size: 3644, NumBlocks: 2

»  ipfs files stat /ipfs/Qma5pU4piKaN84xzafFK8QGEZYThcDSqpQLptGLod5qVFK
Qma5pU4piKaN84xzafFK8QGEZYThcDSqpQLptGLod5qVFK
Size: 0
CumulativeSize: 3644
ChildBlocks: 1
Type: directory

# two file in a directory, both files dupes with diff names.
»  cp blah/rnlq6zl7wr11 blah/rnlq6zl7wr11-dupe1        

»  ipfs add -r blah                                                    
added QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2 blah/rnlq6zl7wr11
added QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2 blah/rnlq6zl7wr11-dupe1
added QmTPQPCBWHS9SSo2QKErK87LzJcJMw2PaffWfP9dKTBPer blah
 6.98 KiB / 6.98 KiB [=================================================================================================================================================================================================================================================================] 100.00%

# file one and two have same CID, so same output here
»  ipfs dag stat QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2                                              
Size: 3585, NumBlocks: 1

»  ipfs files stat /ipfs/QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2                                              
QmQG34hwzQPbPbyJfVMZGeMZcJuMyEbkXD3xKm6MeZ4CA2
Size: 3574
CumulativeSize: 3585
ChildBlocks: 0
Type: file

# dag stat of directory is only slightly larger than last time, still with 2 blocks
»  ipfs dag stat QmTPQPCBWHS9SSo2QKErK87LzJcJMw2PaffWfP9dKTBPer                                              
Size: 3705, NumBlocks: 2

# files stat of dir is twice size of file (7,148), plus 142 bytes for metadata?
»  ipfs files stat /ipfs/QmTPQPCBWHS9SSo2QKErK87LzJcJMw2PaffWfP9dKTBPer                                              
QmTPQPCBWHS9SSo2QKErK87LzJcJMw2PaffWfP9dKTBPer
Size: 0
CumulativeSize: 7290
ChildBlocks: 2
Type: directory