search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
16.2k stars 3.03k forks source link

CoreAPI: ObjectStat and DataSize #5325

Open djdv opened 6 years ago

djdv commented 6 years ago

When trying to migrate data out of IPFS with the CoreAPI, I noticed that an ObjectStat's DataSize member, is the size for the object's entire data segment, not just the size of the raw data stored under the type/format. For example, current UnixFS nodes storing the data "IPFS", will be larger than 4bytes.

I believe adding a .RawDataSize member, would be valuable, for people who wish to migrate data out of IPFS. It would allow them to consider the data segment's raw/target data size, and make decisions based around it, when dealing with foreign filesystems. For example, when dealing with file allocation sizes on an external drive of another format, prior to calling Cat().

My current workaround seems suboptimal: reference

Ping @magik6k, @schomatis

schomatis commented 6 years ago

Yes, I think that rather than modify the ObjectAPI, is the UnixfsAPI that should be extended to get information regarding file size and similar (as the first deals only with the DAG layer and doesn't know that a file might be encoded inside of a DAG node, but the second one should).

Stebalien commented 6 years ago

For context, the object API is for interacting with merkledag objects (it's not even generalized over all IPLD objects), which may not be unixfs nodes. Basically, all this API knows is that we have some "object" with a data field (raw bytes) and a links field (list of links).

Unfortunately, we don't really have a good "unixfs" equivalent, as fr as I know.

magik6k commented 6 years ago

I haven't verified this, but api.Unixfs().Get().(files.SizeFile).Size() should be returning the right size - https://github.com/ipfs/go-ipfs/blob/master/core/coreapi/unixfile.go#L122

schomatis commented 6 years ago

Yes, going through the DagReader seems the correct way.