search

ipld / js-ipld-ethereum

JavaScript Implementation of All Ethereum IPLD formats
MIT License
40 stars 11 forks source link

Eth blocks missing IPLD links #25

Open olizilla opened 6 years ago

olizilla commented 6 years ago

When we transform Eth blocks to objects, it's links aren't expanded out to '/': Buffer style IPLD links, they are just properties that point to Buffers, so when they are run through the IPLD explorer, it can't show any of their linked nodes.

Currently:

screenshot 2018-08-01 11 38 19

If the IPLD stucture was the same as that suggested here in go-ipld-eth https://github.com/ipfs/go-ipld-eth/blob/4199044661a07667f13ec1ca4d228b3741b75b09/plugin/README.md#usage-and-examples 

{
    "bloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "coinbase": "0x4bb96091ee9d802ed039c4d1a5f6216f90f81b01",
    "difficulty": 11966502474733,
    "extra": "0xd783010400844765746887676f312e352e31856c696e7578",
    "gaslimit": 3141592,
    "gasused": 21000,
    "mixdigest": "0x2565992ba4dbd7ab3bb08d1da34051ae1d90c79bc637a21aa2f51f6380bf5f6a",
    "nonce": "0xf7a14147c2320b2d",
    "number": 997522,
    "parent": {
        "/": "z43AaGF24mjRxbn7A13gec2PjF5XZ1WXXCyhKCyxzYVBcxp3JuG"
    },
    "receipts": {
        "/": "z44vkPhjt2DpRokuesTzi6BKDriQKFEwe4Pvm6HLAK3YWiHDzrR"
    },
    "root": {
        "/": "z45oqTRunK259j6Te1e3FsB27RJfDJop4XgbAbY39rwLmfoVWX4"
    },
    "time": 1455362245,
    "tx": {
        "/": "z443fKyLvyDQBBQRGMNnPb8oPhPerbdwUX2QsQCUKqte1hy4kwD"
    },
    "uncles": {
        "/": "z43c7o73GVAMgEbpaNnaruD3ZbF4T2bqHZgFfyWqCejibzvJk41"
    }
}

...where the props are expanded out to IPLD links, we get all the auto linking magic for free

Mr0grog commented 6 years ago

Seems like part of the problem is that the CBOR resolver returns an IPLD-friendly representation when you call its util.deserialize() (which calls the CBOR decoder, which is set up to replace CID-tagged bytestrings with the {"/": "<cid>"} structure)…

…but here in js-ipld-ethereum, the util.deserialize() function only returns the Ethereum-native representation: https://github.com/ipld/js-ipld-ethereum/blob/0e4e0be6e2bf641f28c497bbf1a8e624a24e89a7/util/createUtil.js#L6-L8

…and the transformation to an IPLD-friendly representation only happens during path traversal when reading into the object. Actual mapping: https://github.com/ipld/js-ipld-ethereum/blob/0e4e0be6e2bf641f28c497bbf1a8e624a24e89a7/eth-block/index.js#L9-L32

Where it’s called during traversal (resolve() → resolveFromEthObject() → mapFromEthObject()):

https://github.com/ipld/js-ipld-ethereum/blob/0e4e0be6e2bf641f28c497bbf1a8e624a24e89a7/util/createResolver.js#L59-L74

So that means what you get with ipld.get(<CID of an Eth object>) is the Ethereum library’s representation of the object, not the set of paths the IPLD resolver will traverse for you to get values (big surprise to me!).

It seems like this library would have to be restructured in a pretty big way to change that so util.deserialize() gets you a transformed representation… OR IPLD Explorer is doing the wrong thing by calling ipld.get(cid,...) (which returns the result of util.deserialize()) and should instead be calling ipld.treeStream(cid,...) and then calling ipld.get(cid, path, ...) with each item from the tree stream to build up a representation.

Mr0grog commented 6 years ago

Looks like the Bitcoin and ZCash resolves both follow the same approach (util.deserialize() gets you the underlying library’s parsed representation, not a representation of the paths the IPLD resolver will let you traverse).

Mr0grog commented 6 years ago

(Also, apologies if any of the above was just restating the obvious for folks here. I was surprised that a JS ipld.get(cid) gets a very different sort of result from what a Go coreapi.DagAPI.Get(context, cidPath) gets you.)

olizilla commented 6 years ago

That is super useful info for me @Mr0grog !

Mr0grog commented 6 years ago

Minor update: it looks like the js-ipld-git resolver does not follow this pattern; the deserialized representation matches the paths you can traverse with it: https://github.com/ipld/js-ipld-git/blob/3dac2488dbb3aafe96561b913dd130ccc91384e0/src/util/commit.js#L37-L84

So the disconnect between those exists in the resolvers for Ethereum, Bitcoin, and ZCash, but not in PB, CBOR, and Git.

vmx commented 6 years ago

Thanks @olizilla and @Mr0grog for digging into this. It should indeed return a proper link. Please note that what "proper link" means is currently in flux. @mikeal come up with the idea of making the resolvers return JavaScript Objects and not JSON, i.e. it won't be {"somefield": {"/": "basencoded-string"}} but {"somefield": CIDObject}. I'll open corresponding issues, once at least one format switch to the new "using a CID" which can then follow lead.

Update: Reference to the CID discussions: https://github.com/ipld/ipld/issues/44

Mr0grog commented 6 years ago

See also ipld/js-ipld#141, based on follow-on discussions about this on Friday, 2018-08-03.

Mr0grog commented 6 years ago

OK, to bring this around to concrete, actionable issues, IPLD Explorer is actually doing the right thing here, but there are several bugs in js-ipld and the various resolver modules:

@olizilla all that adds up to:

@vmx should we treat this issue as a tracking bug for this module’s side of ipld/interface-ipld-format#34 or should we close it?

vmx commented 6 years ago

I'm happy to leave this issue open until the IPLD side (most notable https://github.com/ipld/interface-ipld-format/issues/34) is fixed.