search

ipfs / go-graphsync

Initial Implementation Of GraphSync Wire Protocol
Other
100 stars 38 forks source link

feat!(messagev2): tweak dag-cbor message schema #354

Closed rvagg closed 2 years ago

rvagg commented 2 years ago

For:

  1. Efficiency: compacting the noisy structures into tuples representations and making top-level components of a message optional.
  2. Migrations: providing a secondary mechanism to lean on for versioning if we want a gentler upgrade path than libp2p protocol versioning.

Closes: https://github.com/ipfs/go-graphsync/issues/351

In terms of what this looks like, an example of an original message is in https://github.com/ipfs/go-graphsync/issues/351 and with this change the same data would look like (as dag-json):

{
  "gs1": {
    "blk": [
      [
        { "/": { "bytes": "AVUSIA" } },
        { "/": { "bytes": "QgTLmh40xfCOmyCqdgkOcAILtWwMo9OvcpbNEFilESiQ/tIYSI8ITY355INftUrQRf/ZNuO/cmGwQmxRNSoJeBbtdEgruQhLSn7YrcUX8zceDgQ0tRFiXNGkF5IkPczc/ogJSw" } }
      ],
      [
        { "/": { "bytes": "AVUSIA" } },
        { "/": { "bytes": "xfPTKlWZ2kO4US4NsU11HtHWQJ2dy3rtBdbMbAmHxjTufEI28FF8INzGqZ6G5LrcAl5fXoG0WWEoUimfNvEQ6Qa6IW4ByDcDUq0ziUrrH+WNzNCC9FaSKBovz42VowZ4zaRVhw" } }
      ]
    ],
    "req": [
      [
        { "/": { "bytes": "GmIWTBeVSRyOznxIsnBumg" } },
        "n",
        101,
        { "/": "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi" },
        {"R":{":>":{"|":[{".":{}},{"a":{">":{"@":{}}}}]},"l":{"none":{}}}},
        {
          "AppleSauce/McGee": "yee haw"
        }
      ],
      [
        { "/": { "bytes": "vd/PRdnYTg27RolghnknVQ" } },
        "n",
        202,
        { "/": "bafyreibdoxfay27gf4ye3t5a7aa5h4z2azw7hhhz36qrbf5qleldj76qfy" },
        {"R":{":>":{"a":{">":{"@":{}}}},"l":{"none":{}}}},
        {}
      ]
    ],
    "rsp": [
      [
        { "/": { "bytes": "GmIWTBeVSRyOznxIsnBumg" } },
        34,
        [
          [
            { "/": "bafyreibdoxfay27gf4ye3t5a7aa5h4z2azw7hhhz36qrbf5qleldj76qfy" },
            "m"
          ]
        ],
        {}
      ],
      [
        { "/": { "bytes": "vd/PRdnYTg27RolghnknVQ" } },
        14,
        [],
        {
          "Hippity+Hoppity": {   "/": {   "bytes": "9V/48SUItj7yv+ynVXrpDfYxGl7BYxtKH6hDMQvZw6cQ6qzlob3XKtC/4El3HBHnVjOL2Thl5kXxreybnJnvQH+9T8aFnnkExa19yb0QpcwWlz1bKOwabdQ9n4L58Yw9A0GONQ"   }   }
        }
      ]
    ]
  }
}

In terms of bytes saved, using the randomish data in TestGraphsyncRoundTrip (i.e. it has some variability in output length and these are not using exactly the same data so consider it approximate):

@mvdan @warpfork care to critique my schema?

Aside: it's interesting that we're even saving bytes written by server from protobuf to dag-cbor even without these changes. The majority of the data sent should be blocks and their CIDs and all these maps with string keys should drown out any saving CBOR gets from its compact int & length representation. Plus we have v1.0 doing int request IDs and v2 doing UUID (as bytes) request IDs so they're longer. Nothing in the protobuf spec is optional, although I believe repeated allows for zero occurrences so in effect the top-level items are optional at least. Perhaps it's all to do with moving the metadata into the core message rather than as an extension (so we save writing the string "graphsync/response-metadata" on each response). An interesting mystery that might be worth investigating at some point.

rvagg commented 2 years ago

Adjusted schema as per feedback above and some discussion today via Zoom. The requests and responses are back to maps with keys because we don't expect them to repeat much and it's nice to have them descriptive. But I've also optionalised more of the fields in there so they could be left out entirely without any pain.

{
    "gs2": {
        "blk": [
            [
                { "/": { "bytes": "AVUSIA" } },
                { "/": { "bytes": "QgTLmh40xfCOmyCqdgkOcAILtWwMo9OvcpbNEFilESiQ/tIYSI8ITY355INftUrQRf/ZNuO/cmGwQmxRNSoJeBbtdEgruQhLSn7YrcUX8zceDgQ0tRFiXNGkF5IkPczc/ogJSw" } }
            ],
            [
                { "/": { "bytes": "AVUSIA" } },
                { "/": { "bytes": "xfPTKlWZ2kO4US4NsU11HtHWQJ2dy3rtBdbMbAmHxjTufEI28FF8INzGqZ6G5LrcAl5fXoG0WWEoUimfNvEQ6Qa6IW4ByDcDUq0ziUrrH+WNzNCC9FaSKBovz42VowZ4zaRVhw" } }
            ]
        ],
        "req": [
            {
                "ext": {
                    "AppleSauce/McGee": "yee haw"
                },
                "id": { "/": { "bytes": "k3Nu8gWHTfq53ypWnVFtwg" } },
                "pri": 101,
                "root": { "/": "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi" },
                "sel": {"R":{":>":{"|":[{".":{}},{"a":{">":{"@":{}}}}]},"l":{"none":{}}}},
                "type": "n"
            },
            {
                "id": { "/": { "bytes": "mdBpy6/wQyqT1LiaihfQWQ" } },
                "pri": 202,
                "root": { "/": "bafyreibdoxfay27gf4ye3t5a7aa5h4z2azw7hhhz36qrbf5qleldj76qfy" },
                "sel": {"R":{":>":{"a":{">":{"@":{}}}},"l":{"none":{}}}},
                "type": "n"
            }
        ],
        "rsp": [
            {
                "meta": [
                    [
                        { "/": "bafyreibdoxfay27gf4ye3t5a7aa5h4z2azw7hhhz36qrbf5qleldj76qfy" },
                        "m"
                    ]
                ],
                "reqid": { "/": { "bytes": "k3Nu8gWHTfq53ypWnVFtwg" } },
                "stat": 34
            },
            {
                "ext": {
                    "Hippity+Hoppity": { "/": { "bytes": "9V/48SUItj7yv+ynVXrpDfYxGl7BYxtKH6hDMQvZw6cQ6qzlob3XKtC/4El3HBHnVjOL2Thl5kXxreybnJnvQH+9T8aFnnkExa19yb0QpcwWlz1bKOwabdQ9n4L58Yw9A0GONQ" } }
                },
                "reqid": { "/": { "bytes": "mdBpy6/wQyqT1LiaihfQWQ" } },
                "stat": 14
            }
        ]
    }
}

In terms of bytes sent, the difference is negligible when averaged out. We're sending a few more bytes for the map keys but not enough of them and the blocks are still tuples, we're also saving on optional fields by skipping them entirely.

Unfortunately there's more pointers in here than I'd like, getting bindnode working with implicit values that don't need to be backed by pointers might be nice to help with that, but tricky (/cc @mvdan - e.g. optional but implicit 0 should mean I don't need a pointer field for that).