How does raw ssz vector ser/dser?

[GrapeBaBa]

type MasterAccumulator struct {
    HistoricalEpochs [][]byte `ssz-max:"1897,32"`
}

Must I create a struct for ssz vector? If Yes, what's the difference about a ssz container only have a ssz vector field vs a raw ssz vector? @ferranbt

[ferranbt]

Hello, do you mean:

type A struct {
  Data [][]byte
}

and:

type BytesVector []byte
type B struct {
  Data []BytesVector
}

There is no difference from the code generator standpoint. I recommend option 'A' because it is less complicated for the SSZ generator. This is covered in the testcases.

The benefit of using option 'B' is that you can add functions to the vector object.

[GrapeBaBa]

@ferranbt Sorry i didn't describe clearly, I want to generate a ListCompositeType, but now it generate ContainerType with a field with ListCompositeType.

[ferranbt]

@ferranbt Sorry i didn't describe clearly, I want to generate a ListCompositeType, but now it generate ContainerType with a field with ListCompositeType.

Can you show me the type structs of the specific use case and what do you expect from it?

[GrapeBaBa]

@ferranbt Sorry i didn't describe clearly, I want to generate a ListCompositeType, but now it generate ContainerType with a field with ListCompositeType.

Can you show me the type structs of the specific use case and what do you expect from it?

sure, we are working on portal network Go version, the wire protocol use a complex list type for Enrs, you can check the JS impl https://github.com/ethereumjs/ultralight/blob/master/packages/portalnetwork/src/wire/types.ts#L43.

but we define like this https://github.com/optimism-java/shisui/blob/portal/p2p/discover/portalwire/messages.go#L102,

the generated code actually includes an additional offset https://github.com/optimism-java/shisui/blob/portal/p2p/discover/portalwire/messages_encoding.go#L952,

I guess it is the cause that ultralight can't decode our message.

[ferranbt]

sure, we are working on portal network Go version, the wire protocol use a complex list type for Enrs, you can check the JS impl https://github.com/ethereumjs/ultralight/blob/master/packages/portalnetwork/src/wire/types.ts#L43.

Is there a specification of the format somewhere not in typescript code?

[GrapeBaBa]

sure, we are working on portal network Go version, the wire protocol use a complex list type for Enrs, you can check the JS impl https://github.com/ethereumjs/ultralight/blob/master/packages/portalnetwork/src/wire/types.ts#L43.

Is there a specification of the format somewhere not in typescript code?

This is the test vector https://github.com/ethereum/portal-network-specs/blob/master/portal-wire-test-vectors.md#content-response---multiple-enrs,

This is the spec https://github.com/ethereum/portal-network-specs/blob/master/portal-wire-protocol.md#content-union-definition, the content is union type and has three possible types to return.

[ferranbt]

Ok, the code generator does not support union types (there are none in the consensus specs) so you would have to build that type by hand using the ssz primitive library.

type A struct {
}

func (a *A) MarshalSSZTo(...) {} // this is code generated

type B struct {
}

func (b *B) MarshalSSZTo(...) {} // this is code generated

type Union struct {
  A *A
  B *B
}

func (u *Union) MarshalSSZTo(dst []byte) []byte {
   // check that both u.A and u.B are not set at the same time.
   if u.A != nil {
      // append 0 to dst
      dst = ...
      dst = u.A.MarshalSSZTo(dst)
   } else if u.B != nil {
      // append 1 to dst
      dst = ...
      dst = u.B.MarshalSSZTo(dst)
   }

   return dst
}

[GrapeBaBa]

Ok, the code generator does not support union types (there are none in the consensus specs) so you would have to build that type by hand using the ssz primitive library.

type A struct {
}

func (a *A) MarshalSSZTo(...) {} // this is code generated

type B struct {
}

func (b *B) MarshalSSZTo(...) {} // this is code generated

type Union struct {
  A *A
  B *B
}

func (u *Union) MarshalSSZTo(dst []byte) []byte {
   // check that both u.A and u.B are not set at the same time.
   if u.A != nil {
      // append 0 to dst
      dst = ...
      dst = u.A.MarshalSSZTo(dst)
   } else if u.B != nil {
      // append 1 to dst
      dst = ...
      dst = u.B.MarshalSSZTo(dst)
   }

   return dst
}

Ok, the code generator does not support union types (there are none in the consensus specs) so you would have to build that type by hand using the ssz primitive library.

type A struct {
}

func (a *A) MarshalSSZTo(...) {} // this is code generated

type B struct {
}

func (b *B) MarshalSSZTo(...) {} // this is code generated

type Union struct {
  A *A
  B *B
}

func (u *Union) MarshalSSZTo(dst []byte) []byte {
   // check that both u.A and u.B are not set at the same time.
   if u.A != nil {
      // append 0 to dst
      dst = ...
      dst = u.A.MarshalSSZTo(dst)
   } else if u.B != nil {
      // append 1 to dst
      dst = ...
      dst = u.B.MarshalSSZTo(dst)
   }

   return dst
}

What about merkle methods for the union type?

[ferranbt]

What about merkle methods for the union type?

From this, it might be something like:

func (u *Union) HashTreeRootWith(hh ssz.HashWalker) (err error) {
    indx := hh.Index()

    var indexElem uint64
    if u.A != nil {
        indexElem = 0
        if err = u.A.HashTreeRootWith(hh); err != nil {
             return err
        }
    } else if u.B != nil {
        indexElem = 1
        if err = u.B.HashTreeRootWith(hh); err != nil {
             return err
        }
    }

    hh.MerkleizeWithMixin(indx, indexElem, 2)
    return
}

[GrapeBaBa]

What about merkle methods for the union type?

From this, it might be something like:

func (u *Union) HashTreeRootWith(hh ssz.HashWalker) (err error) {
    indx := hh.Index()

    var indexElem uint64
    if u.A != nil {
        indexElem = 0
        if err = u.A.HashTreeRootWith(hh); err != nil {
             return err
        }
    } else if u.B != nil {
        indexElem = 1
        if err = u.B.HashTreeRootWith(hh); err != nil {
             return err
        }
    }

    hh.MerkleizeWithMixin(indx, indexElem, 2)
    return
}

Actually this is not the issue we met, can you look at the Enrs struct, the generated code always has offset 4, but it should be 0 since other client encoding it.

[ferranbt]

the generated code always has offset 4, but it should be 0 since other client encoding it

Offset in the serialisation or merkleize?

[GrapeBaBa]

the generated code always has offset 4, but it should be 0 since other client encoding it

Offset in the serialisation or merkleize?

serialisation https://github.com/optimism-java/shisui/blob/portal/p2p/discover/portalwire/messages_encoding.go#L952, for interop test, we need remove this offset part in generated code.

[ferranbt]

sszgen does not support unions so It cannot remove the offset from the generated code, otherwise, it will break the other encodings. You have to write the encodings by hand to do it.

[GrapeBaBa]

sszgen does not support unions so It cannot remove the offset from the generated code, otherwise, it will break the other encodings. You have to write the encodings by hand to do it.

Got it.