How to specify sizeof for slice defined in another struct?

[mewmew]

Looking at the package documentation, the following example is presented:

struct {
    Length int `struct:"int32,sizeof=Packets"`
    Packets []struct{
        Source    string    `struct:"[16]byte"`
        Timestamp int       `struct:"int32,big"`
        Data      [256]byte `struct:"skip=8"`
    }
}

Would it be possible to use the sizeof parameter, even if Length was defined in a different struct than Packets?

For instance:

type File struct {
        Hdr Header
    Packets []struct{
        Source    string    `struct:"[16]byte"`
        Timestamp int       `struct:"int32,big"`
        Data      [256]byte `struct:"skip=8"`
    }
}

type Header struct {
    Length int `struct:"int32,sizeof=Packets"`
}

Note, the above example is a contrived once, and for the file format I've been trying to parse, and I'd like to avoid simply flattening the Header struct into the File struct if at all possible.

So, can we refer to say, File.Packets from the Length field in the Header struct somehow?

Happy for any suggestions on how to do this, and thanks a lot for sharing this wonderful library with the community!

Cheers, /u

[umeat]

I have a similar issue where I want to make a custom sizeof for a type (so it implements some interface which gives len effectively). For example when you have a mask field where the sum of set bits is the length of a data field:

  type Message struct {
      Head uint8 `struct:"uint8:2"`
      Mask Mask `struct:"sizeof=C"`
      Data []bool `struct:"uint8:1,variantbool"`
  }  

  type Mask struct {
      M uint8 `struct:"uint8:6"`
  }

  func (m Mask) SizeOf() int {
      return bits.OnesCount8(m.M)
  }              

Or, more simply, if you could specify a sizer tag or something which specifies the function which should be used to get the size. So any function which takes the specified type and returns int.

  type Message struct {
      Head uint8 `struct:"uint8:2"`
      Mask uint8 `struct:"uint8:6,sizeof=C,sizer=bits.OnesCount8"`
      Data []bool `struct:"uint8:1,variantbool"`
  }                                             

Could the expression language PR do this?

[jchv]

Unfortunately, the expression language couldn't really do that, at least not quite that way. It mostly enables you to do calculations inline. To be honest with you, for the amount of complexity and increased bug surface area it brings, I'm not really satisfied with what it has to offer, hence the reason it is pretty much stalled.

Trying to balance between serving Restruct's original purpose of being simple and easy, but reasonably powerful, versus the desire to support as many binary structures in an intuitive manner as possible, has been difficult. Just bolting an expression language on does actually open up a fairly large amount of new capabilities, but it's unclear it's worth it for the cost.

I'm still trying to figure out what the next generation of restruct should really look like. I think for the most part requiring code generation is a no-go, but simultaneously some more advanced functionalities may require codegen to work at all, which makes it hard.

Hopefully there is some happy medium we can find in the future that can handle the litany of different desired use cases effectively, but I can't help but feel some of the bigger efforts will give diminishing returns that lead to a local maximum. But, getting past this local maximum would require a fairly dramatic shift, and may not serve the original intended niche that restruct exists for. A full binary schema scripting language could potentially solve a much wider domain of problems, potentially even wider than Kaitai Struct, but at that point it's pretty far away from what the project was really meant to do. (Even bitfield support is beyond what I had originally imagined, though evidently it is pretty important.)

---

More on topic, I think you are best off simply doing this bit manually for now. I realize this may be a contrived example, but you probably want to combine Mask and Data into a single struct, and use the BitSizer interface since the original Sizer predates bit-level support and counts bytes instead.

[umeat]

Do you have any examples of using BitSizer? For example, how would you implement the above?

For what it's worth - my non-contrived example is over at https://github.com/GeoscienceAustralia/go-rtcm - where I am now using restruct (thanks very much, it's very good) for everything except for rtcm3/msm.go. Mainly because I haven't been able to figure out how to get the Header mask attribute sizes correct (the bits.OnesCount example above). Even if that is resolved with the BitSizer interface, I don't think all of the signal and satellite data types could inherit the correct lengths from the Header.

[jchv]

I’ll take a look later on today, or at the latest this weekend. I don’t think there is an example of using BitSizer; in general, restruct could do with better documentation (specifically, examples.) The actual interface is here:

https://godoc.org/github.com/go-restruct/restruct#BitSizer

[jchv]

Alright so, I haven't responded, but I've actually been working on this. I think we're going to need an expression language to make this case really work well.

The issue is, interleaving restruct and non-restruct serialization is just not really possible, at least not in any manner that would be worth it; you're practically better off just doing it all manually, as you have been doing. At the same time, this functionality is obviously a bit too specialized to ship as a feature of restruct.

So, I'm going to attempt to ship an expression language and some new features aimed at handling these cases. To that end, I investigated the state of expression languages in Go. There's a few promising ones: CEL was probably the best looking one, but it has fairly heavy dependencies. The existing expression language PR is going in an OK direction, but has issues of its own (especially in the AST,) and I don't like the generated parser very much.

I'm going to try to implement an expression language simpler than the existing PR (reusing a bit of the code, but not much) and ship it soon. It will likely have some compatibility breaks over time, but I'm hoping that the end result is mostly a superset of what is first shipped, and that simple use cases do not break.

[jchv]

A weekend late, but there is now expressions support in master. I basically had to rewrite it from scratch because I was just not happy with the 30k line monstrosity of mostly generated code that occurred in the first PR.

It is unstable and not completely finished, so it is currently behind a flag. Call restruct.EnableExprBeta() to enable it.

Once that's done, you should now be able to do something like this:

  type Message struct {
      Head uint8 `struct:"uint8:2"`
      Mask uint8 `struct:"uint8:6"`
      Data []bool `struct:"size=bits.OnesCount8(Mask)"`
  }

The expr library currently produces pretty bad error messages, and it does not support strings or complex types and has some other omissions as well. However, despite that, I think it is going in the right-ish direction and so it's probably safe to start writing code that relies on it - I just want to wait for better test coverage and implementing the remaining functionality before removing the flag.

The functions exposed to expressions has to be done manually. Most of the bits package is exposed by default (if you are on Go 1.9 or above - math/bits didn't exist prior) and suggestions for other packages to expose by default are welcome. There is currently no proper mechanism for exposing additional functionality.

Along with size, bits is available to programmatically control the number of bits of a field (as an alternative to specifying it on the type.) in allows you to set the value of the struct field. out allows you to control the value that is outputted from the struct field when written. Finally, if allows you to conditionally omit and include struct fields based on an expression. (There are still plans to add while.)

[jchv]

It is now possible to refer to the parent or root structs using expressions. Take a look at the new PNG.File struct as an example - specifically, ChunkIDAT, which refers back to its parent.

I cannot guarantee the semantics won't change. I am thinking about adding _parent and _root as special variables you can refer to in expressions without needing the parent flag, and may deprecate the parent flag and remove it before v2.

[umeat]

Is it possible to use an expression to specify the number of bits in a uint64 (for example)?

The following doesn't work because of "validSizeType" restrictions:

type Foo struct {
    A uint8 `struct:"uint8"`
    B uint8 `struct:"uint8"`
    C uint64 `struct-size:"bits.OnesCount8(A)*bits.OnesCount8(B)"`
}

[jchv]

For that you need to use bits instead, which also takes an expression.