search

google / proto-lens

API for protocol buffers using modern Haskell language and library patterns.
https://google.github.io/proto-lens
BSD 3-Clause "New" or "Revised" License
464 stars 110 forks source link

Support lazy construction for byte fields #435

Open blackgnezdo opened 2 years ago

blackgnezdo commented 2 years ago

Proto-lens currently generates an API taking strict ByteString for bytes fields. This makes for suboptimal performance because the user is required to make a copy which is then internally turned into a Builder again for serializing the message in Data.ProtoLens.Encoding.Bytes. To avoid breaking the APIs we should introduce alternative lenses for setting the bytes fields to lazy values (or even a Builder). A better API design is welcome.

Internally, we should store a Builder. We'll automatically apply Builder.byteString in generated code. This appears to be efficient so I don't expect any performance regression.

There's a possible continuation of this theme for Text but I don't have a pressing need and the story is not as clear.

@awpr @judah @jinwoo WDYT?

jinwoo commented 2 years ago

Sounds like a good change but what do I know about proto-lens internals :)

TravisWhitaker commented 10 months ago

This makes for suboptimal performance because the user is required to make a copy

I'm not sure if the parsers generated by proto-lens always make copies of ByteStrings, but regardless use of strict ByteString in the generated type definitions doesn't require the underlying bytes to be copied on construction. See for example the implementation of splitAt

Internally, we should store a Builder.

This would be a performance loss for many usage patterns.