Open martin-traverse opened 4 months ago
Hey Martin,
I think the issue here stems from our stance on field presence. Our current view is that implicit field presence was a mistake, and the default behavior should be explicit presence (e.g. proto2 "optional"). This particular issue you're pointing to would only show up if you were mixing explicit/implicit fields in the same file, in which case whichever semantics are in the minority would require annotations (you can set the file-level default to IMPLICIT or leave it unspecified to get EXPLICIT defaults).
I understand that if you had a proto3 file where you used the "optional" keyword to convey some semantic meaning, a behavior-preserving migration to edition 2023 would introduce some cruft. But is there a particular reason that switching to explicit presence everywhere would be a problem? What language runtime(s) are you using? Runtimes generally shouldn't treat non-present fields as "null", but just containing their default value. Our plan is to gradually turn down implicit presence and deprecate it in a future edition.
On a related note, the "optional" keyword in proto3 had it's own problems. For one, it was implemented by inserting a synthetic oneof around the field which added a huge amount of complexity for reflection. It also had some unexpected semantics around message fields and extensions. Message fields and extensions always have explicit presence, and the "optional" keyword didn't actually affect the behavior at all.
What language does this apply to? protobuf editions
Describe the problem you are trying to solve.
Hi, I've been reading the docs for the new protobuf editions. I'd like to raise an issue for optional support. The proposal is to replace optional / required from proto3 / proto2 with the new field_presence feature. While the other feature flags are mostly to do with technical concerns around encoding, validation etc., the optional flag carries business-focused semantics on a field-by-field basis. We use protobuf to describe both APIs and business metadata, describing which fields are optional is a key part of our service and data-model design, i.e. it is part of the semantic type specification, not a technical implementation detail.
I think it would be better to describe optional fields in a concise, readable way as part of the type description. Requiring annotations for every optional field seems like a step backwards.
Describe the solution you'd like
Allow the keyword "optional" in the editions syntax, with the same meaning it has in proto3. Technically this would be syntactic sugar that gets translated to [features.field_presence = EXPLICIT], and would only be useful if field_presence is set to implicit at the file (or type) level.
Describe alternatives you've considered
We could just keep using proto3 indefinitely, assuming it will still be supported, but as a general rule we prefer to move to latest stable with our major releases. Or we update our service and metadata files with the new field_presence syntax, which just makes them less clear to read.
Our product is a middleware data & analytics platform, so for us as platform developers using the field_presence syntax is just a less good developer experience. More importantly though, we publish our proto files as part of the documentation for our services and metadata model, so I'm more concerned about readability for our customers.
Additional context
There was a similar issue when proto3 itself was released and optional semantics were dropped altogether, only to be added back some time later.
One other data point, in the early versions of our product we tried to get away from explicit null / not-null declarations in our own metadata model and say that everything except business keys is always nullable. Clients wouldn't accept it, we ended up adding explicit notNull flags back in as well. We concluded that nullability is a core part of the type specification.