search

TheThingsNetwork / lorawan-stack

The Things Stack, an Open Source LoRaWAN Network Server
https://www.thethingsindustries.com/stack/
Apache License 2.0
975 stars 306 forks source link

Migrate away from gogo/protobuf #2798

Closed rvolosatovs closed 1 year ago

rvolosatovs commented 4 years ago

Summary

https://github.com/gogo/protobuf is not mantained any more https://github.com/gogo/protobuf/issues/691 (currently)

Why do we need this?

It's our dependency, which is incompatible with new golang/protobuf version, which more and more packages depend on, hence we need to replace the golang/protobuf version, depending on outdated versions of our direct dependencies and potentially even breaking packages this way

What is already there? What do you see now?

gogo/protobuf dependency

What is missing? What do you want to see?

Figure this out

How do you propose to implement this?

Figure out if a new maintainer will appear or different plugin with feature parity? Use just vanilla protobuf?

How do you propose to test this?

tests

Can you do this yourself and submit a Pull Request?

yes

rvolosatovs commented 4 years ago

Validation plugin we're using dropped support for GoGo https://github.com/envoyproxy/protoc-gen-validate/pull/340

htdvisser commented 4 years ago

I think the best way forward is to follow the ecosystem and migrate away from gogo/protobuf. With more and more of our other dependencies moving away from gogo, I think it will become increasingly difficult to keep using it. Of course it's going to be a lot of work to migrate, so if we do it, we need to come up with a good plan.

@rvolosatovs probably knows more about the custom options that are set in our gogottn generator, but here's what I found for the explicit options in our proto files:

Maybe this is also a good time to start thinking about our v4 API, because I can imagine we might discover some more (API breaking) surprises.

rvolosatovs commented 4 years ago

I think the best way forward would be to first try out https://github.com/alta/protopatch. Depending on the result:

Looking forward, I don't think we should be directly using vanilla protobuf protos in components internally at runtime(given the provided feature set of protobuf today). It only makes sense to use protobuf for (de-)serialization, so for storage and on API layer. Internally, however, using plain vanilla generated Go protos makes no sense to me. So, for example, NS:

  1. get *ttnpb.EndDevice (vanilla generated Go type) from the registry, deserialized from stored binary data
  2. convert *ttnpb.EndDevice into T_device, (NOTE: perhaps that could be just a wrapper initially or forever)
  3. use T_device internally in NS
  4. convert T_device into *ttnpb.EndDevice (NOTE: this could be a trivial, very fast task if we're using a wrapper, since we only need to modify changed fields and that could even be performed on binary data directly)
  5. set *ttnpb.EndDevice, serialize into binary data
htdvisser commented 4 years ago

Refs also https://github.com/TheThingsNetwork/lorawan-stack/issues/342 (generated populators)

johanstokking commented 4 years ago

I'm not in favor of a (smaller) alternative to gogo. It feels like pushing the can. Let's keep things as vanilla as possible, especially when we need to decide again what the best way forward is.

I do agree that we can consider using intermediary types in some places, instead of relying everywhere on the generated protos. It's basically separating data transfer objects (DTOs: protos, also for storage) from data access objects (DAOs: how we use them). If that is primarily reading, we can also declare interfaces and see how far we get with that.

That said, I wouldn't go as far as changing the entire NS to using T_device, but rather specific structs and/or interfaces as needed.

htdvisser commented 3 years ago

Let's move this discussion to November

johanstokking commented 3 years ago

@rvolosatovs what is your objection against moving to vanilla with a custom JSON marshaler?

rvolosatovs commented 3 years ago

The huge migration burden and loads of boilerplate if we end up just using vanilla protos directly. I don't really object to that though, I just think we should try to find a simple non-intrusive alternative first and if that's not possible, then resort to reworking this whole thing.

johanstokking commented 3 years ago

I'm afraid that any plugin that we start to rely on will end up in an unmaintained state at some point. Generally speaking, I'm in favor of keeping things as close to vanilla as possible. If that means nil checking more often than we like, then so be it. It can also work in our favor that we know that things are not set, instead of an initialized struct.

htdvisser commented 3 years ago

I'm afraid that refactoring our entire codebase is going to be a pain no matter how we do it. Our (gogo-generated) proto structs are used everywhere right now (gRPC API, HTTP API, events, errors, internally, Redis DB, ...), so changing to something else (whatever that other thing is) is going to touch pretty much everything in our codebase, and the way it looks now, all at the same time.

The hard requirement is that we don't break compatibility of our v3 API. Even if we decide to use this situation as the moment to start working on a v4 API (at least internally), we will still have to keep supporting that v3 API for existing users.

In the long term, I think we would do ourselves a big favor by decoupling our (versioned, stable-within-major) external APIs from our internal (unversioned, stable-within-minor) API and our (versioned, stable) database documents. We could then write or generate functions to convert between our internal APIs and the others.

But I think there are some steps that we can already take now:

JSON

In order to keep our v3 JSON API compatible, I think our first TODO is to work on generating JSON marshalers and unmarshalers that understand how to marshal/unmarshal our custom types. I think doing this is smart anyway because there is no stability promise for Go's implementation of the JSON format for protocol buffers, so we better have control over that ourselves. Doing this could also allow us to consider field masks when marshaling to JSON. In the grpc-gateway runtime we can register codecs, so we can just write a codec that calls our own (generated) (un)marshalers instead of {gogo,golang}/protobuf's jsonpb.

Generate new protos next to the old ones

I already tried that here: https://github.com/TheThingsNetwork/lorawan-stack/commit/a41f62d98ae7ee719b576e6fcd2009a79cd38f4c

This does make protobuf complain about the types registry, so we may need to remove golang_proto.RegisterType from our old protos to make this work. Removing that could potentially break resolving of google.protobuf.Any, but we only use those in errors and events, so we can probably find workarounds for those specific cases.

Generate functions to convert between old and new protos

This is for the transition period only, but for the long term solution, we'd want to generate similar converters.

Update services one by one

I already tried that with a simple service here: https://github.com/TheThingsNetwork/lorawan-stack/commit/cd7d75c8b42ad15eee1ac594ff6d0f2d5a75eb67, but for more complicated services we'd definitely need those converters.

Note that this only changes the grpc service itself. The grpc-gateway still uses the old gogo stuff on the JSON side, and then calls the internal gRPC server, which then runs the new implementation.

johanstokking commented 3 years ago

Pushed some initial dependency updates and backwards compatibility work-arounds here: https://github.com/TheThingsNetwork/lorawan-stack/compare/issue/2798-codec

htdvisser commented 3 years ago

More and more of our dependencies are upgrading to protobuf 1.4 and the V2 API, and the longer we keep this open, the more problems we'll have when trying to upgrade our dependencies.

We should really give this some more priority and make a decision on what we're going to do about all this.

johanstokking commented 3 years ago

Please plan a call for next week so we can discuss offline.

I think we should go through this pain process and concentrate on solving this in a week or two. And to avoid that we do other things as this is going to cause lots of conflicts otherwise. Having as many hands as possible requires knowing exactly what we are going to do in which cases, dividing tasks as much as possible and keeping eyes on the prize.

htdvisser commented 3 years ago

Next steps:

  1. @rvolosatovs updates the tooling to be as close as possible to "vanilla":
    • Remove things like unconvert, gofumpt and whatever else we're doing on top of protoc
    • Switch from protoc-gen-gogottn to protoc-gen-gofast (or whatever is closests to vanilla)
    • Explicitly add (gogoproto.*) options in our proto files, so that they render the same as now
  2. We need to refactor our code to use Getters instead of direct field access. Perhaps tools like gopls and rf can help with this.
  3. We start removing (gogoproto.*) options one by one and update code that uses it. Perhaps tools like gopls and rf can help with this.
    • Removing gogoproto.populate and updating tests (https://github.com/TheThingsNetwork/lorawan-stack/issues/342)
    • Removing gogoproto.customname and changing EUI -> Eui etc.
    • Removing gogoproto.embed. We do need to make sure that messages still implement interfaces such as ValidateContext(context.Context) error and ExtractRequestFields(m map[string]interface{}).
    • Removing gogoproto.nullable and making sure we use Getters where possible, and do nil checks otherwise.
    • ...
htdvisser commented 3 years ago

@rvolosatovs let's try to make those first steps for v3.11.3. When that's done please re-add the other assignees, and let's discuss again.

htdvisser commented 3 years ago

I think it's time for the entire team to start working this, because I really can't (and don't want to) do this alone. It's very time consuming work, and often frustrating, but it needs to be done.

Before my vacation I've spent a lot of time on generating JSON (un)marshalers, which should let us remove the gogoproto.customtypes and hand-written JSON (un)marshalers.

Unfortunately, we still have quite some manual work to do for removing gogoproto.nullable and gogoproto.embed, and I'm afraid this will be a matter of removing hundreds of those options one by one, and making sure that the code that uses those fields doesn't break.

Things to look at:

htdvisser commented 3 years ago

I plan to remove some gogoproto.embed and gogoproto.nullable options from user.proto today.

KrishnaIyer commented 3 years ago

I'll take up gogoproto.embed from deviceclaimingserver.proto today.

KrishnaIyer commented 3 years ago

In the next pass, I'll take up gateway_services.proto and gateway.proto; the latter being pure drudgery.

htdvisser commented 3 years ago

I plan to remove some gogoproto.embed and gogoproto.nullable options from organization.proto today.

htdvisser commented 3 years ago

Will do the same for client.proto.

htdvisser commented 3 years ago

And today for application.proto.

htdvisser commented 3 years ago

@KrishnaIyer I'll do the GatewayIdentifiers in gateway.proto, because I've been doing the same for other identifiers in the last couple of days.

htdvisser commented 3 years ago

So for @adriansmares @bafonins @johanstokking @kschiffer @pgalic96:

KrishnaIyer commented 3 years ago

In the next pass, I'll pick up joinserver.proto and metadata.proto.

johanstokking commented 3 years ago

@KrishnaIyer I'd like to take over joinserver.proto as I'm working on a refactor (https://github.com/TheThingsNetwork/lorawan-stack/issues/4677).

KrishnaIyer commented 3 years ago

Ok sure 👍

adriansmares commented 3 years ago

Picking up the applicationserver_*.proto files.

htdvisser commented 3 years ago

I'm working on the remaining gogoproto.embed options in application.proto, client.proto, gateway.proto, organization.proto and user.proto today.

htdvisser commented 2 years ago

The following files still have gogoproto.embed options left:

And these files (also) have gogoproto.nullable options that need to be removed:

Everyone please claim some more files!

As a reminder:

htdvisser commented 2 years ago

I'm picking up the gogoproto.embed options in identityserver.proto, oauth.proto, rights.proto and search_services.proto today.

adriansmares commented 2 years ago

Taking api/message_services.proto

Edit: This API in unused, will pickup events.proto and gateway.proto.

htdvisser commented 2 years ago

Today I'm going to remove gogoproto.nullable from timestamps in application.proto, client.proto, oauth.proto, organization.proto, rights.proto and user.proto.

adriansmares commented 2 years ago

Picking up embed/nullable in lorawan.proto for gateway related entities, will add the rest later.

KrishnaIyer commented 2 years ago

I'm picking up api/messages.proto today. This one is a PITA.

htdvisser commented 2 years ago

Today I'll be working on getting rid of (gogoproto.stdtime) and (gogoproto.stdduration) options.

KrishnaIyer commented 2 years ago

Today I'll pick up gatewayserver.proto and join.proto.

johanstokking commented 2 years ago

Today I'll pick up deviceclaimingserver.proto, lorawan.proto and qrcodegenerator.proto

johanstokking commented 2 years ago

Turns out that deviceclaimingserver.proto and joinserver.proto are already not using nullable (when not using customtype), embed, stdtime nor stdduration. I'll pick up regional.proto instead.

johanstokking commented 2 years ago

I'm also doing messages.proto as it's quite related

michalborkowski96 commented 2 years ago

I'm picking up metadata.go and end_device.proto

KrishnaIyer commented 2 years ago

I'll pick up api/identifiers.proto tomorrow since that's related to https://github.com/TheThingsNetwork/lorawan-stack/issues/4896. This is one of our most used messages so this is going to be fun.

htdvisser commented 2 years ago

Here's what's left for the field options (still omitting fields with customtype):

EntityIdentifiers entity_ids = 2 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
google.protobuf.Timestamp created_at = 3 [(gogoproto.stdtime) = true];
google.protobuf.Timestamp updated_at = 4 [(gogoproto.stdtime) = true];
MessagePayloadFormatters default_formatters = 13 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
google.protobuf.Duration class_b_timeout = 1 [(gogoproto.stdduration) = true];
google.protobuf.Duration class_c_timeout = 5 [(gogoproto.stdduration) = true];
google.protobuf.Duration status_time_periodicity = 16 [(gogoproto.stdduration) = true];
google.protobuf.Duration class_b_c_downlink_interval = 31 [(gogoproto.stdduration) = true];
MACParameters current_parameters = 1 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
MACParameters desired_parameters = 2 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
DLSettings downlink_settings = 6 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
JoinRequest request = 2 [(gogoproto.nullable) = false];
SessionKeys keys = 3 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
EndDevice end_device = 1 [(gogoproto.embed) = true, (gogoproto.nullable) = false, (validate.rules).message.required = true];
EndDevice end_device = 1 [(gogoproto.embed) = true, (gogoproto.nullable) = false, (validate.rules).message.required = true];
EndDevice end_device = 1 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
EndDevice end_device = 1 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
google.protobuf.Duration token_ttl = 2 [(gogoproto.stdduration) = true];
google.protobuf.Duration deduplication_window = 2 [(gogoproto.stdduration) = true];
google.protobuf.Duration cooldown_window = 3 [(gogoproto.stdduration) = true];
ttn.lorawan.v3.EntityIdentifiers entity_ids = 1 [(gogoproto.nullable) = false];
google.protobuf.Timestamp created_at = 2 [(gogoproto.stdtime) = true];
google.protobuf.Timestamp updated_at = 3 [(gogoproto.stdtime) = true];
LicenseIdentifiers id = 1 [(gogoproto.embed) = true, (gogoproto.nullable) = false, (validate.rules).message.required = true];
LicenseIssuerIdentifiers license_issuer_ids = 2 [(gogoproto.embed) = true, (gogoproto.nullable) = false, (validate.rules).message.required = true];
google.protobuf.Timestamp created_at = 3 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
google.protobuf.Timestamp valid_from = 4 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
google.protobuf.Timestamp valid_until = 5 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
google.protobuf.Duration warn_for = 6 [(gogoproto.nullable) = false, (gogoproto.stdduration) = true];
google.protobuf.Duration limit_for = 7 [(gogoproto.nullable) = false, (gogoproto.stdduration) = true];
google.protobuf.Duration extend_valid_until = 1 [(gogoproto.stdduration) = true];
google.protobuf.Duration interval = 1 [(gogoproto.stdduration) = true, (gogoproto.nullable) = false];
TenantIdentifiers tenant_id = 1 [(gogoproto.embed) = true, (gogoproto.nullable) = false];
htdvisser commented 2 years ago

And here's the files that still have enum options:

  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.enum_stringer) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.enum_stringer) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.enum_stringer) = false;
  option (gogoproto.enum_stringer) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.enum_stringer) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
  option (gogoproto.goproto_enum_prefix) = false;
johanstokking commented 2 years ago

What's the trick for getting rid of the enum options? What is the implication for JSON API compatibility?

htdvisser commented 2 years ago

I don't know yet, but I plan to look into that this week (unless someone else wants to). Other assignees can for the time being focus on the remaining nullable, embed, stdtime, stdduration options in the comment above.

All JSON API is now handled by our generated (un)marshalers and the (thethings.json.*) options in our .proto files.

@pgalic96 has taken over the work on the CLI flag handling, so that part should also be handled soon(ish).

htdvisser commented 2 years ago

I've started removing the first gogoproto.customtype options here: https://github.com/TheThingsIndustries/lorawan-stack/compare/issue/gogoproto-customtype. But until @pgalic96 is done with the CLI flag handling, we can only remove gogoproto.customtype options on response messages (or for anything that isn't used in the CLI).

htdvisser commented 2 years ago

Here's the latest list of what's left to do in this round. I left out (gogoproto.goproto_registration) (because we need that until we switch libraries), the various enum_prefix options (I'm currently working on that) and (gogoproto.customtype) (because that comes in the next round when @pgalic96 is done with the CLI flags stuff).

./api/_api.proto:28:option (gogoproto.sizer_all) = true;
./api/end_device.proto:276:  MACParameters current_parameters = 1 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
./api/end_device.proto:278:  MACParameters desired_parameters = 2 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
./api/end_device.proto:581:  EndDevice end_device = 1 [(gogoproto.embed) = true, (gogoproto.nullable) = false, (validate.rules).message.required = true];
./api/end_device.proto:585:  EndDevice end_device = 1 [(gogoproto.embed) = true, (gogoproto.nullable) = false, (validate.rules).message.required = true];
./api/end_device.proto:620:  EndDevice end_device = 1 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
./api/end_device.proto:634:  EndDevice end_device = 1 [(gogoproto.nullable) = false, (validate.rules).message.required = true];
./api/tti/license.proto:24:  LicenseIdentifiers id = 1 [(gogoproto.embed) = true, (gogoproto.nullable) = false, (validate.rules).message.required = true];
./api/tti/license.proto:27:  LicenseIssuerIdentifiers license_issuer_ids = 2 [(gogoproto.embed) = true, (gogoproto.nullable) = false, (validate.rules).message.required = true];
./api/tti/license.proto:29:  google.protobuf.Timestamp created_at = 3 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
./api/tti/license.proto:31:  google.protobuf.Timestamp valid_from = 4 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
./api/tti/license.proto:33:  google.protobuf.Timestamp valid_until = 5 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
./api/tti/license.proto:36:  google.protobuf.Duration warn_for = 6 [(gogoproto.nullable) = false, (gogoproto.stdduration) = true];
./api/tti/license.proto:38:  google.protobuf.Duration limit_for = 7 [(gogoproto.nullable) = false, (gogoproto.stdduration) = true];
./api/tti/license.proto:78:  google.protobuf.Duration extend_valid_until = 1 [(gogoproto.stdduration) = true];
./api/tti/license.proto:83:  google.protobuf.Duration interval = 1 [(gogoproto.stdduration) = true, (gogoproto.nullable) = false];
./api/tti/license.proto:119:    TenantIdentifiers tenant_id = 1 [(gogoproto.embed) = true, (gogoproto.nullable) = false];
htdvisser commented 2 years ago

Just went through the codebase, and it looks like we only have a few more options left:

Today I'll continue working on removing some gogoproto.customtype options (https://github.com/TheThingsNetwork/lorawan-stack/issues/2798#issuecomment-1004854414).

htdvisser commented 2 years ago

Here's the list of protos with customtype fields that we'll have to remove.

I think the safest migration path is to be "better safe than sorry" and replace the customtype option with:

  1. Validation option: 
    (validate.rules).bytes = { len: 8, ignore_empty: true }
  2. JSON option (even if we currently don't marshal to/from JSON): 
    (thethings.json.field) = {
 marshaler_func: "go.thethings.network/lorawan-stack/v3/pkg/types.MarshalHEXBytes",
 unmarshaler_func: "go.thethings.network/lorawan-stack/v3/pkg/types.Unmarshal8Bytes"
}
  3. Flag option (even if we currently don't have flags for the field): 
    (thethings.flags.field) = {
 set_flag_new_func: "go.thethings.network/lorawan-stack/v3/pkg/types.New8BytesFlag",
 set_flag_getter_func: "go.thethings.network/lorawan-stack/v3/pkg/types.GetBytesFromFlag",
}

I already checked off the ones I started with in https://github.com/TheThingsIndustries/lorawan-stack/compare/issue/gogoproto-customtype

api/end_device.proto:

api/gateway.proto:

api/tti/tenant.proto:

api/tti/configuration.proto:

api/tti/license.proto:

api/join.proto:

api/joinserver.proto:

api/messages.proto:

api/keys.proto:

api/networkserver.proto:

api/metadata.proto:

api/lorawan.proto:

api/application.proto:

api/packetbrokeragent.proto:

api/identifiers.proto:

api/deviceclaimingserver.proto: