ttnpb imports invalid version of grpc-gateway

jpmeijers commented 1 year ago

Summary

When one imports the golang types from ttnpb into a project, there is an error with the required grpc-gateway version.

One can work around this issue by overriding the grpc-gateway version in your go.sum file:

// But the original grpc-gateway v2.
replace github.com/grpc-ecosystem/grpc-gateway/v2 => github.com/grpc-ecosystem/grpc-gateway/v2 v2.10.3

Steps to Reproduce

Add imports to go program:

"go.thethings.network/lorawan-stack/v3/pkg/jsonpb"
"go.thethings.network/lorawan-stack/v3/pkg/ttnpb"

Try to unmarshal an uplink message:

func (handlerContext *Context) PostV3Uplink(w http.ResponseWriter, r *http.Request) {

body, err := ioutil.ReadAll(r.Body)

var packetIn ttnpb.ApplicationUp

contentType := r.Header.Get("Content-Type")
if contentType == "application/json" {
    marshaler := jsonpb.TTN()
    if err := marshaler.Unmarshal(body, &packetIn); err != nil {
        // Can not parse json body
    }
} else if contentType == "application/protobuf" || contentType == "application/x-protobuf" || contentType == "application/octet-stream" { // TTS uses application/octet-stream
    if err := proto.Unmarshal(body, &packetIn); err != nil {
        // Can not parse protobuf body
    }
}

Current Result

    go.thethings.network/lorawan-stack/v3/pkg/ttnpb imports
    github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2/options: github.com/grpc-ecosystem/grpc-gateway/v2@v2.0.0-00010101000000-000000000000: invalid version: unknown revision 000000000000

Expected Result

One should be able to import go.thethings.network/lorawan-stack/v3/pkg/ttnpb without any special overrides.

Relevant Logs

No response

URL

No response

Deployment

The Things Stack Community Edition

The Things Stack Version

3.23.1

Client Name and Version

No response

Other Information

No response

Proposed Fix

Use grpc-gateway release from upstream.

Contributing

[ ] I can help by doing more research.
[ ] I can help by implementing a fix after the proposal above is approved.
[X] I can help by testing the fix before it's released.

Code of Conduct

[X] I agree to follow TTN's Community Code of Conduct.

jpmeijers commented 1 year ago

Using the workaround mentioned, I'm getting another error:

go: gopkg.in/DATA-DOG/go-sqlmock.v1@v1.0.0-00010101000000-000000000000: invalid version: unknown revision 000000000000

Which can be worked around be adding the following to go.sum:

replace gopkg.in/DATA-DOG/go-sqlmock.v1 => gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0

adriansmares commented 1 year ago

References https://github.com/TheThingsNetwork/lorawan-stack/issues/2798.

Indeed, we're currently using a lot of replacements due to our gogoproto dependency. We'll hopefully be able to get rid of the ones mentioned here and a lot more once we finish removing gogoproto.

adriansmares commented 1 year ago

The v3.25 branch now contains the gogoproto-less version of the stack. It has a lot less replacements and lots of dependencies have been updated. Only the throttled replacement will be needed for now.

While the upgrade simplifies the dependencies significantly, it is not fully in place. The move away from gogoproto to vanilla goproto caused the standard type wrappers (timestamps, durations, structs) to be changed to their vanilla counter part. In practice this means that after the upgrade, places that used types.Struct or types.Timestamp will need to become structpb.Struct and timestamppb.Timestamp - it's mainly a find and replace job under VS Code.

Examples of how this mind numbing find and replace went can be found in the commits at https://github.com/TheThingsNetwork/lorawan-stack/pull/6032/ - there is a commit for each type.

jpmeijers commented 1 year ago

I updated my project using: go get go.thethings.network/lorawan-stack/v3/pkg/ttnpb@02d6710 go get go.thethings.network/lorawan-stack/v3/pkg/jsonpb@02d6710

Getting this warning: go: module github.com/golang/protobuf is deprecated: Use the "google.golang.org/protobuf" module instead.

The only override in my go.mod is: replace github.com/throttled/throttled/v2 => github.com/TheThingsIndustries/throttled/v2 v2.7.1-noredis

I only tested unmarshalling JSON, but that works as expected. I'll update another stack that uses protobufs for the webhook and confirm that works too.

Update: I can confirm my Go backend is unmarshalling the Protocol Buffer webhook uplinks correctly.

adriansmares commented 1 year ago

Closing this issue as this individual error is no longer relevant on the current head branch. The clean protobuf publication discussion is to follow in the linked issue.

TheThingsNetwork / lorawan-stack