search

bwNetFlow / flowpipeline

Process network flows using fully configurable pipelines.
GNU General Public License v3.0
17 stars 5 forks source link

Add documentation for protobuf modification & compilation #71

Closed hreese closed 1 year ago

hreese commented 1 year ago

We're building a new segment that needs new fields in EnrichedFlow. We've updated the protobuf definition and compiled it like this: protoc --go_out=. --go_opt=paths=source_relative pb/flow.proto.

go build . builds a new binary without warnings, running the new binary fails like this:

./flowpipeline 
panic: proto: file "pb/flow.proto" is already registered
    previously from: "github.com/netsampler/goflow2/pb"
    currently from:  "github.com/bwNetFlow/flowpipeline/pb"
See https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

goroutine 1 [running]:
google.golang.org/protobuf/reflect/protoregistry.glob..func1({0x20dbc40?, 0xc0000407b0?}, {0x20dbc40?, 0xc0000407f0})
    /home/sprawl/go/pkg/mod/google.golang.org/protobuf@v1.28.1/reflect/protoregistry/registry.go:56 +0x1ee
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile(0xc000012060, {0x20fc9e8?, 0xc0000fe380?})
    /home/sprawl/go/pkg/mod/google.golang.org/protobuf@v1.28.1/reflect/protoregistry/registry.go:130 +0x39f
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0x1af04c8, 0x24}, {0x2b74360, 0x11bc, 0x11bc}, 0x5, 0x1, 0x0, 0x0, {0x20e10f8, ...}, ...})
    /home/sprawl/go/pkg/mod/google.golang.org/protobuf@v1.28.1/internal/filedesc/build.go:112 +0x1d6
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x1af04c8, 0x24}, {0x2b74360, 0x11bc, 0x11bc}, 0x5, 0x1, 0x0, 0x0, {0x0, ...}, ...}, ...})
    /home/sprawl/go/pkg/mod/google.golang.org/protobuf@v1.28.1/internal/filetype/build.go:138 +0x1b8
github.com/bwNetFlow/flowpipeline/pb.file_pb_flow_proto_init()
    /home/sprawl/git/KIT-CERT/flowpipeline/pb/flow.pb.go:1637 +0x1d8
github.com/bwNetFlow/flowpipeline/pb.init.0()
    /home/sprawl/git/KIT-CERT/flowpipeline/pb/flow.pb.go:1604 +0x17

There's a temporary workaround (build using go build -ldflags "-X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=warn" .) but we'd like to do this properly before creating PRs for our work.

I don't understand the underlying problem, but assuming that my protoc invocation wasn't correct: what's to proper way to recompile the protobuf definition?

hreese commented 1 year ago

After discussion with @debugloop (indirectly by proxy 😛) we'll rename flow.proto to enrichedflow.proto to prevent the namespace problem. I'll close the issue when making the matching PR.