proto: panic: duplicate enum registered: google.protobuf.FieldDescriptorProto_Type

Globegitter commented 8 years ago

I am trying to use custom options in our codebase, as mentioned here: https://developers.google.com/protocol-buffers/docs/proto#customoptions

Our codebase actively supports python, go and java. So protos are built against these three environments. It seems while custom options work fine for java and python there seems to be an issue with go leading to following error message when running code coverage on one of our tests:

2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FieldMask
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.Timestamp
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.Any
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FileDescriptorSet
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FileDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.DescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.DescriptorProto.ExtensionRange
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FieldDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.EnumDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.EnumValueDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.ServiceDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.MethodDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FileOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.MessageOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FieldOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.EnumOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.EnumValueOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.ServiceOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.MethodOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.UninterpretedOption
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.UninterpretedOption.NamePart
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.SourceCodeInfo
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.SourceCodeInfo.Location
panic: proto: duplicate enum registered: google.protobuf.FieldDescriptorProto_Type

goroutine 1 [running]:
panic(0x8fa960, 0xc8201552d0)
    /usr/lib/go/src/runtime/panic.go:464 +0x3e6
github.com/golang/protobuf/proto.RegisterEnum(0xbbca60, 0x29, 0xc820199980, 0xc8201999b0)
    /path/to/third_party/go/ptypes/src/github.com/golang/protobuf/proto/properties.go:811 +0xe2
third_party/proto/google/protobuf.init.4()
    third_party/proto/google/protobuf/descriptor.pb.go:1913 +0x706
third_party/proto/google/protobuf.init()
    third_party/proto/google/protobuf/descriptor.pb.go:2798 +0x5cd
main.init()
    /path/to/lobby_test_main.go:149 +0x7e

The issue being here we are importing the module third_party/proto/google/protobuf multiple times like so:

import ( 
...
    _cover9 "third_party/proto/google/protobuf"

    _cover10 "third_party/proto/google/protobuf"

    _cover11 "third_party/proto/google/protobuf"

    _cover12 "third_party/proto/google/protobuf"
)

these are being used to set up the coverage before running the actual tests. If I remove these imports and the coverage instrumentation for them all works fine.

So it seems to me that the init function of the descriptor.pb.go is being called multiple times, which in turn registers the enum multiple times and then fails. I don't quite understand how importing the module multiple times can have any effect. If I import our own protos multiple times (that also register enums) everything seems fine.

Is that an issue with the generated code? And is there any way to fix this? (possibly via https://github.com/golang/protobuf/pull/131 ?)

dsymonds commented 8 years ago

Importing the same package multiple times shouldn't cause multiple runs of its init functions. Is it possible you have the same package being imported under more than one import path? That would cause the observed behaviour.

Globegitter commented 8 years ago

@dsymonds Thanks for the response; that should not be happening but I will look into that.

Also just tested https://github.com/golang/protobuf/pull/131 and it would indeed fix it given that we would add the right go_package statement upstream. Is there any possibility of this happening?

Globegitter commented 8 years ago

There is nowhere that I can find it being imported under more than one import path right now, but given that a solution more or less already exists in two other existing PRs it would be much preferable to wait for that.

Then we don't have to do our custom imports and this would also be on par with how the python library do it and simplify things quite a bit for cross-langauge support.

erikreppel commented 8 years ago

Did this ever get any kind of resolution? I'm hitting this issue when trying to use the gcp datastore library and protobufs in the same project.

douglas-reid commented 8 years ago

I, too, am seeing this issue when trying to use any of the google cloud client libraries and protobufs in the same project.

zombiezen commented 8 years ago

/cc @jba

I would echo dsymond's question: are you sure you haven't put the same generated proto under multiple import paths?

jba commented 8 years ago

@okdave is working on the go-package option in some files.

We did recently (a month ago?) put a lot of protos in a common place. For example, FieldMask is now in https://github.com/google/go-genproto/blob/master/protobuf/field_mask.pb.go, which has the import path "google.golang.org/genproto/protobuf". Does this also live under another import path?

okdave commented 8 years ago

If you're seeing this, it would be good to understand your project's layout. It's possible that you'll see this issue with vendoring – for example, having the same proto .pb.go file vendored into two different places in your source tree.

yohcop commented 8 years ago

I have the same problem, and I'll try to describe what is happening to me. I've tried many different things/layouts, and had 3 problems - and 0 solutions so far. The enum being registered multiple times was the starting point.

Problem 1

I use this package, and my code uses descriptor.pb.go from this repo (because I implement a protoc plugin)
since this repo doesn't include descriptor.proto, I needed to get it from the original repo to use it in my own .proto files
my.proto file uses the descriptor.proto of course, to add field and message extensions
I also use Bazel... so my go_proto_library rule for my.proto, depends on the go_proto_library rule I hand-created for descriptor.proto.
that last rule for descriptor.proto, creates a package, google/protobuf (later I'll explain what happens if that package is set to github.com/golang/....)
I now have 2 libraries, google/protobuf and github.com/golang/.../descriptor defining those enums. They are both linked because I used this package, as well as my proto which is linked against descriptor.proto.

Problem 2

My code was importing github.com/golang/.../descriptor, but the generated proto from the go_proto_library was importing google/protobuf. So I decided to change my code's imports from github.com/golang/.../descriptor to google/protobuf instead.

The problem next was that the package github.com/golang/../plugin has a plugin.CodeGeneratorRequest. That proto contains a ProtoFile field, which are of type github.com/golang/.../descriptor.FileDescriptorProto. And this type is then incompatible with google/protobuf.FileDescriptorProto.

Problem 3

I then decided to move the google/protobuf package to github.com/golang/.../descriptor. But because I use bazel and that the github.com repository is in an external/ directory I'm not supposed to modify it. So I tried to recreate the directory structure github.com/golang/..../descriptor, and move descriptor.proto and the BUILD file there.

But now, I have two different packages both producing a github.com/golang/..../descriptor.a object file. And Bazel complains about it, of course.

Solution?

I think adding descriptor.proto to this repo would actually solve this. I would then have 2 options:

create a go_proto_library that uses both the go_library for descriptor.pb.go and a filegroup for descriptor.proto.
forget about the pre-built descriptor.pb.go and only add a go_proto_library for descriptor.proto.

Any other ideas?

This may be a little contrived, and I hope my explanations are clear enough. It's relatively unlikely @Globegitter's problem is the same exact thing, but who knows...

Thanks!

yohcop commented 8 years ago

So to summarize, yes this was due in my case to having descriptor.pb.go in two packages, linked together, but the problem was created by this repo not having both descriptor.pb.go and descriptor.proto.

https://github.com/google/go-genproto may be a solution, but then again, for this to work this repo should not have a descriptor.pb.go either, or the output of two libraries will clash (problem 3).

I also found #131 like @Globegitter mentioned. I hope this could be resolved soon :)

junghoahnsc commented 8 years ago

I have that same issue. Two other third-party packages are using Descriptor in different package:

org_golang_google_genproto/protobuf/descriptor.pb.go
com_github_golang_protobuf/protoc-gen-go/descriptor/descriptor.pb.go

I think Descriptor should exist in one package. What is the best workaround for now?

songtianyi commented 7 years ago

Any solution here?

jba commented 7 years ago

We are working on de-duplicating the descriptor proto.

mickeyreiss commented 7 years ago

@jba @bcmills Any update on this issue? I'm still seeing this occur after https://github.com/golang/protobuf/commit/cf10ca0ea91371dd30946defd93a0ed43aa1efe7.

jba commented 7 years ago

@pongad is actively working on it.

pongad commented 7 years ago

This problem should be resolved from genproto's side now. By agreement in here and here, descriptor is removed from genproto. The official home for it is now github.com/golang/protobuf/protoc-gen-go/descriptor.

dsnet commented 6 years ago

Closing this as fixed per @pongad's comment.

There are multiple ways that a registration conflict can occur:

Using vendor can cause the same generated proto message to be in multiple packages. These are distinct packages from Go's perspective so the init functions are run for each, even if the code is semantically identical. When init runs, it panics about duplicate registrations. The movement of go tool to versioning should alleviate this issue. See golang/go#24301.
Cloning the .proto file and generating Go messages from each can also result in this issue. It's not clear what should be done in this case since I don't fully understand all the cases why someone clones a proto file. Whether we should do something in the protobuf repo itself to alleviate this problem is a discussion for #268.

If you're still experiencing problems, feel free to open a new issue.

WindGreen commented 5 years ago

my case: dirA/a.proto package a dirB/a.proto package a (should be b) cased this problem

golang / protobuf