search

protocolbuffers / protobuf

Protocol Buffers - Google's data interchange format
http://protobuf.dev
Other
65.44k stars 15.46k forks source link

java_proto_library pulls in duplicate classes #8925

Closed eikemeier closed 3 years ago

eikemeier commented 3 years ago

What version of protobuf and what language are you using? Version: v3.18.0-rc1 Language: Java

What runtime / compiler are you using (e.g., python version or gcc version) Bazel 4.2.0

What did you do? Steps to reproduce the behavior:

  1. Check out https://github.com/eikemeier/proto-java
  2. bazel run //:main

What did you expect to see No duplicate resources.

What did you see instead? Multiple copies of the same class in in @com_google_protobuf//java/core:core-project, @com_google_protobuf//java/core:lite_runtime_only and @com_google_protobuf//:timestamp_proto.

Anything else we should know about your project / environment This is a regression: In v3.17.3 the duplicates of the full and the lite runtime where identical (@com_google_protobuf//:timestamp_proto only).

perezd commented 3 years ago

TLDR: I've done some diffing and comparison and I think this diff is causing this behavior for you, but I'm pretty sure this is intended behavior (I'll explain further): https://github.com/protocolbuffers/protobuf/commit/89a9f459e9b6996c36051f59c99b6c40650ad25b

Maven users of protobuf-java have always had an extra folder "google/protobuf/..." in the runtime jar which contains the .proto source files for all of our well known types. Here's an example of it in 3.17.3: Screen Shot 2021-08-30 at 2 48 19 PM

These are not compiled class files, these are just the source of the complied class files found in the analogous com.google.protobuf.* package.

Having said that, I see you're using protobuf directly through bazel, bypassing maven entirely. Historically, we discovered that the java_library targets that provide the runtimes were inconsistent with what maven was doing RE: these source file resources, and so the commit above fixed it by adding them. Here's a screenshot of the jars from your reproducer in both 3.17.3 and 3.18.0-rc1 built from bazel: Screen Shot 2021-08-30 at 2 50 53 PM

As you can see, this results in the addtion of the new google/protobuf/... folder, which I think your ClassGraph scanner is accidentally interpreting as a duplicate .class file. To be clear, I was unable to detect any duplicate classfiles.

The methodology I used to double check things was this:

bazel build src/main/java/com/example:main_deploy.jar && jar tf bazel-bin/src/main/java/com/example/main_deploy.jar | grep com/google/protobuf | sort

Which produced this result:

com/google/protobuf/
com/google/protobuf/AbstractMessage$Builder.class
com/google/protobuf/AbstractMessage$BuilderParent.class
com/google/protobuf/AbstractMessage.class
com/google/protobuf/AbstractMessageLite$Builder$LimitedInputStream.class
com/google/protobuf/AbstractMessageLite$Builder.class
com/google/protobuf/AbstractMessageLite$InternalOneOfEnum.class
com/google/protobuf/AbstractMessageLite.class
com/google/protobuf/AbstractParser.class
com/google/protobuf/AbstractProtobufList.class
com/google/protobuf/AllocatedBuffer$1.class
com/google/protobuf/AllocatedBuffer$2.class
com/google/protobuf/AllocatedBuffer.class
com/google/protobuf/Android.class
com/google/protobuf/Any$1.class
com/google/protobuf/Any$Builder.class
com/google/protobuf/Any.class
com/google/protobuf/AnyOrBuilder.class
com/google/protobuf/AnyProto.class
com/google/protobuf/Api$1.class
com/google/protobuf/Api$Builder.class
com/google/protobuf/Api.class
com/google/protobuf/ApiOrBuilder.class
com/google/protobuf/ApiProto.class
com/google/protobuf/ArrayDecoders$1.class
com/google/protobuf/ArrayDecoders$Registers.class
com/google/protobuf/ArrayDecoders.class
com/google/protobuf/BinaryReader$1.class
com/google/protobuf/BinaryReader$SafeHeapReader.class
com/google/protobuf/BinaryReader.class
com/google/protobuf/BinaryWriter$1.class
com/google/protobuf/BinaryWriter$SafeDirectWriter.class
com/google/protobuf/BinaryWriter$SafeHeapWriter.class
com/google/protobuf/BinaryWriter$UnsafeDirectWriter.class
com/google/protobuf/BinaryWriter$UnsafeHeapWriter.class
com/google/protobuf/BinaryWriter.class
com/google/protobuf/BlockingRpcChannel.class
com/google/protobuf/BlockingService.class
com/google/protobuf/BoolValue$1.class
com/google/protobuf/BoolValue$Builder.class
com/google/protobuf/BoolValue.class
com/google/protobuf/BoolValueOrBuilder.class
com/google/protobuf/BooleanArrayList.class
com/google/protobuf/BufferAllocator$1.class
com/google/protobuf/BufferAllocator.class
com/google/protobuf/ByteBufferWriter.class
com/google/protobuf/ByteOutput.class
com/google/protobuf/ByteString$1.class
com/google/protobuf/ByteString$2.class
com/google/protobuf/ByteString$AbstractByteIterator.class
com/google/protobuf/ByteString$ArraysByteArrayCopier.class
com/google/protobuf/ByteString$BoundedByteString.class
com/google/protobuf/ByteString$ByteArrayCopier.class
com/google/protobuf/ByteString$ByteIterator.class
com/google/protobuf/ByteString$CodedBuilder.class
com/google/protobuf/ByteString$LeafByteString.class
com/google/protobuf/ByteString$LiteralByteString.class
com/google/protobuf/ByteString$Output.class
com/google/protobuf/ByteString$SystemByteArrayCopier.class
com/google/protobuf/ByteString.class
com/google/protobuf/BytesValue$1.class
com/google/protobuf/BytesValue$Builder.class
com/google/protobuf/BytesValue.class
com/google/protobuf/BytesValueOrBuilder.class
com/google/protobuf/CodedInputStream$1.class
com/google/protobuf/CodedInputStream$ArrayDecoder.class
com/google/protobuf/CodedInputStream$IterableDirectByteBufferDecoder.class
com/google/protobuf/CodedInputStream$StreamDecoder$RefillCallback.class
com/google/protobuf/CodedInputStream$StreamDecoder$SkippedDataSink.class
com/google/protobuf/CodedInputStream$StreamDecoder.class
com/google/protobuf/CodedInputStream$UnsafeDirectNioDecoder.class
com/google/protobuf/CodedInputStream.class
com/google/protobuf/CodedInputStreamReader$1.class
com/google/protobuf/CodedInputStreamReader.class
com/google/protobuf/CodedOutputStream$1.class
com/google/protobuf/CodedOutputStream$AbstractBufferedEncoder.class
com/google/protobuf/CodedOutputStream$ArrayEncoder.class
com/google/protobuf/CodedOutputStream$ByteOutputEncoder.class
com/google/protobuf/CodedOutputStream$HeapNioEncoder.class
com/google/protobuf/CodedOutputStream$OutOfSpaceException.class
com/google/protobuf/CodedOutputStream$OutputStreamEncoder.class
com/google/protobuf/CodedOutputStream$SafeDirectNioEncoder.class
com/google/protobuf/CodedOutputStream$UnsafeDirectNioEncoder.class
com/google/protobuf/CodedOutputStream.class
com/google/protobuf/CodedOutputStreamWriter$1.class
com/google/protobuf/CodedOutputStreamWriter.class
com/google/protobuf/DescriptorMessageInfoFactory$1.class
com/google/protobuf/DescriptorMessageInfoFactory$2.class
com/google/protobuf/DescriptorMessageInfoFactory$3.class
com/google/protobuf/DescriptorMessageInfoFactory$IsInitializedCheckAnalyzer$Node.class
com/google/protobuf/DescriptorMessageInfoFactory$IsInitializedCheckAnalyzer$StronglyConnectedComponent.class
com/google/protobuf/DescriptorMessageInfoFactory$IsInitializedCheckAnalyzer.class
com/google/protobuf/DescriptorMessageInfoFactory$OneofState.class
com/google/protobuf/DescriptorMessageInfoFactory.class
com/google/protobuf/DescriptorProtos$1.class
com/google/protobuf/DescriptorProtos$DescriptorProto$1.class
com/google/protobuf/DescriptorProtos$DescriptorProto$Builder.class
com/google/protobuf/DescriptorProtos$DescriptorProto$ExtensionRange$1.class
com/google/protobuf/DescriptorProtos$DescriptorProto$ExtensionRange$Builder.class
com/google/protobuf/DescriptorProtos$DescriptorProto$ExtensionRange.class
com/google/protobuf/DescriptorProtos$DescriptorProto$ExtensionRangeOrBuilder.class
com/google/protobuf/DescriptorProtos$DescriptorProto$ReservedRange$1.class
com/google/protobuf/DescriptorProtos$DescriptorProto$ReservedRange$Builder.class
com/google/protobuf/DescriptorProtos$DescriptorProto$ReservedRange.class
com/google/protobuf/DescriptorProtos$DescriptorProto$ReservedRangeOrBuilder.class
com/google/protobuf/DescriptorProtos$DescriptorProto.class
com/google/protobuf/DescriptorProtos$DescriptorProtoOrBuilder.class
com/google/protobuf/DescriptorProtos$EnumDescriptorProto$1.class
com/google/protobuf/DescriptorProtos$EnumDescriptorProto$Builder.class
com/google/protobuf/DescriptorProtos$EnumDescriptorProto$EnumReservedRange$1.class
com/google/protobuf/DescriptorProtos$EnumDescriptorProto$EnumReservedRange$Builder.class
com/google/protobuf/DescriptorProtos$EnumDescriptorProto$EnumReservedRange.class
com/google/protobuf/DescriptorProtos$EnumDescriptorProto$EnumReservedRangeOrBuilder.class
com/google/protobuf/DescriptorProtos$EnumDescriptorProto.class
com/google/protobuf/DescriptorProtos$EnumDescriptorProtoOrBuilder.class
com/google/protobuf/DescriptorProtos$EnumOptions$1.class
com/google/protobuf/DescriptorProtos$EnumOptions$Builder.class
com/google/protobuf/DescriptorProtos$EnumOptions.class
com/google/protobuf/DescriptorProtos$EnumOptionsOrBuilder.class
com/google/protobuf/DescriptorProtos$EnumValueDescriptorProto$1.class
com/google/protobuf/DescriptorProtos$EnumValueDescriptorProto$Builder.class
com/google/protobuf/DescriptorProtos$EnumValueDescriptorProto.class
com/google/protobuf/DescriptorProtos$EnumValueDescriptorProtoOrBuilder.class
com/google/protobuf/DescriptorProtos$EnumValueOptions$1.class
com/google/protobuf/DescriptorProtos$EnumValueOptions$Builder.class
com/google/protobuf/DescriptorProtos$EnumValueOptions.class
com/google/protobuf/DescriptorProtos$EnumValueOptionsOrBuilder.class
com/google/protobuf/DescriptorProtos$ExtensionRangeOptions$1.class
com/google/protobuf/DescriptorProtos$ExtensionRangeOptions$Builder.class
com/google/protobuf/DescriptorProtos$ExtensionRangeOptions.class
com/google/protobuf/DescriptorProtos$ExtensionRangeOptionsOrBuilder.class
com/google/protobuf/DescriptorProtos$FieldDescriptorProto$1.class
com/google/protobuf/DescriptorProtos$FieldDescriptorProto$Builder.class
com/google/protobuf/DescriptorProtos$FieldDescriptorProto$Label$1.class
com/google/protobuf/DescriptorProtos$FieldDescriptorProto$Label.class
com/google/protobuf/DescriptorProtos$FieldDescriptorProto$Type$1.class
com/google/protobuf/DescriptorProtos$FieldDescriptorProto$Type.class
com/google/protobuf/DescriptorProtos$FieldDescriptorProto.class
com/google/protobuf/DescriptorProtos$FieldDescriptorProtoOrBuilder.class
com/google/protobuf/DescriptorProtos$FieldOptions$1.class
com/google/protobuf/DescriptorProtos$FieldOptions$Builder.class
com/google/protobuf/DescriptorProtos$FieldOptions$CType$1.class
com/google/protobuf/DescriptorProtos$FieldOptions$CType.class
com/google/protobuf/DescriptorProtos$FieldOptions$JSType$1.class
com/google/protobuf/DescriptorProtos$FieldOptions$JSType.class
com/google/protobuf/DescriptorProtos$FieldOptions.class
com/google/protobuf/DescriptorProtos$FieldOptionsOrBuilder.class
com/google/protobuf/DescriptorProtos$FileDescriptorProto$1.class
com/google/protobuf/DescriptorProtos$FileDescriptorProto$Builder.class
com/google/protobuf/DescriptorProtos$FileDescriptorProto.class
com/google/protobuf/DescriptorProtos$FileDescriptorProtoOrBuilder.class
com/google/protobuf/DescriptorProtos$FileDescriptorSet$1.class
com/google/protobuf/DescriptorProtos$FileDescriptorSet$Builder.class
com/google/protobuf/DescriptorProtos$FileDescriptorSet.class
com/google/protobuf/DescriptorProtos$FileDescriptorSetOrBuilder.class
com/google/protobuf/DescriptorProtos$FileOptions$1.class
com/google/protobuf/DescriptorProtos$FileOptions$Builder.class
com/google/protobuf/DescriptorProtos$FileOptions$OptimizeMode$1.class
com/google/protobuf/DescriptorProtos$FileOptions$OptimizeMode.class
com/google/protobuf/DescriptorProtos$FileOptions.class
com/google/protobuf/DescriptorProtos$FileOptionsOrBuilder.class
com/google/protobuf/DescriptorProtos$GeneratedCodeInfo$1.class
com/google/protobuf/DescriptorProtos$GeneratedCodeInfo$Annotation$1.class
com/google/protobuf/DescriptorProtos$GeneratedCodeInfo$Annotation$Builder.class
com/google/protobuf/DescriptorProtos$GeneratedCodeInfo$Annotation.class
com/google/protobuf/DescriptorProtos$GeneratedCodeInfo$AnnotationOrBuilder.class
com/google/protobuf/DescriptorProtos$GeneratedCodeInfo$Builder.class
com/google/protobuf/DescriptorProtos$GeneratedCodeInfo.class
com/google/protobuf/DescriptorProtos$GeneratedCodeInfoOrBuilder.class
com/google/protobuf/DescriptorProtos$MessageOptions$1.class
com/google/protobuf/DescriptorProtos$MessageOptions$Builder.class
com/google/protobuf/DescriptorProtos$MessageOptions.class
com/google/protobuf/DescriptorProtos$MessageOptionsOrBuilder.class
com/google/protobuf/DescriptorProtos$MethodDescriptorProto$1.class
com/google/protobuf/DescriptorProtos$MethodDescriptorProto$Builder.class
com/google/protobuf/DescriptorProtos$MethodDescriptorProto.class
com/google/protobuf/DescriptorProtos$MethodDescriptorProtoOrBuilder.class
com/google/protobuf/DescriptorProtos$MethodOptions$1.class
com/google/protobuf/DescriptorProtos$MethodOptions$Builder.class
com/google/protobuf/DescriptorProtos$MethodOptions$IdempotencyLevel$1.class
com/google/protobuf/DescriptorProtos$MethodOptions$IdempotencyLevel.class
com/google/protobuf/DescriptorProtos$MethodOptions.class
com/google/protobuf/DescriptorProtos$MethodOptionsOrBuilder.class
com/google/protobuf/DescriptorProtos$OneofDescriptorProto$1.class
com/google/protobuf/DescriptorProtos$OneofDescriptorProto$Builder.class
com/google/protobuf/DescriptorProtos$OneofDescriptorProto.class
com/google/protobuf/DescriptorProtos$OneofDescriptorProtoOrBuilder.class
com/google/protobuf/DescriptorProtos$OneofOptions$1.class
com/google/protobuf/DescriptorProtos$OneofOptions$Builder.class
com/google/protobuf/DescriptorProtos$OneofOptions.class
com/google/protobuf/DescriptorProtos$OneofOptionsOrBuilder.class
com/google/protobuf/DescriptorProtos$ServiceDescriptorProto$1.class
com/google/protobuf/DescriptorProtos$ServiceDescriptorProto$Builder.class
com/google/protobuf/DescriptorProtos$ServiceDescriptorProto.class
com/google/protobuf/DescriptorProtos$ServiceDescriptorProtoOrBuilder.class
com/google/protobuf/DescriptorProtos$ServiceOptions$1.class
com/google/protobuf/DescriptorProtos$ServiceOptions$Builder.class
com/google/protobuf/DescriptorProtos$ServiceOptions.class
com/google/protobuf/DescriptorProtos$ServiceOptionsOrBuilder.class
com/google/protobuf/DescriptorProtos$SourceCodeInfo$1.class
com/google/protobuf/DescriptorProtos$SourceCodeInfo$Builder.class
com/google/protobuf/DescriptorProtos$SourceCodeInfo$Location$1.class
com/google/protobuf/DescriptorProtos$SourceCodeInfo$Location$Builder.class
com/google/protobuf/DescriptorProtos$SourceCodeInfo$Location.class
com/google/protobuf/DescriptorProtos$SourceCodeInfo$LocationOrBuilder.class
com/google/protobuf/DescriptorProtos$SourceCodeInfo.class
com/google/protobuf/DescriptorProtos$SourceCodeInfoOrBuilder.class
com/google/protobuf/DescriptorProtos$UninterpretedOption$1.class
com/google/protobuf/DescriptorProtos$UninterpretedOption$Builder.class
com/google/protobuf/DescriptorProtos$UninterpretedOption$NamePart$1.class
com/google/protobuf/DescriptorProtos$UninterpretedOption$NamePart$Builder.class
com/google/protobuf/DescriptorProtos$UninterpretedOption$NamePart.class
com/google/protobuf/DescriptorProtos$UninterpretedOption$NamePartOrBuilder.class
com/google/protobuf/DescriptorProtos$UninterpretedOption.class
com/google/protobuf/DescriptorProtos$UninterpretedOptionOrBuilder.class
com/google/protobuf/DescriptorProtos.class
com/google/protobuf/Descriptors$1.class
com/google/protobuf/Descriptors$Descriptor.class
com/google/protobuf/Descriptors$DescriptorPool$DescriptorIntPair.class
com/google/protobuf/Descriptors$DescriptorPool$PackageDescriptor.class
com/google/protobuf/Descriptors$DescriptorPool$SearchFilter.class
com/google/protobuf/Descriptors$DescriptorPool.class
com/google/protobuf/Descriptors$DescriptorValidationException.class
com/google/protobuf/Descriptors$EnumDescriptor.class
com/google/protobuf/Descriptors$EnumValueDescriptor.class
com/google/protobuf/Descriptors$FieldDescriptor$JavaType.class
com/google/protobuf/Descriptors$FieldDescriptor$Type.class
com/google/protobuf/Descriptors$FieldDescriptor.class
com/google/protobuf/Descriptors$FileDescriptor$InternalDescriptorAssigner.class
com/google/protobuf/Descriptors$FileDescriptor$Syntax.class
com/google/protobuf/Descriptors$FileDescriptor.class
com/google/protobuf/Descriptors$GenericDescriptor.class
com/google/protobuf/Descriptors$MethodDescriptor.class
com/google/protobuf/Descriptors$OneofDescriptor.class
com/google/protobuf/Descriptors$ServiceDescriptor.class
com/google/protobuf/Descriptors.class
com/google/protobuf/DiscardUnknownFieldsParser$1.class
com/google/protobuf/DiscardUnknownFieldsParser.class
com/google/protobuf/DoubleArrayList.class
com/google/protobuf/DoubleValue$1.class
com/google/protobuf/DoubleValue$Builder.class
com/google/protobuf/DoubleValue.class
com/google/protobuf/DoubleValueOrBuilder.class
com/google/protobuf/Duration$1.class
com/google/protobuf/Duration$Builder.class
com/google/protobuf/Duration.class
com/google/protobuf/DurationOrBuilder.class
com/google/protobuf/DurationProto.class
com/google/protobuf/DynamicMessage$1.class
com/google/protobuf/DynamicMessage$Builder.class
com/google/protobuf/DynamicMessage.class
com/google/protobuf/Empty$1.class
com/google/protobuf/Empty$Builder.class
com/google/protobuf/Empty.class
com/google/protobuf/EmptyOrBuilder.class
com/google/protobuf/EmptyProto.class
com/google/protobuf/Enum$1.class
com/google/protobuf/Enum$Builder.class
com/google/protobuf/Enum.class
com/google/protobuf/EnumOrBuilder.class
com/google/protobuf/EnumValue$1.class
com/google/protobuf/EnumValue$Builder.class
com/google/protobuf/EnumValue.class
com/google/protobuf/EnumValueOrBuilder.class
com/google/protobuf/ExperimentalApi.class
com/google/protobuf/Extension$ExtensionType.class
com/google/protobuf/Extension$MessageType.class
com/google/protobuf/Extension.class
com/google/protobuf/ExtensionLite.class
com/google/protobuf/ExtensionRegistry$1.class
com/google/protobuf/ExtensionRegistry$DescriptorIntPair.class
com/google/protobuf/ExtensionRegistry$ExtensionInfo.class
com/google/protobuf/ExtensionRegistry.class
com/google/protobuf/ExtensionRegistryFactory.class
com/google/protobuf/ExtensionRegistryLite$ExtensionClassHolder.class
com/google/protobuf/ExtensionRegistryLite$ObjectIntPair.class
com/google/protobuf/ExtensionRegistryLite.class
com/google/protobuf/ExtensionSchema.class
com/google/protobuf/ExtensionSchemaFull$1.class
com/google/protobuf/ExtensionSchemaFull.class
com/google/protobuf/ExtensionSchemaLite$1.class
com/google/protobuf/ExtensionSchemaLite.class
com/google/protobuf/ExtensionSchemas.class
com/google/protobuf/Field$1.class
com/google/protobuf/Field$Builder.class
com/google/protobuf/Field$Cardinality$1.class
com/google/protobuf/Field$Cardinality.class
com/google/protobuf/Field$Kind$1.class
com/google/protobuf/Field$Kind.class
com/google/protobuf/Field.class
com/google/protobuf/FieldInfo$1.class
com/google/protobuf/FieldInfo$Builder.class
com/google/protobuf/FieldInfo.class
com/google/protobuf/FieldMask$1.class
com/google/protobuf/FieldMask$Builder.class
com/google/protobuf/FieldMask.class
com/google/protobuf/FieldMaskOrBuilder.class
com/google/protobuf/FieldMaskProto.class
com/google/protobuf/FieldOrBuilder.class
com/google/protobuf/FieldSet$1.class
com/google/protobuf/FieldSet$Builder.class
com/google/protobuf/FieldSet$FieldDescriptorLite.class
com/google/protobuf/FieldSet.class
com/google/protobuf/FieldType$1.class
com/google/protobuf/FieldType$Collection.class
com/google/protobuf/FieldType.class
com/google/protobuf/FloatArrayList.class
com/google/protobuf/FloatValue$1.class
com/google/protobuf/FloatValue$Builder.class
com/google/protobuf/FloatValue.class
com/google/protobuf/FloatValueOrBuilder.class
com/google/protobuf/GeneratedMessage$1.class
com/google/protobuf/GeneratedMessage$2.class
com/google/protobuf/GeneratedMessage$3.class
com/google/protobuf/GeneratedMessage$4.class
com/google/protobuf/GeneratedMessage$5.class
com/google/protobuf/GeneratedMessage$Builder$BuilderParentImpl.class
com/google/protobuf/GeneratedMessage$Builder.class
com/google/protobuf/GeneratedMessage$BuilderParent.class
com/google/protobuf/GeneratedMessage$CachedDescriptorRetriever.class
com/google/protobuf/GeneratedMessage$ExtendableBuilder.class
com/google/protobuf/GeneratedMessage$ExtendableMessage$ExtensionWriter.class
com/google/protobuf/GeneratedMessage$ExtendableMessage.class
com/google/protobuf/GeneratedMessage$ExtendableMessageOrBuilder.class
com/google/protobuf/GeneratedMessage$ExtensionDescriptorRetriever.class
com/google/protobuf/GeneratedMessage$FieldAccessorTable$FieldAccessor.class
com/google/protobuf/GeneratedMessage$FieldAccessorTable$MapFieldAccessor.class
com/google/protobuf/GeneratedMessage$FieldAccessorTable$OneofAccessor.class
com/google/protobuf/GeneratedMessage$FieldAccessorTable$RepeatedEnumFieldAccessor.class
com/google/protobuf/GeneratedMessage$FieldAccessorTable$RepeatedFieldAccessor.class
com/google/protobuf/GeneratedMessage$FieldAccessorTable$RepeatedMessageFieldAccessor.class
com/google/protobuf/GeneratedMessage$FieldAccessorTable$SingularEnumFieldAccessor.class
com/google/protobuf/GeneratedMessage$FieldAccessorTable$SingularFieldAccessor.class
com/google/protobuf/GeneratedMessage$FieldAccessorTable$SingularMessageFieldAccessor.class
com/google/protobuf/GeneratedMessage$FieldAccessorTable$SingularStringFieldAccessor.class
com/google/protobuf/GeneratedMessage$FieldAccessorTable.class
com/google/protobuf/GeneratedMessage$GeneratedExtension$1.class
com/google/protobuf/GeneratedMessage$GeneratedExtension.class
com/google/protobuf/GeneratedMessage.class
com/google/protobuf/GeneratedMessageInfoFactory.class
com/google/protobuf/GeneratedMessageLite$1.class
com/google/protobuf/GeneratedMessageLite$Builder.class
com/google/protobuf/GeneratedMessageLite$DefaultInstanceBasedParser.class
com/google/protobuf/GeneratedMessageLite$ExtendableBuilder.class
com/google/protobuf/GeneratedMessageLite$ExtendableMessage$ExtensionWriter.class
com/google/protobuf/GeneratedMessageLite$ExtendableMessage.class
com/google/protobuf/GeneratedMessageLite$ExtendableMessageOrBuilder.class
com/google/protobuf/GeneratedMessageLite$ExtensionDescriptor.class
com/google/protobuf/GeneratedMessageLite$GeneratedExtension.class
com/google/protobuf/GeneratedMessageLite$MethodToInvoke.class
com/google/protobuf/GeneratedMessageLite$SerializedForm.class
com/google/protobuf/GeneratedMessageLite.class
com/google/protobuf/GeneratedMessageV3$1.class
com/google/protobuf/GeneratedMessageV3$Builder$BuilderParentImpl.class
com/google/protobuf/GeneratedMessageV3$Builder.class
com/google/protobuf/GeneratedMessageV3$BuilderParent.class
com/google/protobuf/GeneratedMessageV3$ExtendableBuilder.class
com/google/protobuf/GeneratedMessageV3$ExtendableMessage$ExtensionWriter.class
com/google/protobuf/GeneratedMessageV3$ExtendableMessage.class
com/google/protobuf/GeneratedMessageV3$ExtendableMessageOrBuilder.class
com/google/protobuf/GeneratedMessageV3$ExtensionDescriptorRetriever.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$FieldAccessor.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$MapFieldAccessor.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$OneofAccessor.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$RepeatedEnumFieldAccessor.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$RepeatedFieldAccessor$MethodInvoker.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$RepeatedFieldAccessor$ReflectionInvoker.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$RepeatedFieldAccessor.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$RepeatedMessageFieldAccessor.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$SingularEnumFieldAccessor.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$SingularFieldAccessor$MethodInvoker.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$SingularFieldAccessor$ReflectionInvoker.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$SingularFieldAccessor.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$SingularMessageFieldAccessor.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable$SingularStringFieldAccessor.class
com/google/protobuf/GeneratedMessageV3$FieldAccessorTable.class
com/google/protobuf/GeneratedMessageV3$UnusedPrivateParameter.class
com/google/protobuf/GeneratedMessageV3.class
com/google/protobuf/Int32Value$1.class
com/google/protobuf/Int32Value$Builder.class
com/google/protobuf/Int32Value.class
com/google/protobuf/Int32ValueOrBuilder.class
com/google/protobuf/Int64Value$1.class
com/google/protobuf/Int64Value$Builder.class
com/google/protobuf/Int64Value.class
com/google/protobuf/Int64ValueOrBuilder.class
com/google/protobuf/IntArrayList.class
com/google/protobuf/Internal$BooleanList.class
com/google/protobuf/Internal$DoubleList.class
com/google/protobuf/Internal$EnumLite.class
com/google/protobuf/Internal$EnumLiteMap.class
com/google/protobuf/Internal$EnumVerifier.class
com/google/protobuf/Internal$FloatList.class
com/google/protobuf/Internal$IntList.class
com/google/protobuf/Internal$ListAdapter$Converter.class
com/google/protobuf/Internal$ListAdapter.class
com/google/protobuf/Internal$LongList.class
com/google/protobuf/Internal$MapAdapter$1.class
com/google/protobuf/Internal$MapAdapter$Converter.class
com/google/protobuf/Internal$MapAdapter$EntryAdapter.class
com/google/protobuf/Internal$MapAdapter$IteratorAdapter.class
com/google/protobuf/Internal$MapAdapter$SetAdapter.class
com/google/protobuf/Internal$MapAdapter.class
com/google/protobuf/Internal$ProtobufList.class
com/google/protobuf/Internal.class
com/google/protobuf/InvalidProtocolBufferException$InvalidWireTypeException.class
com/google/protobuf/InvalidProtocolBufferException.class
com/google/protobuf/IterableByteBufferInputStream.class
com/google/protobuf/JavaType.class
com/google/protobuf/LazyField$1.class
com/google/protobuf/LazyField$LazyEntry.class
com/google/protobuf/LazyField$LazyIterator.class
com/google/protobuf/LazyField.class
com/google/protobuf/LazyFieldLite.class
com/google/protobuf/LazyStringArrayList$ByteArrayListView.class
com/google/protobuf/LazyStringArrayList$ByteStringListView.class
com/google/protobuf/LazyStringArrayList.class
com/google/protobuf/LazyStringList.class
com/google/protobuf/ListFieldSchema$1.class
com/google/protobuf/ListFieldSchema$ListFieldSchemaFull.class
com/google/protobuf/ListFieldSchema$ListFieldSchemaLite.class
com/google/protobuf/ListFieldSchema.class
com/google/protobuf/ListValue$1.class
com/google/protobuf/ListValue$Builder.class
com/google/protobuf/ListValue.class
com/google/protobuf/ListValueOrBuilder.class
com/google/protobuf/LongArrayList.class
com/google/protobuf/ManifestSchemaFactory$1.class
com/google/protobuf/ManifestSchemaFactory$CompositeMessageInfoFactory.class
com/google/protobuf/ManifestSchemaFactory.class
com/google/protobuf/MapEntry$1.class
com/google/protobuf/MapEntry$Builder.class
com/google/protobuf/MapEntry$Metadata$1.class
com/google/protobuf/MapEntry$Metadata.class
com/google/protobuf/MapEntry.class
com/google/protobuf/MapEntryLite$1.class
com/google/protobuf/MapEntryLite$Metadata.class
com/google/protobuf/MapEntryLite.class
com/google/protobuf/MapField$Converter.class
com/google/protobuf/MapField$ImmutableMessageConverter.class
com/google/protobuf/MapField$MutatabilityAwareMap$MutatabilityAwareCollection.class
com/google/protobuf/MapField$MutatabilityAwareMap$MutatabilityAwareIterator.class
com/google/protobuf/MapField$MutatabilityAwareMap$MutatabilityAwareSet.class
com/google/protobuf/MapField$MutatabilityAwareMap.class
com/google/protobuf/MapField$StorageMode.class
com/google/protobuf/MapField.class
com/google/protobuf/MapFieldLite.class
com/google/protobuf/MapFieldSchema.class
com/google/protobuf/MapFieldSchemaFull.class
com/google/protobuf/MapFieldSchemaLite.class
com/google/protobuf/MapFieldSchemas.class
com/google/protobuf/Message$Builder.class
com/google/protobuf/Message.class
com/google/protobuf/MessageInfo.class
com/google/protobuf/MessageInfoFactory.class
com/google/protobuf/MessageLite$Builder.class
com/google/protobuf/MessageLite.class
com/google/protobuf/MessageLiteOrBuilder.class
com/google/protobuf/MessageLiteToString.class
com/google/protobuf/MessageOrBuilder.class
com/google/protobuf/MessageReflection$1.class
com/google/protobuf/MessageReflection$BuilderAdapter.class
com/google/protobuf/MessageReflection$ExtensionAdapter.class
com/google/protobuf/MessageReflection$MergeTarget$ContainerType.class
com/google/protobuf/MessageReflection$MergeTarget.class
com/google/protobuf/MessageReflection.class
com/google/protobuf/MessageSchema$1.class
com/google/protobuf/MessageSchema.class
com/google/protobuf/MessageSetSchema.class
com/google/protobuf/Method$1.class
com/google/protobuf/Method$Builder.class
com/google/protobuf/Method.class
com/google/protobuf/MethodOrBuilder.class
com/google/protobuf/Mixin$1.class
com/google/protobuf/Mixin$Builder.class
com/google/protobuf/Mixin.class
com/google/protobuf/MixinOrBuilder.class
com/google/protobuf/MutabilityOracle$1.class
com/google/protobuf/MutabilityOracle.class
com/google/protobuf/NewInstanceSchema.class
com/google/protobuf/NewInstanceSchemaFull.class
com/google/protobuf/NewInstanceSchemaLite.class
com/google/protobuf/NewInstanceSchemas.class
com/google/protobuf/NioByteString$1.class
com/google/protobuf/NioByteString.class
com/google/protobuf/NullValue$1.class
com/google/protobuf/NullValue.class
com/google/protobuf/OneofInfo.class
com/google/protobuf/Option$1.class
com/google/protobuf/Option$Builder.class
com/google/protobuf/Option.class
com/google/protobuf/OptionOrBuilder.class
com/google/protobuf/Parser.class
com/google/protobuf/PrimitiveNonBoxingCollection.class
com/google/protobuf/ProtoSyntax.class
com/google/protobuf/Protobuf.class
com/google/protobuf/ProtobufArrayList.class
com/google/protobuf/ProtobufLists.class
com/google/protobuf/ProtocolMessageEnum.class
com/google/protobuf/ProtocolStringList.class
com/google/protobuf/RawMessageInfo.class
com/google/protobuf/Reader.class
com/google/protobuf/RepeatedFieldBuilder$BuilderExternalList.class
com/google/protobuf/RepeatedFieldBuilder$MessageExternalList.class
com/google/protobuf/RepeatedFieldBuilder$MessageOrBuilderExternalList.class
com/google/protobuf/RepeatedFieldBuilder.class
com/google/protobuf/RepeatedFieldBuilderV3$BuilderExternalList.class
com/google/protobuf/RepeatedFieldBuilderV3$MessageExternalList.class
com/google/protobuf/RepeatedFieldBuilderV3$MessageOrBuilderExternalList.class
com/google/protobuf/RepeatedFieldBuilderV3.class
com/google/protobuf/RopeByteString$1.class
com/google/protobuf/RopeByteString$Balancer.class
com/google/protobuf/RopeByteString$PieceIterator.class
com/google/protobuf/RopeByteString$RopeInputStream.class
com/google/protobuf/RopeByteString.class
com/google/protobuf/RpcCallback.class
com/google/protobuf/RpcChannel.class
com/google/protobuf/RpcController.class
com/google/protobuf/RpcUtil$1.class
com/google/protobuf/RpcUtil$2.class
com/google/protobuf/RpcUtil$AlreadyCalledException.class
com/google/protobuf/RpcUtil.class
com/google/protobuf/Schema.class
com/google/protobuf/SchemaFactory.class
com/google/protobuf/SchemaUtil.class
com/google/protobuf/Service.class
com/google/protobuf/ServiceException.class
com/google/protobuf/SingleFieldBuilder.class
com/google/protobuf/SingleFieldBuilderV3.class
com/google/protobuf/SmallSortedMap$1.class
com/google/protobuf/SmallSortedMap$DescendingEntryIterator.class
com/google/protobuf/SmallSortedMap$DescendingEntrySet.class
com/google/protobuf/SmallSortedMap$EmptySet$1.class
com/google/protobuf/SmallSortedMap$EmptySet$2.class
com/google/protobuf/SmallSortedMap$EmptySet.class
com/google/protobuf/SmallSortedMap$Entry.class
com/google/protobuf/SmallSortedMap$EntryIterator.class
com/google/protobuf/SmallSortedMap$EntrySet.class
com/google/protobuf/SmallSortedMap.class
com/google/protobuf/SourceContext$1.class
com/google/protobuf/SourceContext$Builder.class
com/google/protobuf/SourceContext.class
com/google/protobuf/SourceContextOrBuilder.class
com/google/protobuf/SourceContextProto.class
com/google/protobuf/StringValue$1.class
com/google/protobuf/StringValue$Builder.class
com/google/protobuf/StringValue.class
com/google/protobuf/StringValueOrBuilder.class
com/google/protobuf/Struct$1.class
com/google/protobuf/Struct$Builder.class
com/google/protobuf/Struct$FieldsDefaultEntryHolder.class
com/google/protobuf/Struct.class
com/google/protobuf/StructOrBuilder.class
com/google/protobuf/StructProto.class
com/google/protobuf/StructuralMessageInfo$Builder.class
com/google/protobuf/StructuralMessageInfo.class
com/google/protobuf/Syntax$1.class
com/google/protobuf/Syntax.class
com/google/protobuf/TextFormat$1.class
com/google/protobuf/TextFormat$InvalidEscapeSequenceException.class
com/google/protobuf/TextFormat$ParseException.class
com/google/protobuf/TextFormat$Parser$Builder.class
com/google/protobuf/TextFormat$Parser$SingularOverwritePolicy.class
com/google/protobuf/TextFormat$Parser$UnknownField$Type.class
com/google/protobuf/TextFormat$Parser$UnknownField.class
com/google/protobuf/TextFormat$Parser.class
com/google/protobuf/TextFormat$Printer$MapEntryAdapter.class
com/google/protobuf/TextFormat$Printer.class
com/google/protobuf/TextFormat$TextGenerator.class
com/google/protobuf/TextFormat$Tokenizer.class
com/google/protobuf/TextFormat$UnknownFieldParseException.class
com/google/protobuf/TextFormat.class
com/google/protobuf/TextFormatEscaper$1.class
com/google/protobuf/TextFormatEscaper$2.class
com/google/protobuf/TextFormatEscaper$ByteSequence.class
com/google/protobuf/TextFormatEscaper.class
com/google/protobuf/TextFormatParseInfoTree$1.class
com/google/protobuf/TextFormatParseInfoTree$Builder.class
com/google/protobuf/TextFormatParseInfoTree.class
com/google/protobuf/TextFormatParseLocation.class
com/google/protobuf/Timestamp$1.class
com/google/protobuf/Timestamp$Builder.class
com/google/protobuf/Timestamp.class
com/google/protobuf/TimestampOrBuilder.class
com/google/protobuf/TimestampProto.class
com/google/protobuf/Type$1.class
com/google/protobuf/Type$Builder.class
com/google/protobuf/Type.class
com/google/protobuf/TypeOrBuilder.class
com/google/protobuf/TypeProto.class
com/google/protobuf/TypeRegistry$1.class
com/google/protobuf/TypeRegistry$Builder.class
com/google/protobuf/TypeRegistry$EmptyTypeRegistryHolder.class
com/google/protobuf/TypeRegistry.class
com/google/protobuf/UInt32Value$1.class
com/google/protobuf/UInt32Value$Builder.class
com/google/protobuf/UInt32Value.class
com/google/protobuf/UInt32ValueOrBuilder.class
com/google/protobuf/UInt64Value$1.class
com/google/protobuf/UInt64Value$Builder.class
com/google/protobuf/UInt64Value.class
com/google/protobuf/UInt64ValueOrBuilder.class
com/google/protobuf/UninitializedMessageException.class
com/google/protobuf/UnknownFieldSchema.class
com/google/protobuf/UnknownFieldSet$1.class
com/google/protobuf/UnknownFieldSet$Builder.class
com/google/protobuf/UnknownFieldSet$Field$Builder.class
com/google/protobuf/UnknownFieldSet$Field.class
com/google/protobuf/UnknownFieldSet$Parser.class
com/google/protobuf/UnknownFieldSet.class
com/google/protobuf/UnknownFieldSetLite.class
com/google/protobuf/UnknownFieldSetLiteSchema.class
com/google/protobuf/UnknownFieldSetSchema.class
com/google/protobuf/UnmodifiableLazyStringList$1.class
com/google/protobuf/UnmodifiableLazyStringList$2.class
com/google/protobuf/UnmodifiableLazyStringList.class
com/google/protobuf/UnsafeByteOperations.class
com/google/protobuf/UnsafeUtil$1.class
com/google/protobuf/UnsafeUtil$Android32MemoryAccessor.class
com/google/protobuf/UnsafeUtil$Android64MemoryAccessor.class
com/google/protobuf/UnsafeUtil$JvmMemoryAccessor.class
com/google/protobuf/UnsafeUtil$MemoryAccessor.class
com/google/protobuf/UnsafeUtil.class
com/google/protobuf/Utf8$DecodeUtil.class
com/google/protobuf/Utf8$Processor.class
com/google/protobuf/Utf8$SafeProcessor.class
com/google/protobuf/Utf8$UnpairedSurrogateException.class
com/google/protobuf/Utf8$UnsafeProcessor.class
com/google/protobuf/Utf8.class
com/google/protobuf/Value$1.class
com/google/protobuf/Value$2.class
com/google/protobuf/Value$Builder.class
com/google/protobuf/Value$KindCase.class
com/google/protobuf/Value.class
com/google/protobuf/ValueOrBuilder.class
com/google/protobuf/WireFormat$1.class
com/google/protobuf/WireFormat$FieldType$1.class
com/google/protobuf/WireFormat$FieldType$2.class
com/google/protobuf/WireFormat$FieldType$3.class
com/google/protobuf/WireFormat$FieldType$4.class
com/google/protobuf/WireFormat$FieldType.class
com/google/protobuf/WireFormat$JavaType.class
com/google/protobuf/WireFormat$Utf8Validation$1.class
com/google/protobuf/WireFormat$Utf8Validation$2.class
com/google/protobuf/WireFormat$Utf8Validation$3.class
com/google/protobuf/WireFormat$Utf8Validation.class
com/google/protobuf/WireFormat.class
com/google/protobuf/WrappersProto.class
com/google/protobuf/Writer$FieldOrder.class
com/google/protobuf/Writer.class
com/google/protobuf/compiler/
com/google/protobuf/compiler/PluginProtos$1.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorRequest$1.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorRequest$Builder.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorRequest.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorRequestOrBuilder.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorResponse$1.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorResponse$Builder.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorResponse$Feature$1.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorResponse$Feature.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorResponse$File$1.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorResponse$File$Builder.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorResponse$File.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorResponse$FileOrBuilder.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorResponse.class
com/google/protobuf/compiler/PluginProtos$CodeGeneratorResponseOrBuilder.class
com/google/protobuf/compiler/PluginProtos$Version$1.class
com/google/protobuf/compiler/PluginProtos$Version$Builder.class
com/google/protobuf/compiler/PluginProtos$Version.class
com/google/protobuf/compiler/PluginProtos$VersionOrBuilder.class
com/google/protobuf/compiler/PluginProtos.class

There are no duplicate classes I can observe.

So having said all this, you might ask why those .proto files are bundled in the maven jar, and I don't have a great answer for you off the top of my head (historical choices?), but what you're running into is us rectifying the discrepancy between what we publish to maven and what we provide through Bazel.

Hope this was helpful!

eikemeier commented 3 years ago

The problem with you analysis is that you use the merged _deploy.jar target, which chooses just one of the duplicates. Of course there can't be duplicate resources there.

This is also obvious when you run

java -jar "$(bazel info bazel-bin)/src/main/java/com/example/main_deploy.jar"

The duplicates are gone.

Also, AFAICT I'm using the official Bazel rules here.

When you do bazel query "kind('java.*library' , deps(//:main))", you'll get

//src/main/proto/example/v1:examplev1_java_proto
@rules_jvm_external//private/tools/java/rules/jvm/external:byte-streams
@com_google_protobuf//java/core:core-lib
@com_google_protobuf//java/core:lite_runtime_only

Then bazel build @com_google_protobuf//java/core:core-lib

...
  bazel-bin/external/com_google_protobuf/java/core/libcore-lib.jar

and for bazel build @com_google_protobuf//java/core:lite_runtime_only

...
  bazel-bin/external/com_google_protobuf/java/core/liblite_runtime_only.jar

bazel build //src/main/proto/example/v1:examplev1_java_proto

...
  bazel-bin/external/com_google_protobuf/libtimestamp_proto-speed.jar
  bazel-bin/src/main/proto/example/v1/libexamplev1_proto-speed.jar
  bazel-bin/src/main/proto/example/v1/examplev1_proto-speed-src.jar

So in 3.17 bazel-bin/external/com_google_protobuf/java/core/libcore-lib.jar, bazel-bin/external/com_google_protobuf/java/core/liblite_runtime_only.jar and bazel-bin/external/com_google_protobuf/libtimestamp_proto-speed.jar were on the class path when you run bazel run //:main. This is also relevant because it is included in java_image container images.

In 3.17 you had only this:

In v3.18.0-rc1 the class path has changed, see bazel query "kind('maven_project_jar' , deps(//:main))", and bazel build @com_google_protobuf//java/core:core-project:

...
  bazel-bin/external/com_google_protobuf/java/core/core-project.jar

So, in v3.18.0-rc1 the dependency to libcore-lib.jar is replaced with core-project.jar, the relevant content being:

This, together with liblite_runtime_only.jar and libtimestamp_proto-speed.jar results in the new duplicates. They are of course gone in the merged target, but not every deployment uses merged targets.

eikemeier commented 3 years ago

@perezd Please reopen, so I don't have to file another issue.

eikemeier commented 3 years ago

I've also added a docker build target to the demonstration in https://github.com/eikemeier/proto-java. bazel build //:container should make the duplicates easier to find.

With tar xOf "$(bazel info bazel-bin)/src/main/java/com/example/container-layer.tar" ./app/com_example_proto_java/src/main/java/com/example/container.classpath | tr : \\n you'll get

...
/app/com_example_proto_java/../com_google_protobuf/libtimestamp_proto-speed.jar
/app/com_example_proto_java/../com_google_protobuf/java/core/core-project.jar
/app/com_example_proto_java/../com_google_protobuf/java/core/liblite_runtime_only.jar
...

which are the libraries mentioned above. So it could be that 89a9f45 is the culprit, especially replacing java_library by java_export.

java_exportexplicitly merges its dependencies, so we'll get the classes of liblite_runtime_only.jar twice. In my container example that's 505991 bytes.

eikemeier commented 3 years ago

Thanks. I've also added https://github.com/eikemeier/proto-java/blob/main/8925.patch, which eliminates the duplicates.

This patch might have other unintended consequences — I'm not familiar enough with the code base —, but it should help diagnosing the issue.

perezd commented 3 years ago

OK This is interesting, I think there may be some unexpected side-effects of rules_jvm_external's java_export rule at play here, will dig in deeper. Thank you for this extended analysis.

perezd commented 3 years ago

Additional context, we've been using these rules to deprecate the maven-based builds to a Bazel-powered build which required us to replicate the jar build logic, however, the unintended consequence was what you've discovered, so we may need to revert the java_export rule changes and provide standard java_library instances for Bazel users that are separate from the release artifacts for maven.

eikemeier commented 3 years ago

This is the line where maven_projectmerges its files: https://github.com/bazelbuild/rules_jvm_external/blob/4.1/private/rules/maven_project_jar.bzl#L25

It's also documented in line 88.

Maybe it would be an idea to not pull in liblite_runtime_only.jar?

perezd commented 3 years ago

So reading this: https://github.com/bazelbuild/rules_jvm_external/blob/master/private/rules/java_export.bzl#L66

I was under the impression the existing behavior was preserved, it's only the case that if you depend on :core-project that you'd invoke maven_project_jar

Here's how it used to look: https://github.com/perezd/protobuf/blob/bc45f92262a8e0e6e1ab7302c72a380eb0346f8e/java/core/BUILD

perezd commented 3 years ago

argh, there's an alias at the end.

eikemeier commented 3 years ago

You could also do the library like before and then let maven_project_jar just merge it with its dependencies.

perezd commented 3 years ago

yeah, that's a PR I'm attempting right now. I'd love for you to patch and verify when it's ready to confirm its what you expect. Thank you for further investigating this with me!

eikemeier commented 3 years ago

Thanks for taking this PR so quickly. It would be awfully nice if we could address libtimestamp_proto-speed.jar while we are here. It's not so much an issue, but somehow annoying.

perezd commented 3 years ago

Sure, I'll see what I can do, it's probably duplicated or missing from a list. Part of our current repo woes is keeping dual build systems w/ Maven+Bazel, so we have discrepancies like this occur, which is why we're trying to unify :)

perezd commented 3 years ago

OK, so if you pull that PR, confirm the original behavior is as expected, and then I can checkout timestamp.

eikemeier commented 3 years ago

OK, so if you pull that PR, confirm the original behavior is as expected, and then I can checkout timestamp.

Seems to work so far:

https://github.com/eikemeier/proto-java/tree/perezd

Thanks again.

perezd commented 3 years ago

Happy to! Thanks for diagnosing the issue so thoroughly!

perezd commented 3 years ago

OK chatting, w/ a teammate, I think that the reason for dupes on timestamp comes from the fact that proto_library deps timestamp_proto, and gets converted automatically to a java proto transitively through the java_proto_library target you define, which makes sense.

The challenge here is that Bazel cannot see that this well known type can be reused because we also compile an identical copy into the srcs of ":core" and ":lite" java_library targets. I'm going to see if I can generate those targets atomically and treat them as deps for core/lite, and see if this goes away as a result.

perezd commented 3 years ago

Naively attempting to create java_proto_library targets for the well known types and then treating them as deps for the core runtime results in a circular dependency w/ bazel toolchains:

 //src/main/java/com/example:main
 //src/main/proto/example/v1:examplev1_java_proto
 //src/main/proto/example/v1:examplev1_proto with aspect BazelJavaProtoAspect
.-> @com_google_protobuf//:java_toolchain
|   @com_google_protobuf//java/core:toolchain
|   @com_google_protobuf//java/core:core
|   @com_google_protobuf//:descriptor_java_proto
|   @com_google_protobuf//:descriptor_proto with aspect BazelJavaProtoAspect
`-- @com_google_protobuf//:java_toolchain

This is the core reason for the dupes. You'll find that if you depend on "@com_google_protobuf//:descriptor_proto" in a proto_library you'll get dupes of that too, it's any proto we provide that you might use that the runtime also depends on that'll end up like this unfortunately.

perezd commented 3 years ago

OK I think I've almost got all the dupes erased, we're stuck on one blocker here: https://github.com/bazelbuild/bazel/issues/13928

Yannic commented 3 years ago

/cc @comius FYI

perezd commented 3 years ago

OK, lite proto support for blacklisted_protos attribute is now checked into bazel, so it won't do much until the next cut of bazel but it will eventually eliminate the duplicate classes. I'll go ahead and send the PR for it now.

perezd commented 3 years ago

Actually, it appears I already submitted it. So, it'll start working as intended in the next Bazel release, thanks for helping us with this bug!