search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.31k stars 1.24k forks source link

PinotOutputFormatTest.testPinotOutputFormat fails with java.lang.NoSuchMethodError #7015

Open daniellavoie opened 3 years ago

daniellavoie commented 3 years ago

Our internal CI picked up a consistent regression with the PinotOutputFormatTest from pinot-hadoop.

The build is running using docker image 3.6.3-adoptopenjdk-8 and running the with mvn verify -Pbin-dist.

[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.512 s <<< FAILURE! - in org.apache.pinot.hadoop.io.PinotOutputFormatTest
[ERROR] org.apache.pinot.hadoop.io.PinotOutputFormatTest.testPinotOutputFormat  Time elapsed: 5.95 s  <<< FAILURE!
java.lang.NoSuchMethodError: org.reflections.util.ConfigurationBuilder.filterInputsBy(Lcom/google/common/base/Predicate;)Lorg/reflections/util/ConfigurationBuilder;
    at org.apache.pinot.segment.spi.loader.SegmentDirectoryLoaderRegistry.<clinit>(SegmentDirectoryLoaderRegistry.java:49)
    at org.apache.pinot.segment.local.segment.index.converter.SegmentV1V2ToV3FormatConverter.copyIndexData(SegmentV1V2ToV3FormatConverter.java:143)
    at org.apache.pinot.segment.local.segment.index.converter.SegmentV1V2ToV3FormatConverter.convert(SegmentV1V2ToV3FormatConverter.java:89)
    at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.convertFormatIfNecessary(SegmentIndexCreationDriverImpl.java:371)
    at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:304)
    at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:257)
    at org.apache.pinot.hadoop.io.PinotRecordWriter.createSegment(PinotRecordWriter.java:106)
    at org.apache.pinot.hadoop.io.PinotRecordWriter.close(PinotRecordWriter.java:92)
    at org.apache.pinot.hadoop.io.PinotOutputFormatTest.testPinotOutputFormat(PinotOutputFormatTest.java:90)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
timsants commented 3 years ago

I suspect we are getting a conflict with the org.reflections library. The latest version does not depend on the com.google.common.base.Predicate predicate anymore and now depends on java.util.function.Predicate

https://github.com/ronmamo/reflections/blob/master/src/main/java/org/reflections/util/ConfigurationBuilder.java#L26

timsants commented 3 years ago

@xiangfu0's PR with the fix: https://github.com/apache/incubator-pinot/pull/7030.

TLDR: Test was depending on a shaded version of pinot-common which shaded com.google.guava. So the shading on pinot-common was removed and transferred over to the jar of pinot-hadoop.