search

holdenk / spark-testing-base

Base classes to use when writing tests with Spark
Apache License 2.0
1.51k stars 359 forks source link

hadoop-hdfs: NoSuchMethodError - org.apache.hadoop.tracing.SpanReceiverHost.getInstance #260

Open sachinlala opened 6 years ago

sachinlala commented 6 years ago

I'm trying to write unit-tests to assert Spark and HDFS interactions. I am using SharedJavaSparkContext from spark-testing-base lib.

Below is the @BeforeClass setup:

Configuration hdfsConfiguration = new HdfsConfiguration();
File testDir = new File("./target/hdfs/").getAbsoluteFile();
FileUtil.fullyDelete(testDir);
hdfsConfiguration.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, testDir.getAbsolutePath());
MiniDFSCluster.Builder builder = new MiniDFSCluster.Builder(hdfsConfiguration);
MiniDFSCluster hdfsCluster = builder.build();
FileSystem fileSystem = FileSystem.get(hdfsConfiguration);

The POM level dependency is as follows:

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-hdfs</artifactId>
  <version>2.7.0</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-minicluster</artifactId>
  <version>2.7.0</version>
  <scope>test</scope>
</dependency>

I'm getting a failure at this line:

MiniDFSCluster hdfsCluster = builder.build();

Error Stack:

java.lang.NoSuchMethodError: org.apache.hadoop.tracing.SpanReceiverHost.getInstance(Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/tracing/SpanReceiverHost;

at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:641)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:810)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:794)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1487)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1115)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:986)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:815)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:475)
at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:434)

Investigation so far:

I have run mvn dependency:tree to confirm that the hadoop-hdfs library version is upwards of 2.6* (confirmed for the transitive dependencies).

Checked also in ~/.m2/repository and cleaned-up the org/apache/hadoop directory to ensure there is no stale version present there.

Any help/pointers will be appreciated. I will post an update if I'm able to resolve it meanwhile.

goujon-j commented 4 years ago

Hi, I'm looking for the same mistake. Did you find what you were looking for?