search

aditya-grover / node2vec

http://snap.stanford.edu/node2vec/
MIT License
2.61k stars 912 forks source link

Exception in thread "main" java.lang.NoSuchMethodError #61

Open anirband opened 5 years ago

anirband commented 5 years ago

127:node2vec ani.das$ /Users/ani.das/spark-2.4.0-bin-hadoop2.7/bin/spark-submit --class com.navercorp.Main ./node2vec_spark/target/node2vec-0.0.1-SNAPSHOT.jar --cmd randomwalk --p 10.0 --q 10.0 --walkLength 50 --numWalks 5 --weighted --window 6 --input /Users/ani.das/Projects/phase2/data/runa/tf_idf_1.txt --output /Users/ani.das/Projects/phase2/data/runa/emb/tf_idf_1.emb 2019-01-26 22:29:11 WARN Utils:66 - Your hostname, 127.0.0.1 resolves to a loopback address: 127.0.0.1; using 192.168.1.73 instead (on interface en0) 2019-01-26 22:29:11 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address 2019-01-26 22:29:11 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror; at com.navercorp.Main$Params.(Main.scala:15) at com.navercorp.Main$.(Main.scala:32) at com.navercorp.Main$.(Main.scala) at com.navercorp.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 2019-01-26 22:29:12 INFO ShutdownHookManager:54 - Shutdown hook called 2019-01-26 22:29:12 INFO ShutdownHookManager:54 - Deleting directory /private/var/folders/4c/s3nt_0s96z57zfxc3dlyq3swjl4fq9/T/spark-07c375e6-0b97-4c68-8040-797a0e18d68b

DozenCoder commented 5 years ago

@anirband You should change your spark version, I also found that this program cannot work in spark2.4.0, it works well in spark2.3.0 and 2.3.2.

anirband commented 5 years ago

Any suggestion on what to do with spark 2.4.0?

neilaronson commented 5 years ago

I've tried with Spark 2.3.2, Spark 2.4.0 and Spark 2.1 and all give me this error. This package seems effectively unusable

zsavvas commented 5 years ago

Had the same issue. I managed to solve it by running a Spark version that supports Scala version 2.10. In my case i was able to run it with Spark version 1.6.0.

DozenCoder commented 5 years ago

Maybe you should change other dependency and plugin version in pom.xml to make them work with spark new version, you can view my pom.xml.

ZhiyuanCao98 commented 5 years ago

Maybe you should change other dependency and plugin version in pom.xml to make them work with spark new version, you can view my pom.xml.

Can you show your pom.xml to me since I encounter the same error Thanks!

DozenCoder commented 5 years ago

@q463746583 , you can try this: 

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.navercorp</groupId>
    <artifactId>node2vec</artifactId>
    <packaging>jar</packaging>
    <version>0.1.2-SNAPSHOT</version>

    <name>node2vec_spark</name>
    <url>https://github.com/superPershing</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <shadedClassifier>bin</shadedClassifier>
        <maven-shade-plugin.version>3.2.1</maven-shade-plugin.version>
        <exec-maven-plugin.version>1.6.0</exec-maven-plugin.version>
        <java.version>1.8</java.version>
        <scala.binary.version>2.11</scala.binary.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.1.1</version>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/lib</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <skip>false</skip>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.7.3</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.binary.version}.8</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.binary.version}</artifactId>
            <version>2.3.2</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_${scala.binary.version}</artifactId>
            <version>2.3.2</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>com.github.scopt</groupId>
            <artifactId>scopt_${scala.binary.version}</artifactId>
            <version>3.3.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-library</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>27.0.1-jre</version>
        </dependency>
    </dependencies>

</project>
ZhiyuanCao98 commented 5 years ago

@q463746583 , you can try this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.navercorp</groupId>
    <artifactId>node2vec</artifactId>
    <packaging>jar</packaging>
    <version>0.1.2-SNAPSHOT</version>

    <name>node2vec_spark</name>
    <url>https://github.com/superPershing</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <shadedClassifier>bin</shadedClassifier>
        <maven-shade-plugin.version>3.2.1</maven-shade-plugin.version>
        <exec-maven-plugin.version>1.6.0</exec-maven-plugin.version>
        <java.version>1.8</java.version>
        <scala.binary.version>2.11</scala.binary.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.1.1</version>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/lib</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <skip>false</skip>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.7.3</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.binary.version}.8</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.binary.version}</artifactId>
            <version>2.3.2</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_${scala.binary.version}</artifactId>
            <version>2.3.2</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>com.github.scopt</groupId>
            <artifactId>scopt_${scala.binary.version}</artifactId>
            <version>3.3.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-library</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>27.0.1-jre</version>
        </dependency>
    </dependencies>

</project>

Thank you! But I face an another problem,

19/07/08 16:33:59 DEBUG storage.DiskBlockManager: Adding shutdown hook
19/07/08 16:33:59 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
19/07/08 16:33:59 DEBUG scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: init
19/07/08 16:33:59 DEBUG spark.SecurityManager: Created SSL options for ui: SSLOptions{enabled=false, port=None, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
19/07/08 16:34:00 DEBUG util.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.spark_project.jetty.util.log) via org.spark_project.jetty.util.log.Slf4jLog
19/07/08 16:34:00 INFO util.log: Logging initialized @6961ms
19/07/08 16:34:00 DEBUG util.Jetty: 
java.lang.NumberFormatException: For input string: "unknown"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Long.parseLong(Long.java:589)
    at java.lang.Long.valueOf(Long.java:803)
    at org.spark_project.jetty.util.Jetty.formatTimestamp(Jetty.java:89)
    at org.spark_project.jetty.util.Jetty.<clinit>(Jetty.java:61)
    at org.spark_project.jetty.server.Server.getVersion(Server.java:159)
    at org.spark_project.jetty.server.handler.ContextHandler.<clinit>(ContextHandler.java:128)
    at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:143)
    at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:130)
    at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:83)
    at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:65)
    at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:65)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:65)
    at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:62)
    at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:80)
    at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:175)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:444)
    at com.navercorp.Main$$anonfun$main$1.apply(Main.scala:95)
    at com.navercorp.Main$$anonfun$main$1.apply(Main.scala:93)
    at scala.Option.map(Option.scala:146)
    at com.navercorp.Main$.main(Main.scala:93)
    at com.navercorp.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
19/07/08 16:34:00 DEBUG util.DecoratedObjectFactory: Adding Decorator: org.spark_project.jetty.util.DeprecationWarning@16c587de
19/07/08 16:34:00 DEBUG component.ContainerLifeCycle: o.s.j.s.ServletContextHandler@18460128{/,null,null} added {org.spark_project.jetty.servlet.ServletHandler@74d3b638,MANAGED}

Do you have any idea? Thanks a lot

DozenCoder commented 5 years ago

@q463746583 It seems that error happens while parsing command parameters(in Main.scala:93). Maybe there are some wrongs in your command input.

shboy commented 3 years ago

@q463746583 , you can try this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.navercorp</groupId>
    <artifactId>node2vec</artifactId>
    <packaging>jar</packaging>
    <version>0.1.2-SNAPSHOT</version>

    <name>node2vec_spark</name>
    <url>https://github.com/superPershing</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <shadedClassifier>bin</shadedClassifier>
        <maven-shade-plugin.version>3.2.1</maven-shade-plugin.version>
        <exec-maven-plugin.version>1.6.0</exec-maven-plugin.version>
        <java.version>1.8</java.version>
        <scala.binary.version>2.11</scala.binary.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.1.1</version>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/lib</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <skip>false</skip>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.7.3</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.binary.version}.8</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.binary.version}</artifactId>
            <version>2.3.2</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_${scala.binary.version}</artifactId>
            <version>2.3.2</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>com.github.scopt</groupId>
            <artifactId>scopt_${scala.binary.version}</artifactId>
            <version>3.3.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-library</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>27.0.1-jre</version>
        </dependency>
    </dependencies>

</project>

remember CHANGE the package version to 0.1.2-SNAPSHOT when submit the task ==|