search

USCDataScience / sparkler

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
http://irds.usc.edu/sparkler/
Apache License 2.0
411 stars 143 forks source link

Build Failure on JDK 9 #134

Closed thammegowda closed 3 years ago

thammegowda commented 6 years ago

Issue Description

Tests doesnt pass on JDK9.

Cause: Some deprecated APIs are dropped. Tests starts a server and stops it. Test Web server works fine on JDK8. JDK9 prints a WARNING about deprecated API, and by default it doesnt activate the test web server

How to reproduce it

Install JDK 9 export JAVA_HOME to JDK 9 then do mvn --version to see that maven is actually using it then do mvn test

Environment and Version Information

mvn --version
Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T12:39:06-07:00)
Maven home: /usr/local/Cellar/maven/3.5.0/libexec
Java version: 9.0.1, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.13", arch: "x86_64", family: "mac"
✔ ~/work/irds/sparkler/tmp/sparkler [master|✔]
23:06 $ mvn test
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for edu.usc.irds.sparkler:sparkler-api:bundle:0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ edu.usc.irds.sparkler:sparkler-api:[unknown-version], /Users/tg/work/irds/sparkler/tmp/sparkler/sparkler-api/pom.xml, line 67, column 21
[WARNING] The expression ${pom.groupId} is deprecated. Please use ${project.groupId} instead.
[WARNING] The expression ${pom.artifactId} is deprecated. Please use ${project.artifactId} instead.
[WARNING] The expression ${parent.version} is deprecated. Please use ${project.parent.version} instead.
[WARNING]
[WARNING] Some problems were encountered while building the effective model for edu.usc.irds.sparkler:sparkler-app:jar:0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ edu.usc.irds.sparkler:sparkler-parent:0.1-SNAPSHOT, /Users/tg/work/irds/sparkler/tmp/sparkler/pom.xml, line 78, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for edu.usc.irds.sparkler.plugin:urlfilter-regex:bundle:0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ edu.usc.irds.sparkler:sparkler-parent:0.1-SNAPSHOT, /Users/tg/work/irds/sparkler/tmp/sparkler/pom.xml, line 78, column 21
[WARNING] The expression ${artifactId} is deprecated. Please use ${project.artifactId} instead.
[WARNING]
[WARNING] Some problems were encountered while building the effective model for edu.usc.irds.sparkler.plugin:fetcher-jbrowser:bundle:0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ edu.usc.irds.sparkler:sparkler-parent:0.1-SNAPSHOT, /Users/tg/work/irds/sparkler/tmp/sparkler/pom.xml, line 78, column 21
[WARNING] The expression ${groupId} is deprecated. Please use ${project.groupId} instead.
[WARNING] The expression ${artifactId} is deprecated. Please use ${project.artifactId} instead.
[WARNING]
[WARNING] Some problems were encountered while building the effective model for edu.usc.irds.sparkler.plugin:fetcher-htmlunit:bundle:0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ edu.usc.irds.sparkler:sparkler-parent:0.1-SNAPSHOT, /Users/tg/work/irds/sparkler/tmp/sparkler/pom.xml, line 78, column 21
[WARNING] The expression ${groupId} is deprecated. Please use ${project.groupId} instead.
[WARNING] The expression ${artifactId} is deprecated. Please use ${project.artifactId} instead.
[WARNING]
[WARNING] Some problems were encountered while building the effective model for edu.usc.irds.sparkler.plugin:sparkler-plugins:pom:0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ edu.usc.irds.sparkler:sparkler-parent:0.1-SNAPSHOT, /Users/tg/work/irds/sparkler/tmp/sparkler/pom.xml, line 78, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for edu.usc.irds.sparkler:sparkler-tests-base:jar:0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ edu.usc.irds.sparkler:sparkler-parent:0.1-SNAPSHOT, /Users/tg/work/irds/sparkler/tmp/sparkler/pom.xml, line 78, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for edu.usc.irds.sparkler:sparkler-ui:war:0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ edu.usc.irds.sparkler:sparkler-parent:0.1-SNAPSHOT, /Users/tg/work/irds/sparkler/tmp/sparkler/pom.xml, line 78, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for edu.usc.irds.sparkler:sparkler-parent:pom:0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ line 78, column 21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] sparkler-parent
[INFO] sparkler-tests-base
[INFO] sparkler-api
[INFO] sparkler
[INFO] sparkler-plugins
[INFO] urlfilter-regex
[INFO] fetcher-jbrowser
[INFO] fetcher-htmlunit
[INFO] sparkler-ui
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building sparkler-parent 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building sparkler-tests-base 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ sparkler-tests-base ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ sparkler-tests-base ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ sparkler-tests-base ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/tg/work/irds/sparkler/tmp/sparkler/sparkler-tests-base/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ sparkler-tests-base ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ sparkler-tests-base ---
[INFO] No tests to run.
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building sparkler-api 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:copy-resources (copy-resources) @ sparkler-api ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 20 resources
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ sparkler-api ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 20 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.6.1:compile (default-compile) @ sparkler-api ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ sparkler-api ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.6.1:testCompile (default-testCompile) @ sparkler-api ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ sparkler-api ---

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running edu.usc.irds.sparkler.util.FetcherDefaultTest
2017-10-27 23:07:06 INFO  FetcherDefault:107 [main] - DEFAULT FETCHER http://localhost:8080/res/index.html
2017-10-27 23:07:06 WARN  FetcherDefault:151 [main] - FETCH-ERROR http://localhost:8080/res/index.html
2017-10-27 23:07:06 INFO  FetcherDefault:107 [main] - DEFAULT FETCHER http://localhost:8080/res/jspage.html
2017-10-27 23:07:07 WARN  FetcherDefault:151 [main] - FETCH-ERROR http://localhost:8080/res/jspage.html
2017-10-27 23:07:07 INFO  FetcherDefault:107 [main] - DEFAULT FETCHER http://localhost:8080/res/vacduyc_NOT_FOUND.html
2017-10-27 23:07:07 WARN  FetcherDefault:151 [main] - FETCH-ERROR http://localhost:8080/res/vacduyc_NOT_FOUND.html
2017-10-27 23:07:07 INFO  FetcherDefault:107 [main] - DEFAULT FETCHER http://localhost:8080/res/index.html
2017-10-27 23:07:07 INFO  FetcherDefault:107 [main] - DEFAULT FETCHER http://localhost:8080/slavesite?action=return-headers
2017-10-27 23:07:07 INFO  FetcherDefault:107 [main] - DEFAULT FETCHER http://localhost:8080/slavesite?action=read-timeout&timeout=11000
Tests run: 5, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 1.019 sec <<< FAILURE! - in edu.usc.irds.sparkler.util.FetcherDefaultTest
fetch1(edu.usc.irds.sparkler.util.FetcherDefaultTest)  Time elapsed: 0.384 sec  <<< FAILURE!
junit.framework.AssertionFailedError: expected:<200> but was:<400>
    at edu.usc.irds.sparkler.util.FetcherDefaultTest.fetch1(FetcherDefaultTest.java:90)

fetch(edu.usc.irds.sparkler.util.FetcherDefaultTest)  Time elapsed: 0.105 sec  <<< ERROR!
java.net.ConnectException: Connection refused (Connection refused)
    at edu.usc.irds.sparkler.util.FetcherDefaultTest.fetch(FetcherDefaultTest.java:64)

testHeaders(edu.usc.irds.sparkler.util.FetcherDefaultTest)  Time elapsed: 0.102 sec  <<< ERROR!
java.net.ConnectException: Connection refused (Connection refused)
    at edu.usc.irds.sparkler.util.FetcherDefaultTest.testHeaders(FetcherDefaultTest.java:121)

testReadTimeout(edu.usc.irds.sparkler.util.FetcherDefaultTest)  Time elapsed: 0.102 sec  <<< FAILURE!
junit.framework.AssertionFailedError: Socket Exception expected, but found:java.net.ConnectException
    at edu.usc.irds.sparkler.util.FetcherDefaultTest.testReadTimeout(FetcherDefaultTest.java:108)

Results :

Failed tests:
  FetcherDefaultTest.fetch1:90 expected:<200> but was:<400>
  FetcherDefaultTest.testReadTimeout:108 Socket Exception expected, but found:java.net.ConnectException
Tests in error:
  FetcherDefaultTest.fetch:64 » Connect Connection refused (Connection refused)
  FetcherDefaultTest.testHeaders:121 » Connect Connection refused (Connection re...

Tests run: 5, Failures: 2, Errors: 2, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] sparkler-parent .................................... SUCCESS [  0.030 s]
[INFO] sparkler-tests-base ................................ SUCCESS [  0.547 s]
[INFO] sparkler-api ....................................... FAILURE [  2.960 s]
[INFO] sparkler ........................................... SKIPPED
[INFO] sparkler-plugins ................................... SKIPPED
[INFO] urlfilter-regex .................................... SKIPPED
[INFO] fetcher-jbrowser ................................... SKIPPED
[INFO] fetcher-htmlunit ................................... SKIPPED
[INFO] sparkler-ui ........................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.348 s
[INFO] Finished at: 2017-10-27T23:07:07-07:00
[INFO] Final Memory: 19M/64M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on project sparkler-api: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/tg/work/irds/sparkler/tmp/sparkler/sparkler-api/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :sparkler-api
amar500 commented 4 years ago

Is there a fix for it. I cloned the git repo and tried building with JDK 13 and get the same error.