Notice for code smell fix session on Dec 2nd

[hy00nc]

On Dec 2nd, we'll do a hands-on coding session to fix some real-world code smell issues in Apache Nemo.

Please prepare the prerequisites below before the session, so that we can proceed with the session without delay:

1. Clone Apache Nemo, and complete the tasks specified in "Nemo prerequisites and setup" to install Apache Nemo on your laptop (You may skip the "Yarn settings" part).

2. Download and install IntelliJ. With IntelliJ, import the Apache Nemo project cloned in your laptop as a Maven project. Details can be found here.

During the session, we'll edit some Apache Nemo code using Intellij, and create pull requests to the Apache Nemo GitHub repository.

See you in class :)

[bgchun]

Make sure you clone the nemo repo before the Monday class and run "mvn clean install" successfully. If you have questions, please let us know.

[bgchun]

If everything goes well, you will see the following at the end. On my macbook pro, "mvn clean install" took 12 minutes 40 seconds.

[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Nemo Project 0.2-SNAPSHOT: [INFO] [INFO] Nemo Project ....................................... SUCCESS [ 9.406 s] [INFO] Nemo Job Configuration ............................. SUCCESS [ 2.677 s] [INFO] Nemo Common ........................................ SUCCESS [ 7.187 s] [INFO] Nemo Compiler ...................................... SUCCESS [ 0.183 s] [INFO] Nemo Compiler Frontend: Beam ....................... SUCCESS [ 1.617 s] [INFO] Nemo Runtime ....................................... SUCCESS [ 0.213 s] [INFO] Nemo Runtime Common ................................ SUCCESS [ 10.392 s] [INFO] Nemo Compiler Optimizer ............................ SUCCESS [ 5.551 s] [INFO] Nemo Compiler Backend .............................. SUCCESS [ 1.632 s] [INFO] Nemo Runtime Test .................................. SUCCESS [ 0.578 s] [INFO] Nemo Runtime Master ................................ SUCCESS [ 22.374 s] [INFO] Nemo Runtime Executor .............................. SUCCESS [ 30.892 s] [INFO] Nemo Driver ........................................ SUCCESS [ 0.890 s] [INFO] Nemo Client ........................................ SUCCESS [ 27.950 s] [INFO] Nemo Compiler Frontend: Spark ...................... SUCCESS [ 14.415 s] [INFO] Nemo Examples ...................................... SUCCESS [ 1.528 s] [INFO] Nemo Examples: Beam ................................ SUCCESS [08:59 min] [INFO] Nemo Compiler Test ................................. SUCCESS [ 41.066 s] [INFO] Nemo Examples: Spark ............................... SUCCESS [ 28.526 s] [INFO] Nemo Examples: NEXMark ............................. SUCCESS [ 13.375 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 12:40 min [INFO] Finished at: 2019-11-30T14:12:36+09:00 [INFO] ------------------------------------------------------------------------

[lightb0x]

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Nemo Project 0.2-SNAPSHOT:
[INFO] 
[INFO] Nemo Project ....................................... SUCCESS [  4.398 s]
[INFO] Nemo Job Configuration ............................. SUCCESS [  3.000 s]
[INFO] Nemo Common ........................................ FAILURE [  5.840 s]
[INFO] Nemo Compiler ...................................... SKIPPED
[INFO] Nemo Compiler Frontend: Beam ....................... SKIPPED
[INFO] Nemo Runtime ....................................... SKIPPED
[INFO] Nemo Runtime Common ................................ SKIPPED
[INFO] Nemo Compiler Optimizer ............................ SKIPPED
[INFO] Nemo Compiler Backend .............................. SKIPPED
[INFO] Nemo Runtime Test .................................. SKIPPED
[INFO] Nemo Runtime Master ................................ SKIPPED
[INFO] Nemo Runtime Executor .............................. SKIPPED
[INFO] Nemo Driver ........................................ SKIPPED
[INFO] Nemo Client ........................................ SKIPPED
[INFO] Nemo Compiler Frontend: Spark ...................... SKIPPED
[INFO] Nemo Examples ...................................... SKIPPED
[INFO] Nemo Examples: Beam ................................ SKIPPED
[INFO] Nemo Compiler Test ................................. SKIPPED
[INFO] Nemo Examples: Spark ............................... SKIPPED
[INFO] Nemo Examples: NEXMark ............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  13.738 s
[INFO] Finished at: 2019-12-02T02:07:14+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project nemo-common: Compilation failure
[ERROR] /Users/lightb0x/Documents/workspace/swpp_codesmell/incubator-nemo/common/src/main/java/org/apache/nemo/common/ir/IRDAG.java:[221,84] unreported exception X; must be caught or declared to be thrown
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :nemo-common

위와 같은 에러가 발생합니다.

[wonook]

@lightb0x Can you show around 50 lines of the log above the message that you've pasted? Also, provide us with the information regarding the maven version ($ mvn -version), whether it's Oracle Java or OpenJDK($ java -version), and your execution environment (Ubuntu, mac, etc), and it would be easier for us to diagnose. Also, check with $ git status whether the code matches the current status in the master branch.

[lightb0x]

[WARNING] Unable to locate Source XRef to link to - DISABLED
[INFO] 
[INFO] --- build-helper-maven-plugin:3.0.0:add-source (add-source) @ nemo-common ---
[INFO] Source directory: /Users/lightb0x/Documents/workspace/swpp_codesmell/incubator-nemo/common/target/generated-sources added.
[INFO] 
[INFO] --- protobuf-maven-plugin:0.5.0:compile (default) @ nemo-common ---
[INFO] /Users/lightb0x/Documents/workspace/swpp_codesmell/incubator-nemo/common/src/main/proto does not exist. Review the configuration or consider disabling the plugin.
[INFO] 
[INFO] --- protobuf-maven-plugin:0.5.0:compile-custom (default) @ nemo-common ---
[INFO] /Users/lightb0x/Documents/workspace/swpp_codesmell/incubator-nemo/common/src/main/proto does not exist. Review the configuration or consider disabling the plugin.
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ nemo-common ---
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ nemo-common ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ nemo-common ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 124 source files to /Users/lightb0x/Documents/workspace/swpp_codesmell/incubator-nemo/common/target/classes
[INFO] /Users/lightb0x/Documents/workspace/swpp_codesmell/incubator-nemo/common/src/main/java/org/apache/nemo/common/ir/executionproperty/ExecutionPropertyMap.java: Some input files use unchecked or unsafe operations.
[INFO] /Users/lightb0x/Documents/workspace/swpp_codesmell/incubator-nemo/common/src/main/java/org/apache/nemo/common/ir/executionproperty/ExecutionPropertyMap.java: Recompile with -Xlint:unchecked for details.
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /Users/lightb0x/Documents/workspace/swpp_codesmell/incubator-nemo/common/src/main/java/org/apache/nemo/common/ir/IRDAG.java:[221,84] unreported exception X; must be caught or declared to be thrown
[INFO] 1 error
[INFO] -------------------------------------------------------------

% java -version
java version "1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)

% mvn -version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3/libexec
Java version: 1.8.0_72, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk1.8.0_72.jdk/Contents/Home/jre
Default locale: ko_KR, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.1", arch: "x86_64", family: "mac"

env: macOS 10.15.1

[wonook]

Can you share your output for the $ protoc --version command as well? It seems that the protobuf library is not properly installed. ("proto does not exist")

$ protoc --version
libprotoc 2.5.0

If the output doesn't match the result that I've provided above, then please try again after running these commands (this solution is for mac OS users):

$ brew tap homebrew/versions
$ brew install protobuf@2.5

Solutions for the other OS users are provided at https://github.com/apache/incubator-nemo

[lightb0x]

% protoc --version
libprotoc 2.5.0

it was installed.

As a sidenote,

Now that homebrew/versions has been deprecated, homebrew/core supports multiple versions of formulae with a new naming format.

as of homebrew doc, so I had to install it manually from source.

[haesookim]

[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.957 s - in org.apache.nemo.examples.beam.BroadcastITCase
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] org.apache.nemo.examples.beam.PerPercentileAverageITCase.test(org.apache.nemo.examples.beam.PerPercentileAverageITCase)
[ERROR]   Run 1: PerPercentileAverageITCase.test » TestTimedOut test timed out after 240000 mil...
[ERROR]   Run 2: PerPercentileAverageITCase.tearDown:60 » Runtime Test output mismatch while co...
[ERROR]   Run 3: PerPercentileAverageITCase.test:71 » Runtime java.lang.Exception: Thread Runti...
[INFO]
[INFO]
[ERROR] Tests run: 25, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (default) @ nemo-examples-beam ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Nemo Project 0.2-SNAPSHOT:
[INFO]
[INFO] Nemo Project ....................................... SUCCESS [  3.032 s]
[INFO] Nemo Job Configuration ............................. SUCCESS [  2.327 s]
[INFO] Nemo Common ........................................ SUCCESS [  6.348 s]
[INFO] Nemo Compiler ...................................... SUCCESS [  0.319 s]
[INFO] Nemo Compiler Frontend: Beam ....................... SUCCESS [  1.414 s]
[INFO] Nemo Runtime ....................................... SUCCESS [  0.363 s]
[INFO] Nemo Runtime Common ................................ SUCCESS [  4.760 s]
[INFO] Nemo Compiler Optimizer ............................ SUCCESS [  4.036 s]
[INFO] Nemo Compiler Backend .............................. SUCCESS [  1.683 s]
[INFO] Nemo Runtime Test .................................. SUCCESS [  0.557 s]
[INFO] Nemo Runtime Master ................................ SUCCESS [ 17.539 s]
[INFO] Nemo Runtime Executor .............................. SUCCESS [ 22.766 s]
[INFO] Nemo Driver ........................................ SUCCESS [  0.598 s]
[INFO] Nemo Client ........................................ SUCCESS [ 23.749 s]
[INFO] Nemo Compiler Frontend: Spark ...................... SUCCESS [ 11.410 s]
[INFO] Nemo Examples ...................................... SUCCESS [  0.569 s]
[INFO] Nemo Examples: Beam ................................ FAILURE [10:26 min]
[INFO] Nemo Compiler Test ................................. SKIPPED
[INFO] Nemo Examples: Spark ............................... SUCCESS [ 34.013 s]
[INFO] Nemo Examples: NEXMark ............................. SUCCESS [ 14.812 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11:52 min (Wall Clock)
[INFO] Finished at: 2019-12-02T14:17:44+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M1:verify (default) on project nemo-examples-beam: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/haesookim/Documents/2019/Fall/SWPP/Tutorial/incubator-nemo/examples/beam/target/failsafe-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :nemo-examples-beam

I have received the above error on running mvn clean install -T 2C :(

[sungbin92]

[INFO] Reactor Summary for Nemo Project 0.2-SNAPSHOT: [INFO] [INFO] Nemo Project ....................................... SUCCESS [ 3.754 s] [INFO] Nemo Job Configuration ............................. SUCCESS [ 6.172 s] [INFO] Nemo Common ........................................ SUCCESS [ 14.111 s] [INFO] Nemo Compiler ...................................... SUCCESS [ 1.158 s] [INFO] Nemo Compiler Frontend: Beam ....................... SUCCESS [ 2.301 s] [INFO] Nemo Runtime ....................................... SUCCESS [ 1.112 s] [INFO] Nemo Runtime Common ................................ FAILURE [ 3.388 s] [INFO] Nemo Compiler Optimizer ............................ SKIPPED [INFO] Nemo Compiler Backend .............................. SKIPPED [INFO] Nemo Runtime Test .................................. SKIPPED [INFO] Nemo Runtime Master ................................ SKIPPED [INFO] Nemo Runtime Executor .............................. SKIPPED [INFO] Nemo Driver ........................................ SKIPPED [INFO] Nemo Client ........................................ SKIPPED [INFO] Nemo Compiler Frontend: Spark ...................... SKIPPED [INFO] Nemo Examples ...................................... SUCCESS [ 1.114 s] [INFO] Nemo Examples: Beam ................................ SKIPPED [INFO] Nemo Compiler Test ................................. SKIPPED [INFO] Nemo Examples: Spark ............................... SKIPPED [INFO] Nemo Examples: NEXMark ............................. SUCCESS [ 21.050 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 26.654 s (Wall Clock) [INFO] Finished at: 2019-12-02T14:22:51+09:00 [INFO] ------------------------------------------------------------------------ nemo runtime common에서 에러가 납니다.

[hy00nc]

@haesookim Some tests do not successfully run on some local environments. Could you try mvn -DskipTests install?

[hy00nc]

@lightb0x We assume something went wrong with protobuf, yet we are not 100% sure. We will try to figure it out.

@sungbin92 Could you provide us with an error message and the versions of the prerequisites?

[hy00nc]

@lightb0x Does this /Users/lightb0x/Documents/workspace/swpp_codesmell/incubator-nemo/common/src/main/proto exist in your local repository? Is it not being recognized?

[lightb0x]

There is no proto in incubator-nemo/common/src/main. Only java and resources.

[bgchun]

It looks like some students cannot compile nemo. We will also do a quick tutorial on compiling nemo in the beginning of the lecture.

[haesookim]

@hy00nc mvn -DskipTests install worked for me! thank you :)

[ulgal]

@sungbin92 님은 JAVA 환경설정이 되어있지 않아서 PATH를 추가해줬더니 설치됐습니다. 혹시 linux, openjdk 사용하시고 같은 에러가 나면 JAVA 환경설정 확인해보세요