reif_libcg

[ctreude]

@mreif

I pulled and ran the container and tried to reproduce the evaluation using sbt run with option 2, but was given the following error message:

[info] Running org.opalj.evaluation.EvaluationStarter
[info]
[info]
[info] PROCESSING: 0 with org.scala-lang | scala-compiler | 2.10.4
[info]
[info]
[info] CPFILE: /home/libcg/evalProjects/org.scala-lang/scala-compiler/jars/scala
-compiler-2.10.4.jar
[info] LibFiles: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib
[info] /home/libcg/evalProjects/org.scala-lang/scala-library/jars/scala-library-
2.10.4.jar
[info] /home/libcg/evalProjects/org.scala-lang/scala-reflect/jars/scala-reflect-
2.10.4.jar
java.lanat scala.sys.package$.error(package.scala:27)d from runner: 137
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code returned from runner: 137
[error] Total time: 201 s, completed Jul 22, 2016 3:55:14 AM

Any idea what might be the cause? I'm on Windows 8.1.

[hongyujohn]

Brief Summary This tool could run as described. It contains two program: Call Edge Count Analysis and Evaluation Starter.

Insightful This tool can construct call graph for Java libraries and give some analysis results. The effect of different assumptions on the build call graph can then be studied. The list of dead methods could also provide insights into quality issues of long living software projects.

Useful The list of dead-methods could be useful. Docker container is easy to use.

Usable The source code of OPAL is available on Bitbucket. There is an official website of OPAL and libcg. But the document for this tool is not clearly displayed. A Docker container is provided, so it’s easy to start. The generate custom call graph numbers also works well. One thing is that, we need machines with at least 6G memory to run the tool, instead of 4G as described by the authors. There is also no explanation about where the graph produced is stored and what it looks like.

[mreif]

@ctreude

I tried to investigate your error message but I'm not able to reproduce the error. Can you please run it again with "sbt compile:run" to print the full stack trace.

I would additionally like to know the configuration of your docker environment, since the suppressed error could be an OutOfMemoryException. You are required to give your docker process at least 6GB (as hongyujohn pointed out, this is actually a type that needs to be fixed.) of RAM.

best Michael

[ctreude]

@mreif

Yes, looks like it's a memory problem:

[info] Running org.opalj.evaluation.EvaluationStarter
[info]
[info]
[info] PROCESSING: 0 with org.scala-lang | scala-compiler | 2.10.4
[info]
[info]
[info] CPFILE: /home/libcg/evalProjects/org.scala-lang/scala-compiler/jars/scala
-compiler-2.10.4.jar
[info] LibFiles: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib
[info] /home/libcg/evalProjects/org.scala-lang/scala-library/jars/scala-library-
2.10.4.jar
[info] /home/libcg/evalProjects/org.scala-lang/scala-reflect/jars/scala-reflect-
2.10.4.jar
[info] #
[info] # There is insufficient memory for the Java Runtime Environment to contin
ue.
[info] # Native memory allocation (mmap) failed to map 196608 bytes for committi
ng reserved memory.
[info] # An error report file with more information is saved as:
[info] # /home/libcg/.sbt/0.13/staging/847a683ed034ccc78f22/evaluation/hs_err_pi
d332.log
[error] OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006f880
3000, 196608, 0) failed; error='Cannot allocate memory' (errno=12)
java.lang.RuntimeException: Nonzero exit code returned from runner: 1
        at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code returned from runner: 1
[error] Total time: 83 s, completed Jul 25, 2016 2:43:20 AM

I'm running docker run -ti -m 6g mreif/fse2016:evaluation but that doesn't seem to do the trick.

[timm]

Note these labels are still "under discussion" and are still subject to change prior to the final notifications Friday.

[shaiduc]

@mreif @ctreude

Got the same error message as Christoph at first. Then tried to increase the memory, but was not able to run the container so far. Below is what was tried:

First, the Java Virtual Machine was not able to allocate memory. This error was detected after running the docker container using the following command line:

docker run -ti mreif/fse2016:evaluation

Therefore, proceeded to increase the memory limit for the container. This was the new command line to start the container, with 20GB as its memory limit:

docker run -t -i -m 20g mreif/fse2016:evaluation

The attempts using the previous command line threw the following error:

[info] Running org.opalj.evaluation.EvaluationStarter [info] [info] [info] PROCESSING: 0 with org.scala-lang | scala-compiler | 2.10.4 [info] [info] [info] CPFILE: /home/libcg/evalProjects/org.scala-lang/scala-compiler/jars/scala -compiler-2.10.4.jar [info] LibFiles: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib [info] /home/libcg/evalProjects/org.scala-lang/scala-library/jars/scala-library- 2.10.4.jar [info] /home/libcg/evalProjects/org.scala-lang/scala-reflect/jars/scala-reflect- 2.10.4.jar java.lang.RuntimeException: Nonzero exit code returned from runner: 137 at scala.sys.package$.error(package.scala:27) [trace] Stack trace suppressed: run last compile:run for the full output. error Nonzero exit code returned from runner: 137 [error] Total time: 46 s, completed Jul 25, 2016 11:24:43 PM libcg@148213ac3539:~/evaluation$

I am not familiar with the Scala programming language, therefore not sure how to approach this further... I am submitting my preliminary review for now, based on what I could try out and evaluate.

[shaiduc]

[Recommendation]: Maybe Gold

The problem addressed by the artifacts is hard to determine based on the paper and unfortunately a pre-print of the research paper is not available in order to better understand the ideas behind this set of artifacts. As it stands, the paper is rather confusing, as it discusses the insightfulness aspect related to call graphs, the usefulness for the list of dead methods and the usability of the list and the source code.

[Insightful]

• Timely: Section 2 does not illustrate appropriately the problem that this tool/artifact is solving. The space constraint for this submission would be the main reason to not include a complete explanation, but in this case a preprint version of the research paper as part of the submission would be appropriate.
• Makes researchers smarter: The tool seems to support researchers when they want to design static analysis on bytecode. Therefore, one can say that it makes researchers more productive.

[Useful]

• Serves a useful purpose?: Definitely. It helps to conduct static analysis on Java bytecode.
• Serves a purpose that would otherwise be tedious, prolonged, awkward, or impossible? Yes. Since bytecode is a low-level language, making it difficult to understand, tools that facilitate its abstraction are always helpful.
• Cost-effective? It can require quite some time to understand the source code, so it may not be very cost-effective.

[Usable]

• Easy to understand? The existing build offers a script to run custom analysis. Extensions of the tool will involve the comprehension of the source code, which might not be an easy task.
• Accompanied by tutorial notes? I was unable to get the Docker container (see below, in the “Easy to download” section for the errors I got). However, the paper contains enough information to replicate the initial evaluation, and to run additional analysis on custom Java programs.
• Easy to download: The code is in a public repository. However, Docker is needed to obtain the other artifact. Got errors that I could not solve trying to do that, so one of these artifacts is definitely not easy to download. In the readme file, the provided link for the list of dead methods does not work. The “.org” in the link (instead of “.de”) seems to be the culprit.
• Available in a virtual machine image? Yes. It uses Docker.
• Supported by configuration management tools to permit easy updates? Yes. The source code is in a public Git repository, and the virtual machine was created using Docker.

Other comments:

• There is a typo in the title of Section 2: INSIGHTFULNES -> INSIGHTFULNESS
• The README file in GitHub mentions in the DISCLAIMER that “The numbers produced by this version of the algorithms will slightly differ from the ones in the paper. This is due to the usage of a different Java Class Library (in terms of operating system and version) and, in particular, also due to minor bug fixes. The camera ready copy of the paper will contain the updated numbers. Note, that none of the messages stated in the paper are affected at all.” Does this refer to this artifact paper or also the research paper? Do the new numbers impact the results obtained in the main research track paper?

[timm]

@obaysal: 1) looks like @shaiduc agrees with our "gold" label, therefore i would say "gold". do you agree?

2) the access to a pre-print describing the work is an issue that plagued several reviewers. i do not think it necessarily harmed this cycle of reviewing (since other artifact tracks also work on such brief info)

3) that said, for FUTURE artifact tracks, I would propose that authors deposit (a) the artifact 2 pager and (b) an article pre-print.

[obaysal]

@timm agree. Keeping the decision as is.