Nopol crashes on Closure-51

[jose]

Hi,

I have just tried to run Nopol on Closure-51:

$ docker run -it --rm -v /tmp:/results tdurieux/repairthemall Nopol -b Defects4J -i Closure_51

and instead of a successful run (as reported in here), I got the following error:

...
18:14:34.750 [main] INFO  fr.inria.lille.repair.nopol.NoPol - Java version: 1.8.0_111
18:14:34.750 [main] INFO  fr.inria.lille.repair.nopol.NoPol - JAVA_HOME: /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/
18:14:34.751 [main] INFO  fr.inria.lille.repair.nopol.NoPol - PATH: /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/:/defects4j/framework/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
java.util.concurrent.ExecutionException: java.lang.NullPointerException
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:206)
    at fr.inria.lille.repair.Main.main(Main.java:106)
Caused by: java.lang.NullPointerException
    at com.gzoltar.core.GZoltar.run(GZoltar.java:51)
    at fr.inria.lille.localization.GZoltarFaultLocalizer.run(GZoltarFaultLocalizer.java:163)
    at fr.inria.lille.localization.GZoltarFaultLocalizer.<init>(GZoltarFaultLocalizer.java:94)
    at fr.inria.lille.localization.GZoltarFaultLocalizer.<init>(GZoltarFaultLocalizer.java:68)
    at fr.inria.lille.localization.GZoltarFaultLocalizer.createInstance(GZoltarFaultLocalizer.java:60)
    at fr.inria.lille.repair.nopol.NoPol.createLocalizer(NoPol.java:179)
    at fr.inria.lille.repair.nopol.NoPol.build(NoPol.java:131)
    at fr.inria.lille.repair.Main$1.call(Main.java:101)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Usage: java -jar nopol.jar
                          [(-m|--mode) <repair|ranking>] (-e|--type) <condition|precondition|pre_then_cond|loop|arithmetic> [(-o|--oracle) <angelic|symbolic>] [(-y|--synthesis) <smt|dynamoth>] [(-l|--solver) <z3|cvc4>] [(-p|--solver-path) <solverPath>] (-s|--source) source1:source2:...:sourceN  (-c|--classpath) <classpath> [(-t|--test) test1:test2:...:testN ] [--complianceLevel <complianceLevel>] [--maxTime <maxTime>] [--maxTimeType <maxTimeType>] [(-z|--flocal) < cocospoon|dumb|gzoltar>] [--output <outputFolder>] [--json[:<outputJson>]]
...

Please find the repair.log file in here.

-- Best, Jose

[tdurieux]

This error is produced when no failing test-case is found.

I will look at why you don't obtain the same result like the one we obtained during our evaluation. Note that we did not use the docker image for our experiment, we did use the source file directly. The environment is slightly different and consequently can produce different behaviors.

[jose]

Hi @tdurieux,

This error is produced when no failing test-case is found.

To discard the possibility of this being a Java-7 vs Java-8 issue, I have just ran the buggy version of Closure-51 on Java-8 and got 3 failing tests (1 that reveals the fault, i.e., the expected trigger test and 2 others that fail due to the Java version):

com.google.javascript.jscomp.CodePrinterTest::testIssue582  <--- trigger test
com.google.javascript.jscomp.CrossModuleMethodMotionTest::testTwoMethods
com.google.javascript.jscomp.CrossModuleMethodMotionTest::testClosureVariableReads3

I will look at why you don't obtain the same result like the one we obtained during our evaluation.

Thank you.

Note that we did not use the docker image for our experiment, we did use the source file directly. The environment is slightly different and consequently can produce different behaviors.

Did you by any chance use a different Nopol version in your experiments?

Note that we did not use the docker image for our experiment, we did use the source file directly.

But, internally, the docker image runs the same source file right? 🙂

-- Best, Jose

[tdurieux]

The only difference between the docker image and directly from the source is the environment.

It is gzoltar that is producing the exception and the first step of the repair, and we did not change gzoltar for 3 years now. By experience, the classpath has a huge influence on the test results (I dont know why) and I observe that the order of the classpath is not the same :/

[jose]

I observe that the order of the classpath is not the same

Indeed.

RepairThemAll_experiment (log file)

build/classes/
build/test/
/tmp/Nopol_Defects4J_Closure_51/build/classes
/tmp/Nopol_Defects4J_Closure_51/build/test
/tmp/Nopol_Defects4J_Closure_51/build/lib/rhino.jar
/tmp/Nopol_Defects4J_Closure_51/lib/ant.jar
/tmp/Nopol_Defects4J_Closure_51/lib/ant-launcher.jar
/tmp/Nopol_Defects4J_Closure_51/lib/args4j.jar
/tmp/Nopol_Defects4J_Closure_51/lib/guava.jar
/tmp/Nopol_Defects4J_Closure_51/lib/jarjar.jar
/tmp/Nopol_Defects4J_Closure_51/lib/json.jar
/tmp/Nopol_Defects4J_Closure_51/lib/jsr305.jar
/tmp/Nopol_Defects4J_Closure_51/lib/junit.jar
/tmp/Nopol_Defects4J_Closure_51/lib/caja-r4314.jar
/tmp/Nopol_Defects4J_Closure_51/lib/protobuf-java.jar
/home/tdurieux/defects4j4repair/script/../repair_tools/nopol.jar

RepairThemAll (docker image)

build/classes/
build/test/
/tmp/Nopol_Defects4J_Closure_51/build/classes
/tmp/Nopol_Defects4J_Closure_51/build/test
/tmp/Nopol_Defects4J_Closure_51/build/lib/rhino.jar
/tmp/Nopol_Defects4J_Closure_51/lib/guava.jar
/tmp/Nopol_Defects4J_Closure_51/lib/caja-r4314.jar
/tmp/Nopol_Defects4J_Closure_51/lib/args4j.jar
/tmp/Nopol_Defects4J_Closure_51/lib/jsr305.jar
/tmp/Nopol_Defects4J_Closure_51/lib/jarjar.jar
/tmp/Nopol_Defects4J_Closure_51/lib/ant.jar
/tmp/Nopol_Defects4J_Closure_51/lib/ant-launcher.jar
/tmp/Nopol_Defects4J_Closure_51/lib/protobuf-java.jar
/tmp/Nopol_Defects4J_Closure_51/lib/json.jar
/tmp/Nopol_Defects4J_Closure_51/lib/junit.jar
/script/../repair_tools/nopol.jar

By experience, the classpath has a huge influence on the test results (I dont know why)

The order of the classpath can indeed influence the execution of a Java program. The only problem I can think of is, a classpath with multiple versions of the same code. For instance, suppose a project that requires v4.0 of a library X to run but it compiles just fine with v3.0 of the same library. If you define the classpath as, e.g., target/classes:X-v3.0:target/test-classes:X-v4.0, the virtual machine tries to first load classes from X-v3.0 (which may break the execution of the program) before looking at X-v4.0. I have just listed all classes on the classpath and there are a total of 7249 classes, no repetitions. So, the issue might be due to other reason than the classpath or the order of entries 🤷🏻‍♂️

-- Best, Jose

[tdurieux]

Yes :/

More I work with APR more I found that the execution of a java program can be completely random :/

[jose]

How about I try to run the repair.py script in the docker image with a hardcoded classpath, i.e., same classpath as in here?

After running the following commands:

docker run -it --entrypoint /bin/bash tdurieux/repairthemall
cd script

which file(s) would I need to modify, and how, to force the repair.py script to use a hardcoded classpath for this particular experiment (Nopol on Closure-51)?

[jose]

I think I need to edit the classpath function in the script/core/benchmarks/Defects4J.py file, right?

But, there is not any text editor installed in the docker image, at least no vi, vim, nano, or even joe, and apt-get install vim does not seem to be working either:

root@1e74921c71ed:/script# apt-get install vim
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libgdbm3
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
  libgpm2 libtinfo6 vim-common vim-runtime xxd
Suggested packages:
  gpm ctags vim-doc vim-scripts
The following NEW packages will be installed:
  libgpm2 libtinfo6 vim vim-common vim-runtime xxd
0 upgraded, 6 newly installed, 0 to remove and 419 not upgraded.
Need to get 7391 kB/7751 kB of archives.
After this operation, 34.3 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Err http://ftp.us.debian.org/debian/ unstable/main xxd amd64 2:8.1.0875-4
  404  Not Found [IP: 64.50.236.52 80]
Err http://ftp.us.debian.org/debian/ unstable/main vim-common all 2:8.1.0875-4
  404  Not Found [IP: 64.50.236.52 80]
Err http://ftp.us.debian.org/debian/ unstable/main vim-runtime all 2:8.1.0875-4
  404  Not Found [IP: 64.50.236.52 80]
Err http://ftp.us.debian.org/debian/ unstable/main vim amd64 2:8.1.0875-4
  404  Not Found [IP: 64.50.236.52 80]
E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/v/vim/xxd_8.1.0875-4_amd64.deb  404  Not Found [IP: 64.50.236.52 80]

E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/v/vim/vim-common_8.1.0875-4_all.deb  404  Not Found [IP: 64.50.236.52 80]

E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/v/vim/vim-runtime_8.1.0875-4_all.deb  404  Not Found [IP: 64.50.236.52 80]

E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/v/vim/vim_8.1.0875-4_amd64.deb  404  Not Found [IP: 64.50.236.52 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Any other suggestion?

[jose]

Ok, with a mix of cat, head, tail, echo, and then python repair.py Nopol -b Defects4J -i Closure_51, I managed to hardcode the classpath and re-run the experiment. Long story short, it did not work. I got the exact same NullPointerException with the classpath you used in your experiments :-/

Please find the repair.log file in here.

[jose]

Yet another attempt.

The java version available in the docker image is 1.8.0_111 and in your experiments it seems you used 1.8.0_181. I downloaded jre-8u181-linux-x64 from here, updated the JAVA8_HOME in the config.py file, and re-ran the experiment. Got the exact same NullPointerException. Please find the repair.log file in here.

I am running out of ideas that could explain this issue :-/

[jose]

Here is another thing that is bugging me.

According to the script/core/benchmarks/Defects4J.py file, the path to the D4J benchmark is:

        self.path = os.path.join(REPAIR_ROOT, "benchmarks", "defects4j")

by other words, it is /script/../benchmarks/defects4j (which of course exists and the hash of the last commit is fcf6b9b Jan/2019, same one as defined in here). But, if that path is correct, how am I getting /defects4j/framework/bin in the repair.log file?

...
PATH: /script/down/jdk1.8.0_181/bin/:/defects4j/framework/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
...

/defects4j/ does exist and the hash of the last commit is 8eb59db Nov/2018. So, could this issue be due to a much older version of D4J?

$ cd /
$ mv defects4j defects4j.backup
$ ln -s /benchmarks/defects4j defects4j

$ cd /script/
$ python repair.py Nopol -b Defects4J -i Closure_51

Same NullPointerException.

[DehengYang]

Dear @tdurieux and @jose,

Sorry for disturbing you here. I am also quite interested in this issue. But what I am currently concerning is that:

To discard the possibility of this being a Java-7 vs Java-8 issue, I have just ran the buggy version of Closure-51 on Java-8 and got 3 failing tests (1 that reveals the fault, i.e., the expected trigger test and 2 others that fail due to the Java version)

The java version available in the docker image is 1.8.0_111 and in your experiments it seems you used 1.8.0_181. I downloaded jre-8u181-linux-x64 from here, updated the JAVA8_HOME in the config.py file, and re-ran the experiment. Got the exact same NullPointerException.

Based on the descriptions I get to know that the Nopol used in this experiment is built under jdk 1.8. And I would like to ask how do you deal with unexpected failing tests of Defects4J bugs(e.g., the abovementioned closure-51) when running repair experiment under jdk 1.8 version?

It would be much appreciated if any help is given. Thank you so much for you time and consideration.

Best, dale

[jose]

Hi @tdurieux,

Just to let you know I've managed to successfully run Nopol on Closure-51 directly from RepairThemAll's source code (rather than using the docker image), so I am closing this issue.

Nevertheless, you might want to recreate the docker image with the latest version of the source code or perhaps remove the support for docker, as it does not seem to produce the same data you currently have in the RepairThemAll_experiment repository.

-- Best, Jose

[tdurieux]

The goal of the Dockerimage is not really to reproduce the exact results that we obtained (it is probably impossible), but more to be able to experiment with APR or try easily new version of a tool.

I would really like to be able to have a 100% stable result and reproducible but I honestly don't think it is possible with the current APR techniques.