Moving Above Java 8 - Githubissues

lapplislazuli commented 3 years ago

Hi @martinezmatias @monperrus ,

as found in #243, the bad dependency that blocks using newer Java Versions is Nopol. Nopol really needs the features from com.sun.tools 1.4 and there seems to be no easy fix (See the failing tests in https://github.com/SpoonLabs/nopol/pull/212).

While one can put everything in docker and run it there, IDEs are also not supported in newer java versions because the dependency trees are not building properly (atleast that is happening on my Java 15 with IntelliJ). This means developing mostly "blind" for me.

To make development easier, and maybe allow for more approaches to be added, I would suggest to do the following:

Proposal

Make a clear tag / release to mark nopols deprecation
Drop Nopol as a dependency
Remove Nopols usage in tests and examples
Move to Java 11

Benefits

Support for Java>8 runtimes
Support for Java>8 developers
I expect that it might run faster
Removal of a hidden constraint

Drawbacks

Bit of work
Nopol won't be easily added to future benchmarks & projects
A bit of documentation overhead to point to nopol versions / experiments

I think the primary drawback, that is loosing a working version of nopol in astor, is mostly addressed with the versioning and Benchmark-Repository.

Options

Rewrite Nopol to not use com.sun.jdi (lots of work)
Maybe: (Somehow) add the .jar as a lib into the repo, but I think this might not work (either because sun needs other sun deps from old java, or sun might have a nasty licence).
Maybe: Make a java8 nopol-plugin but keep it separate

Note

I personally want to extend astor for my PhD-Research with @Apanichella, so I am likely to make a fork and do the proposed changes. But obviously we also want to contribute back to such a great tool instead of separating efforts :)

Downgrading to Java8 is the last thing I will do in my life though. I have sworn an oath to all java-devs before me to break the curse of java8.

martinezmatias commented 3 years ago

Hi @Twonki

Thanks for the detailed document.

so I am likely to make a fork and do the proposed changes.

Great. I think that it will be nice to integrate you changes into Astor.

I personally want to extend astor for my PhD-Research with @apanichella,

Nice. Happy to help you if you need some something along the way.

To make development easier, and maybe allow for more approaches to be added, I would suggest to do the following:

Thanks for the detailed plan. We need the Nopol dependency to use the Dynamoth synthesis used by some prototypes we developed.

Maybe: (Somehow) add the .jar as a lib into the repo, but I think this might not work (either because sun needs other sun deps from old java, or sun might have a nasty licence).

We can try: add the tool lib into Astor project and installing using Maven. The problem there is what to do with the Nopol classes that Astor uses.

@monperrus : The approaches that use Dynamoth synthesis are the sketch repair approaches. One solution could be to extract those approaches and to put them in a new project (new repo on github) which will have as dependency the current version of Astor. WDYT?

Regards Matias

monperrus commented 3 years ago

The approaches that use Dynamoth synthesis are the sketch repair approaches. One solution could be to extract those approaches and to put them in a new project (new repo on github) which will have as dependency the current version of Astor. WDYT?

LGTM

lapplislazuli commented 3 years ago

Nice to hear :)

When I remove the dependency I get ~100 Errors in 13 files, whereof 6 are in fr.inria.astor.core.manipulation.synthesis.dynamoth but also 6 are in fr.inria.astor.approaches.tos.core.evaltos.

Dynamoth can be deleted safely, but deleting the tos cascades to 9 errors in places that look more serious, namely

Does this sound reasonable? Can these also be savely left out? That there are issues in LauncherJUnitProcess looks troubling for me 😨

martinezmatias commented 3 years ago

Hi @Twonki

Thanks for the detailed description.

I will do the following steps: a) Create a new project temporally named AstorMoth where we will put the approaches that depend on Dynamoth (i.e., the classes you remark). That project will depend on the current version of Astor. b) Release the current version. c) Create a new version of Astor which a) will not have the dependency to Nopol, B) will not have any of the mentioned classes. d) Release the new version.

lapplislazuli commented 3 years ago

@martinezmatias , do you have an estimate how long you think it will take? Are there things I can support you for the refactoring?

martinezmatias commented 3 years ago

Hi @Twonki I will start asap Regards Matias

martinezmatias commented 3 years ago

Hi @Twonki

I remove the dependency to Nopol and the approaches that use the Dynamoth synthesis. Please, tell me if all seems fine.

Regards Matias

lapplislazuli commented 3 years ago

@martinezmatias wow that was quick! :D

I did some checks:

✔️ mvn compile with OpenJDK15 on Windows works
✔️ mvn compile with OpenJDK15 on MaOS BigSur works
✔️ mvn install -DskipTests=true with OpenJDK15 on Windows in IntelliJ works
✔️ mvn install -DskipTests=true with OpenJDK15 on MaOS BigSur in IntelliJ works
❌ console-run mvn install -DskipTests=true with OpenJDK15 on MaOS BigSur fails with below error
❌ console-run mvn install -DskipTests=true with OpenJDK15 on Windows fails with below error
✔️ sudo console-run mvn install -DskipTests=true with OpenJDK15 on MaOS BigSur works
✔️ admin console-run mvn install -DskipTests=true with OpenJDK15 on Windows fails works
✔️ mvn package -DskipTests=true on non-admin Windows Console
❌ mvn package -DskipTests=true on MacOs Console
✔️ sudo mvn package -DskipTests=true on MacOs Console

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project astor: 
ExecutionException; nested exception is java.util.concurrent.ExecutionException: 
java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. 
VM crash or System.exit called ?
[ERROR] Command wascmd.exe /X /C ""C:\Program Files\Java\jdk-15\bin\java" -
javaagent:C:\\Users\\xxx\\.m2\\repository\\org\\jacoco\\org.jacoco.agent\\0.7.5.201505241946\\org.jacoco.agent-0.7.5.201505241946-runtime.jar=destfile=C:\\Users\\xxx\\Code\\astor\\target\\jacoco.exec 
org.apache.maven.surefire.booter.ForkedBooter C:\Users\xxx\Code\astor\target\surefire\surefire16275582123885752881tmp
C:\Users\xxx\Code\astor\target\surefire\surefire_012126818369709033486tmp"

And the (fatal) error producing the Maven build error [There are many such errors, i just copied one to keep it short]:

FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.lang.RuntimeException: Class java/util/UUID could not be instrumented.
        at org.jacoco.agent.rt.internal_b0d6a23.core.runtime.ModifiedSystemClassRuntime.createFor(ModifiedSystemClassRuntime.java:139)
        at org.jacoco.agent.rt.internal_b0d6a23.core.runtime.ModifiedSystemClassRuntime.createFor(ModifiedSystemClassRuntime.java:100)
        at org.jacoco.agent.rt.internal_b0d6a23.PreMain.createRuntime(PreMain.java:55)
        at org.jacoco.agent.rt.internal_b0d6a23.PreMain.premain(PreMain.java:47)
        ... 6 more
Caused by: java.lang.NoSuchFieldException: $jacocoAccess
        at java.base/java.lang.Class.getField(Class.java:2092)
        at org.jacoco.agent.rt.internal_b0d6a23.core.runtime.ModifiedSystemClassRuntime.createFor(ModifiedSystemClassRuntime.java:137)
        ... 9 more
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at ./open/src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422

The error on mac is the same, just with different paths.

I mean it makes sense that I cannot install stuff as non-admin, but the error message still seems off.

The package error on non-admin macos might be due to mixing sudo and non-sudo usage so maybe the su created a file and then the normal user can't use it etc. I think thats just a minor issue.

lapplislazuli commented 3 years ago

I am currently trying to run on some of the examples, and they fail because in gzoltar-faultlocalization none of the tests fail (or is seen as a failing test).

I picked (random) the Math-Issue-340, where I first had the issue that my maven could not compile java 1.4 and needs atleast 1.7. I changed that in the pom, and then I could compile and test the Math-340 producing the required files and having 2 failing test-cases.

The tests are found when I run over debugging, just none is found failing.

Could this be that gzoltar has constraints on Java post 8 ?

martinezmatias commented 3 years ago

Hi @Twonki The restrictions comes mostly from the dataset of bugs (Defects4J): to reproduce some bugs it's necessary to run with specific Java versions.

lapplislazuli commented 3 years ago

Good Afternoon @martinezmatias ,

I am still trying to figure everything out with the new version 😅

To have it more reproducible, I started a docker image (see attached) and to build I had to add some dependencies explicit (some eclipse mylin, see attached "new" pom). At the moment all the things are hardcoded for the issue time_2, so I compile & test that one first and afterwards run

docker build . -t astor:unstable 
docker run astor:unstable

The first issue is that gzoltar run returns without finding any issues. Gzoltar itself is invoked and seems to be okaish, the last log I get there is

After that it fails as there is no line with suspicious code and it ends the program.

When I manually say accept susp=0, it fails a bit further down with the following error:

This is created in Abstract Main where astor wants to run mvn com.github.tdurieux:project-config-maven-plugin but fails doing so.

However, I checked and after doing mvn dependency:resolve there is a fitting jar in my local repository. I also get the very same error on my mac.

I also checked tdurieux docker but it seems to be only using v0.0.2, same for RepairThemAll.

Oh, on thing I had to adjust was here in ProjectConfigruation, where I had to change

        } catch (Exception e) {
            logger.error("Error retrieving java version: " + e);
        }

to

        } catch (Exception e) {
            logger.warn("Error retrieving java version: " + e + " - Retrying with slightly different path");
            logger.debug("Retrying to get Java-Version with slightly different path");
            try {
                String jvmPathB = jvmPath + File.separator + "bin" + File.separator + "java";

                List<String> command = new ArrayList<String>();
                command.add(jvmPathB);
                command.add("-version");

                ProcessBuilder pb = new ProcessBuilder(command.toArray(new String[command.size()]));
                pb.redirectOutput();
                pb.redirectErrorStream(true);
                pb.directory(new File((ConfigurationProperties.getProperty("location"))));
                Process p = pb.start();
                BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream()));

                String line = null;
                while ((line = br.readLine()) != null) {
                    if (line.contains("\"")) {
                        String version = line.split("\"")[1];
                        logger.info("Version of the JVM used: " + version);
                        return version;
                    }
                    processOutput += line;
                }
            } catch (Exception et){
                logger.error("Error retrieving java version (2nd attempt): " + et);
            }
        }

So I just quick and dirty added a second check on a different path, as my jvm looked a bit different than to what was expected.

astor_dockerpatches.zip

martinezmatias commented 3 years ago

Hi @lapplislazuli

Which is the test that has the problem you mention? If the test is not executed by the CI, we can ignore it.

I imagine that, as the execution of mvn com.github.tdurieux:project-config-maven-pluginfails (is it installed), then Astor then cannot determine the failing test case, thus, it does not pass to the FL the failing test case. For that reason, FL could not find any suspicious.

lapplislazuli commented 3 years ago

Good Morning @martinezmatias,

the issues are not in the CI - they appear when I just want to run it as per getting started option 2. I have the same issues for running on time_2 and for math-issue-340 (but math-340 was compiled on j8 so it has to failures and I picked time_2 for showing here).

The plugin from tdurieux is properly downloaded and in my (and the containers) local repository, but I think it is not a proper plugin. Do you use v.1.0-SNAPSHOT ?

Do you have clues why my Gzoltar finds failing and passing tests but no suspicious code? How can I get more information here?

If you prefer I can make a PR with the current changes

martinezmatias commented 3 years ago

Hi @lapplislazuli

Do you have clues why my Gzoltar finds failing and passing tests but no suspicious code? How can I get more information here?

We have experimented different and several issues with GZoltar along the way. For example, GZoltar usually fails to instrument java11 code. We have started developing a new FL (https://github.com/SpoonLabs/flacoco) and I started to integrated into Astor. (Unfortunately I could not have time to finish it)

If you prefer I can make a PR with the current changes

PR are always welcome, thanks :) Which are the changes that you will propose?

lapplislazuli commented 3 years ago

@martinezmatias

I would open with the current Docker-Parts described above so it's easier to check out .

I have actually no clue where to look at at the moment 😅

What does the plugin from tdurieux do? Does it "just" parse the Pom?

martinezmatias commented 3 years ago

What does the plugin from tdurieux do?

The plugin determines:

The source code folders (e.g. src/main/java..)
The bin folder (e.g., /target/)
The list of dependencies
The version of the source code (e.g. java 8)

Without that plug-in, the Astor's user have to determine that information and pass to Astor via parameters.

lapplislazuli commented 3 years ago

Good Morning @martinezmatias,

I have opened a PR #312 with the issues I have for reproduction, but I think I could narrow down the issues with gzoltar a bit:

The dependency on gzoltar.core 1.7.2 is not used. All usages of gzoltar point to the version 0.1.1 in the Pom, which does not work above Java8.

martinezmatias commented 3 years ago

Hi @lapplislazuli

The dependency on gzoltar.core 1.7.2 is not used. All usages of gzoltar point to the version 0.1.1 in the Pom, which does not work above Java8.

Exactly. We use Gzoltar 0.1.1. We tried to integrate 1.7.2 but the APi are completely incompatible and we could finish the integration of 1.7.2.

Regards Matias

lapplislazuli commented 3 years ago

Good Afternoon @martinezmatias,

I am slowly making progress with Gzoltar after asking the Maintainer. The suggested use of Gzoltar according to Jose is to use the CLI when possible, mostly because the elements to be analyzed need to be in the classpath, which might chain to bigger problems. Another Issue is that GZoltar also didn't work above Java 8, but that has been addressed with 1.7.3-SNAPSHOT.

What I can currently do Java-wise is

List the Tests of a method Instrument the .class files Create the gzoltar.ser, which has all information required from the spectrum Create a report using the CLI Creating the report could also be done using the library in the java code, but is slightly discouraged. Also, it fails at the moment above Java 8, but I think the issue is similar to what happened in the CLI. The question is whether the files created on the CLI are suited / nicely be integrated with the interfaces.

These are the files created:

The Project is a toy example from my Professor, I just took it as it compiles without legacy java. Similarly to the ochiai-ranking one can create a tarantula ranking as well.

Do you think these artefacts can be used and re-merged to the existing project parts? Where would I start looking for that ?

martinezmatias commented 3 years ago

Hi @lapplislazuli

We have already one implementation that interacts with GZ 1.x, which executes some of the steps you mentioned (e.g., parsing spectra.csv and tests.csv). However, beyond having that initial code, we have not fully testing on it, and we havent use it for the experiments. In any case, we could start take a look to upgrade it. The class is GZoltarClientMasterFaultLocalization.java

WDYT? Regards Matias

lapplislazuli commented 3 years ago

Good morning @martinezmatias , I have updated the PR #312, could you have a look? I wrote the open points in the comment below the PR, I did mostly what's in the GzoltarClientMasterFaultLocalization.

best Leonhard

martinezmatias commented 3 years ago

Hi @lapplislazuli

I have updated the PR #312, could you have a look?

Thanks! I'll take a look now.

I did mostly what's in the GzoltarClientMasterFaultLocalization.

Perfect. I remember that class was quite complete but , at the time I wrote it some years ago, I did not have enough time to test the code and we continue using the GZoltar 0.X.

Thanks! Regards Matias

lapplislazuli commented 3 years ago

So, I am still working on this, just to keep everything somewhat documented: With the Code in #312 I get the program above Java 8 to run, for both above Java 8 Code as well as above Java 8 JVMs.

There are two new Issues:

Gzoltar does not run for Junit5, as recognized by the maintainers. Once there is a new version with better Test-Discovery it should be quickly adaptable. This is not really a Java Issue, but it still blocks usage of modern programs.
JGenProg seems to have major issues running on newer projects. Particularly, nearly all operators fail due to spoon issues. Catching all the exceptions feels like an antipattern to be honest, and they also keep popping up.

The Stacktraces look like these (for good measure, I copied and manually faulted Fraction.java from Apache Commons Lang 3.12 :

  14:15:04 [main] ERROR Line:86 AstorCoreEngine - Error at generation 32
  java.lang.IndexOutOfBoundsException: Index: -1, Size: 9
  java.lang.IndexOutOfBoundsException: Index: -1, Size: 9
    at java.base/java.util.ArrayList.rangeCheckForAdd(ArrayList.java:788)
    at java.base/java.util.ArrayList.add(ArrayList.java:513)
    at spoon.support.reflect.code.CtStatementImpl$InsertType$1.insertFromFirstStatement(CtStatementImpl.java:177)
    at spoon.support.reflect.code.CtStatementImpl$InsertVisitor.visitCtBlock(CtStatementImpl.java:94)
    at spoon.support.reflect.code.CtBlockImpl.accept(CtBlockImpl.java:58)
    at spoon.reflect.visitor.CtInheritanceScanner.scan(CtInheritanceScanner.java:178)
    at spoon.support.reflect.code.CtStatementImpl.insertBefore(CtStatementImpl.java:76)
    at spoon.support.reflect.code.CtStatementImpl.insertBefore(CtStatementImpl.java:57)
    at spoon.support.reflect.code.CtStatementImpl.insertBefore(CtStatementImpl.java:249)
    at fr.inria.astor.approaches.jgenprog.operators.InsertBeforeOp.applyChangesInModel(InsertBeforeOp.java:27)
[...]

  16:16:26 [main] ERROR Line:86 AstorCoreEngine - Error at generation 32
  java.lang.IndexOutOfBoundsException: Index 9 out of bounds for length 9
  java.lang.IndexOutOfBoundsException: Index 9 out of bounds for length 9
    at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
    at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
    at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
    at java.base/java.util.Objects.checkIndex(Objects.java:372)
    at java.base/java.util.ArrayList.get(ArrayList.java:459)
    at spoon.support.util.ModelList.get(ModelList.java:46)
    at spoon.support.reflect.code.CtBlockImpl.getStatement(CtBlockImpl.java:69)
    at fr.inria.astor.approaches.jgenprog.operators.StatementSupporter.remove(StatementSupporter.java:39)
    at fr.inria.astor.approaches.jgenprog.operators.InsertBeforeOp.undoChangesInModel(InsertBeforeOp.java:51)
    at fr.inria.astor.core.entities.OperatorInstance.undoModification(OperatorInstance.java:181)
[...]

  spoon.SpoonException: The default behavior has changed, a new check has been added! Don't worry, you can disable this check
  with one of the following options:
   - by configuring Spoon with getEnvironment().setSelfChecks(true)
   - by removing the node from its previous parent (element.delete())
   - by cloning the node before adding it here (element.clone())

    at spoon.support.util.ModelList.linkToParent(ModelList.java:87)
    at spoon.support.util.ModelList.add(ModelList.java:123)
    at spoon.support.util.ModelList.add(ModelList.java:30)
    at java.base/java.util.AbstractCollection.addAll(AbstractCollection.java:352)
    at spoon.support.util.ModelList.set(ModelList.java:57)
    at spoon.support.reflect.code.CtBlockImpl.setStatements(CtBlockImpl.java:164)
    at spoon.support.reflect.code.CtStatementImpl$InsertType$1.insertFromFirstStatement(CtStatementImpl.java:182)
    at spoon.support.reflect.code.CtStatementImpl$InsertVisitor.visitCtBlock(CtStatementImpl.java:94)
    at spoon.support.reflect.code.CtBlockImpl.accept(CtBlockImpl.java:58)
    at spoon.reflect.visitor.CtInheritanceScanner.scan(CtInheritanceScanner.java:178)
    at spoon.support.reflect.code.CtStatementImpl.insertBefore(CtStatementImpl.java:76)
    at spoon.support.reflect.code.CtStatementImpl.insertBefore(CtStatementImpl.java:57)
    at spoon.support.reflect.code.CtStatementImpl.insertBefore(CtStatementImpl.java:249)
    at fr.inria.astor.approaches.jgenprog.operators.InsertBeforeOp.applyChangesInModel(InsertBeforeOp.java:28)
[...]

So the operators count out of the AST for both doing changes and undoing changes, but they do so lower down in Spoon. I fear that just returning false in case of errors, a) the runtime might suffer, as many attempts are just discarded b) the search space might suffer, as actually good solutions cannot be tried

JakkuSakura commented 3 years ago

I have tried java 11 and 16 on Manjaro linux. java 8 is fine. I'm planing to make astor an plugin for Intellij Idea, and lastest version requires java 11.

It compiles easily, but bugged out at runtime,

java -cp target/astor-1.1.0-jar-with-dependencies.jar fr.inria.main.evolution.AstorMain -mode jGenProg -srcjavafolder /src/java/ -srctestfolder /src/test/ -binjavafolder /target/classes/ -bintestfolder /target/test-classes -location /home/jack/Dev/program-repair/intellij-idea-astor-plugin/astor/examples/math_70/ -dependencies  /home/jack/Dev/program-repair/intellij-idea-astor-plugin/astor/examples/math_70/lib
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
19:08:31.073 [main] ERROR fr.inria.astor.core.setup.ProjectConfiguration - Error retrieving java version: java.io.IOException: Cannot run program "/usr/lib/jvm/java-11-openjdk/java" (in directory "/home/jack/Dev/program-repair/intellij-idea-astor-plugin/astor/examples/math_70"): error=2, No such file or directory
19:08:31.074 [main] ERROR fr.inria.astor.core.setup.ProjectConfiguration - Error retrieving java version output obtained: 

Exception in thread "main" java.lang.ExceptionInInitializerError
        at fr.inria.astor.core.manipulation.MutationSupporter.<clinit>(MutationSupporter.java:169)
        at fr.inria.main.evolution.AstorMain.createEngine(AstorMain.java:77)
        at fr.inria.main.evolution.AstorMain.run(AstorMain.java:173)
        at fr.inria.main.evolution.AstorMain.execute(AstorMain.java:233)
        at fr.inria.main.evolution.AstorMain.main(AstorMain.java:204)
Caused by: java.lang.UnsupportedOperationException: No class provided, and an appropriate one cannot be found.
        at org.apache.logging.log4j.LogManager.callerClass(LogManager.java:576)
        at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:601)
        at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:588)
        at spoon.support.reflect.declaration.CtElementImpl.<clinit>(CtElementImpl.java:77)
        ... 5 more

SpoonLabs / astor

Moving Above Java 8 #307

Proposal

Options

Note