jose
commented
1 year ago I've just tried Major v2.0.0 on a few bugs and I'm afraid it doesn't work as expected.
Chart-10
[mkdir] Created dir: /localtmp/slurm_328512/jcampos-464/build
[javac] /tmp/defects4j/framework/projects/Chart/Chart.build.xml:57: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 578 source files to /localtmp/slurm_328512/jcampos-464/build
[javac] Picked up _JAVA_OPTIONS: -Xmx8192M -XX:MaxHeapSize=1024M
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.4
[javac] warning: [options] source value 1.4 is obsolete and will be removed in a future release
[javac] warning: [options] target value 1.4 is obsolete and will be removed in a future release
[javac] warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
[javac] Exception in thread "main" java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
[javac] at com.sun.tools.javac.main.Main.compile(Main.java:559)
[javac] at com.sun.tools.javac.main.Main.compile(Main.java:381)
[javac] at com.sun.tools.javac.main.Main.compile(Main.java:370)
[javac] at com.sun.tools.javac.main.Main.compile(Main.java:361)
[javac] at com.sun.tools.javac.Main.compile(Main.java:56)
[javac] at com.sun.tools.javac.Main.main(Main.java:42)
[javac] Caused by: java.lang.OutOfMemoryError: Java heap space
[javac] at java.util.Arrays.copyOf(Arrays.java:3332)
[javac] at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
[javac] at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
[javac] at java.lang.StringBuilder.append(StringBuilder.java:136)
[javac] at java.lang.StringBuilder.append(StringBuilder.java:76)
[javac] at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:484)
[javac] at java.lang.StringBuilder.append(StringBuilder.java:166)
[javac] at java.lang.StringBuilder.append(StringBuilder.java:76)
[javac] at java.util.Formatter$FixedString.print(Formatter.java:2595)
[javac] at java.util.Formatter.format(Formatter.java:2508)
[javac] at java.util.Formatter.format(Formatter.java:2455)
[javac] at java.lang.String.format(String.java:2940)
[javac] at org.mutationtesting.major.mutation.tree.AttrTreeCopier.visitMethod(AttrTreeCopier.java:833)
[javac] at org.mutationtesting.major.mutation.tree.AttrTreeCopier.visitMethod(AttrTreeCopier.java:132)
[javac] at com.sun.tools.javac.tree.JCTree$JCMethodDecl.accept(JCTree.java:800)
[javac] at org.mutationtesting.major.mutation.tree.AttrTreeCopier.copy(AttrTreeCopier.java:216)
[javac] at org.mutationtesting.major.mutation.tree.AttrTreeCopier.copy(AttrTreeCopier.java:244)
[javac] at org.mutationtesting.major.mutation.tree.AttrTreeCopier.visitClass(AttrTreeCopier.java:667)
[javac] at org.mutationtesting.major.mutation.tree.AttrTreeCopier.visitClass(AttrTreeCopier.java:132)
[javac] at com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:720)
[javac] at org.mutationtesting.major.mutation.tree.AttrTreeCopier.copy(AttrTreeCopier.java:216)
[javac] at org.mutationtesting.major.mutation.tree.AttrTreeCopier.copy(AttrTreeCopier.java:207)
[javac] at org.mutationtesting.major.mutation.Mutator.isMutantValid(Mutator.java:2313)
[javac] at org.mutationtesting.major.mutation.Mutator.replaceTreeWithDefault(Mutator.java:1741)
[javac] at org.mutationtesting.major.mutation.Mutator.visitVarDef(Mutator.java:753)
[javac] at com.sun.tools.javac.tree.JCTree$JCVariableDecl.accept(JCTree.java:852)
[javac] at com.sun.tools.javac.tree.TreeTranslator.translate(TreeTranslator.java:58)
[javac] at org.mutationtesting.major.mutation.Mutator.translate(Mutator.java:448)
[javac] at com.sun.tools.javac.tree.TreeTranslator.translate(TreeTranslator.java:70)Collections-25
mutation.test:
[echo] Running mutation analysis ...
[junit] MAJOR: Mutation analysis enabled
[junit] MAJOR: ------------------------------------------------------------
[junit] MAJOR: Run 682 ordered test(s)to verify independence
[junit] MAJOR: junit.framework.TestSuite$1[warning] failed!
[junit] MAJOR: -> AssertionFailedError
[junit] MAJOR: -> "Class org.apache.commons.collections4.bidimap.AbstractBidiMapTest$TestInverseBidiMap has no public constructor TestCase(String name) or TestCase()"Compress-40
commons-compress-from-maven.compile:
[mkdir] Created dir: /localtmp/slurm_328514/jcampos-9379/target/classes
[javac] /localtmp/slurm_328514/jcampos-9379/maven-build.xml:89: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 166 source files to /localtmp/slurm_328514/jcampos-9379/target/classes
[javac] Picked up _JAVA_OPTIONS: -Xmx8192M -XX:MaxHeapSize=1024M
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[javac] /localtmp/slurm_328514/jcampos-9379/src/main/java/org/apache/commons/compress/compressors/snappy/PureJavaCrc32C.java:116: error: code too large
[javac] private static final int[] T = new int[] {
[javac] ^
[javac] Generated 27861 mutants (19401 ms)
[javac] 1 error
[javac] 1 warningGson-14
Compiling mutant definition (mml).......................................... FAIL
Executed command: /tmp/defects4j/major/bin/mmlc /localtmp/slurm_328515/jcampos-12643/.mml/default.mml 2>&1
Warning: Operator LVR has already been disabled for [org, apache, commons, math3, util, FastMathLiteralArrays]
Warning: Operator ORU has already been disabled for [org, apache, commons, math3, util, FastMathLiteralArrays]
Warning: Operator LVR has already been disabled for [org, apache, commons, math, optimization, direct, BOBYQAOptimizer, bobyqb]
Warning: Operator STD has already been disabled for [org, apache, commons, math, optimization, direct, BOBYQAOptimizer, bobyqb]
line 62:42 Error: invalid scope: com.google.gson.internal.$Gson$Types
line 63:42 Error: invalid scope: com.google.gson.internal.$Gson$Types
line 64:42 Error: invalid scope: com.google.gson.internal.$Gson$Types
line 65:42 Error: invalid scope: com.google.gson.internal.$Gson$Types
line 66:42 Error: invalid scope: com.google.gson.internal.$Gson$Types
line 67:42 Error: invalid scope: com.google.gson.internal.$Gson$Types
line 68:42 Error: invalid scope: com.google.gson.internal.$Gson$Types
line 69:42 Error: invalid scope: com.google.gson.internal.$Gson$Types
[!!!] Parsing encountered an error
[!!!]
The compilation process ended with errors!
Cannot compile mml file: /localtmp/slurm_328515/jcampos-12643/.mml/default.mml! at /tmp/defects4j/framework/core/Mutation.pm line 99.
Compilation failed in require at /tmp//defects4j/framework/bin/defects4j line 195.JacksonDatabind-50
jackson-databind-from-maven.compile:
[mkdir] Created dir: /localtmp/slurm_328516/jcampos-13143/target/classes
[javac] /localtmp/slurm_328516/jcampos-13143/maven-build.xml:92: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 399 source files to /localtmp/slurm_328516/jcampos-13143/target/classes
[javac] /localtmp/slurm_328516/jcampos-13143/src/main/java/com/fasterxml/jackson/databind/ser/PropertyBuilder.java:158: warning: [deprecation] WRITE_EMPTY_JSON_ARRAYS in SerializationFeature has been deprecated
[javac] && !_config.isEnabled(SerializationFeature.WRITE_EMPTY_JSON_ARRAYS)) {
[javac] ^
[javac] Exception in thread "main" java.lang.RuntimeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
[javac] at com.sun.tools.javac.main.Main.compile(Main.java:559)
[javac] at com.sun.tools.javac.main.Main.compile(Main.java:381)
[javac] at com.sun.tools.javac.main.Main.compile(Main.java:370)
[javac] at com.sun.tools.javac.main.Main.compile(Main.java:361)
[javac] at com.sun.tools.javac.Main.compile(Main.java:56)
[javac] at com.sun.tools.javac.Main.main(Main.java:42)Jsoup-4
jsoup-from-maven.compile:
[mkdir] Created dir: /localtmp/slurm_328517/jcampos-24404/target/classes
[javac] /localtmp/slurm_328517/jcampos-24404/maven-build.xml:74: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 32 source files to /localtmp/slurm_328517/jcampos-24404/target/classes
[javac] Picked up _JAVA_OPTIONS: -Xmx8192M -XX:MaxHeapSize=1024M
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.5
[javac] warning: [options] source value 1.5 is obsolete and will be removed in a future release
[javac] warning: [options] target value 1.5 is obsolete and will be removed in a future release
[javac] warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
[javac] /localtmp/slurm_328517/jcampos-24404/src/main/java/org/jsoup/nodes/Entities.java:24: error: code too large
[javac] private static final Pattern unescapePattern = Pattern.compile("&(#(x|X)?([0-9a-fA-F]+)|[a-zA-Z]+);?");
[javac] ^
[javac] Generated 6646 mutants (4301 ms)
[javac] Note: /localtmp/slurm_328517/jcampos-24404/src/main/java/org/jsoup/nodes/Node.java uses unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error
[javac] 4 warningsJsoup-67
mutation.test:
[echo] Running mutation analysis ...
[junit] MAJOR: Mutation analysis enabled
[junit] MAJOR: ------------------------------------------------------------
[junit] MAJOR: Run 28 ordered test(s)to verify independence
[junit] MAJOR: org.jsoup.parser.HtmlParserTest[handlesDeepStack] failed!
[junit] MAJOR: -> AssertionFailedError
[junit] MAJOR: -> "null"Mockito-1
Check out program version: Mockito-1f...................................... OK
Compiling mutant definition (mml).......................................... OK
Running ant (mutate)....................................................... OK
Cannot open mutants log: No such file or directory at /tmp/defects4j/framework/core/Project.pm line 856.
Compilation failed in require at /tmp//defects4j/framework/bin/defects4j line 195Mockito-6
Check out program version: Mockito-6f...................................... OK
Compiling mutant definition (mml).......................................... OK
Running ant (mutate)....................................................... OK
Cannot open mutants log: No such file or directory at /tmp/defects4j/framework/core/Project.pm line 856.
Compilation failed in require at /tmp//defects4j/framework/bin/defects4j line 195.
rjust
Hi @rjust,
Have you performed mutation analysis on all bugs recently? I've just done it and it has failed on 92 bugs.
-Xmx8192M -XX:MaxHeapSize=1024M)-Xmx8192M -XX:MaxHeapSize=1024M)-Xmx8192M -XX:MaxHeapSize=1024M)Chart-10
Closure-{73, 145}
Collections-{25, 26, 27, 28}
Compress-{40, 41, 42, 43, 44, 45, 46, 47}
Csv-{14, 15, 16}
Gson-{14, 16, 18}
JacksonDatabind-{50, 51, 52, 54, 59, 60, 61, 63, 64, 65, 66, 67, 68, 70, 73, 74, 75, 76, 78, 83, 84, 85, 87, 88, 89, 91, 93, 95}
JacksonDatabind-{80, 81, 109}
JacksonDatabind-{82, 86, 90, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 110, 111, 112}
JacksonDatabind-92
Jsoup-{4, 6, 9}
Quick workaround: we could setup major to not apply one specific mutation operator (or several) to the
org.jsoup.nodes.Entitiesclass as we've done for other classes. Question is, which mutation operator(s) should we exclude from being applied to theorg.jsoup.nodes.Entitiesclass to avoid thecode too largeerror?Jsoup-67
Mockito-{1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 18, 19, 20, 21}
Mockito-{6, 35}
This analysis was conducted with Major v1.3.1 and I'm now wondering whether the latest version v2.0.0 addresses some of these issues. For example, according to its changelog, v2.0.0 addresses the
code too largeerror. I will give v2.0.0 a try and let you know whether some of the issues reported above are no longer an issue in v2.0.0, or if there is any other unknown issue. (I might need to use the update-major branch for that.)-- Best, Jose