search

joakim-brannstrom / dextool

Suite of C/C++ tooling built on LLVM/Clang
http://joakim-brannstrom.github.io/dextool/
Other
101 stars 27 forks source link

Cannot Execute Mutation Tests with GTest #1732

Closed pigniczkim closed 5 months ago

pigniczkim commented 5 months ago

Hi, I want to integrate dextool into my toolchain for C/C++ testing, so I make a small example project to try it out. Unfortunately when I want to run dextool mutate test it ends up in an error.

I attach the project that I have. example.zip

I built the latest version of dextool (from GIT) using ldc 1.38.0. I am trying to execute the test project under an ubunutu:jammy docker container with llvm-18.

So far the dextool mutate analyze seems to recognize the files correctly, but when it comes to executing the mutation tests it fails.

I turned on debugging so you can see the error logs coming:

(ldc-1.38.0)root@0d89f26698a3:/work/example# dextool mutate test -d
trace: ["/opt/dextool_install/bin/dextool-mutate", "test"] [dextool.plugin.runner.runPlugin:46]
trace: ArgParser(MiniConfig(".dextool_mutate.toml", /work/example/.dextool_mutate.toml, false), ConfigAdmin(none, unknown, unknown, "", 0, "", [], []), ConfigAnalyze([all], [], ["src/*"], GlobFilter(["/work/example/src/*"], []), 0, false, true, false, false, false, ["test"], [/work/example/test], [], ["*"], GlobFilter(["/work/example/*"], []), relaxed), ConfigCompileDb(["./build/compile_commands.json"], [], CompileCommandFilter(["-c", "-o", "-m", "-nodevicelib", "-Waddr-space-convert", "-non-static", "-Bstatic", "-Bdynamic", "-Xbind-lazy", "-Xbind-now", "-f", "-static", "-shared", "-rdynamic", "-s", "-l", "-L", "-z", "-u", "-T", "-Xlinker", "-l", "-MT", "-MF", "-MD", "-MQ", "-MMD", "-MP", "-MG", "-E", "-cc1", "-S", "-M", "-MM", "-###"], 0)), ConfigCompiler([], false, "", AllowErrors(false)), ConfigMutationTest([shell command './test.sh'], [], [], recursive, shell command './build.sh', [], [gtest, ctest], Nullable.null, "", 30 minutes, bySize, false, resetAlive, remove, test, 0, OldMutantPercentage(1), 52 weeks, TestConstraint([]), false, Nullable.null, 0, true, nothing, LoadThreshold(15), ContinuesCheckTestSuite(true), ContinuesCheckTestSuitePeriod(100), TestCmdChecksum(false), MaxTestCaseOutputCaptureMbyte(10), UseSkipMutant(true), MaxMemoryUsage(90)), ConfigReport(plain, [summary], , top, 20, [], false, false, HighInterestMutantsNr(5), None()), ConfigWorkArea(".", /work/example, [], ["src/*"], GlobFilter(["/work/example/src/*"], [])), ConfigGenerate(0, []), ConfigCoverage(true, inject, false, []), ConfigSchema(true, inject, MutantsPerSchema(1000), MinMutantsPerSchema(3), true, false, false, [], SchemaTrainGenerator(false), 12, 2), Data(/work/example/dextool_mutate.sqlite3, Ok, test_mutants, false, []), GetoptResult(false, [Option("-L", "", "restrict testing to the requested files and lines (<file>:<start>-<end>)", false), Option("", "--build-cmd", "program used to build the application", false), Option("", "--cont-test-suite", "enable continues check of the test suite", false), Option("", "--cont-test-suite-period", "how often to check the test suite", false), Option("-c", "--config", "load configuration (default: .dextool_mutate.toml)", false), Option("", "--db", "sqlite3 database to use (default: dextool_mutate.sqlite3)", false), Option("", "--diff-from-stdin", "restrict testing to the mutants in the diff", false), Option("", "--dry-run", "do not write data to the filesystem", false), Option("", "--exclude", "do not mutate the files matching any the patterns (default: <empty>)", false), Option("", "--include", "only mutate the files matching at least one of the patterns (default: *)", false), Option("", "--load-behavior", "how to behave when the threshold is hit [nothing|slowdown|halt]", false), Option("", "--load-threshold", "the 15min loadavg threshold (default: 15)", false), Option("", "--log-coverage", "write the instrumented coverage files to a separate file", false), Option("", "--max-alive", "stop after NR alive mutants is found (only 
effective with -L or --diff-from-stdin)", false), Option("", "--max-runtime", "max time to run the mutation testing for (default: 52 weeks)", false), Option("", "--metadata", "prioritieses files that are sent by JSON", false), Option("-m", "--mutant", "do not use. this option is deprecated", false), Option("", "--no-skipped", "do not skip mutants that are covered by others", false), Option("", "--order", "determine in what order mutants are chosen [random|consecutive|bySize]", false), Option("", "--out", "path used as the root for mutation/reporting of files (default: .)", false), Option("", "--schema-check", "sanity check a schemata before it is used", false), Option("", "--schema-log", "write mutant schematan to a separate file for later 
inspection", false), Option("", "--schema-min-mutants", "mini number of mutants per schema", false), Option("", "--schema-only", "stop testing after the last schema has been executed", false), Option("", "--schema-parallel-mutants", "nr of mutants to test in parallel", false), Option("", "--schema-train", "train the schema generator by only compiling the scheman", false), Option("", "--schema-use", "use schematas to speed-up testing", false), Option("", "--test-case-analyze-builtin", "builtin analyzer of output from testing frameworks to find failing test cases", false), Option("", "--test-case-analyze-cmd", "program used to find what test cases killed the mutant", false), Option("", "--test-cmd", "program used to run the test suite", false), Option("", "--test-cmd-checksum", "use checksums of the original to avoid running unmodified tests", false), Option("", "--test-timeout", "timeout to use for the test suite (msecs)", false), Option("", "--timeout-scale", "the factor of which the timeout time is multiplied with", false), Option("", "--use-early-stop", "stop executing tests for a mutant as soon as one kill a mutant to speed-up testing", false), Option("-h", "--help", "This help information.", false)]), ["generate":void delegate(string[]) @system, "report":void delegate(string[]) @system, "analyze":void delegate(string[]) @system, "test":void delegate(string[]) @system, "admin":void delegate(string[]) @system]) [dextool.plugin.runner.runPlugin.__lambda4:51]
trace: ToolMode: test_mutants [dextool.plugin.mutate.frontend.frontend.runMutate:33]
info: Using /work/example/dextool_mutate.sqlite3 [dextool.plugin.mutate.frontend.frontend.runMutate:46]
trace: Opening database /work/example/dextool_mutate.sqlite3 [dextool.plugin.mutate.backend.database.schema.initializeDB:171]
trace: None() -> Initialize(false) [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
info: Initializing worklist [dextool.plugin.mutate.backend.test_mutant.TestDriver.opCall:645]
info: Memory limit set minium 1584 Mbyte [dextool.plugin.mutate.backend.test_mutant.TestDriver.opCall:660]
trace: Initialize(false) -> SanityCheck(false) [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
info: Sanity check of files to mutate [dextool.plugin.mutate.backend.test_mutant.TestDriver.opCall:766]
info: Ok [dextool.plugin.mutate.backend.test_mutant.TestDriver.opCall:819]
trace: SanityCheck(false) -> PreCompileSut(false) [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
info: Checking the build command [dextool.plugin.mutate.backend.test_mutant.TestDriver.opCall:1220]
trace: Opening database /work/example/dextool_mutate.sqlite3 [dextool.plugin.mutate.backend.database.schema.initializeDB:171]
trace: Opening database /work/example/dextool_mutate.sqlite3 [dextool.plugin.mutate.backend.database.schema.initializeDB:171]
trace: PreCompileSut(false) -> UpdateTestCmds() [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
trace: UpdateTestCmds() -> ChooseMode() [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
trace: ChooseMode() -> AnalyzeTestCmdForTestCase(false, []) [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
shell command './test.sh' | false | [DrainElement(stdout, [84, 69, 83, 84, 73, 78, 71, 46, 46, 46, 10]), DrainElement(stdout, [82, 117, 110, 110, 105, 110, 103, 32, 116, 101, 115, 116, 115, 46, 46, 46, 10]), DrainElement(stdout, [84, 101, 115, 116, 32, 112, 114, 111, 106, 101, 99, 116, 32, 47, 119, 111, 114, 107, 47, 101, 120, 97, 109, 112, 108, 101, 47, 98, 117, 105, 108, 100, 10]), DrainElement(stdout, [32, 32, 32, 32, 83, 116, 97, 114, 116, 32, 49, 58, 32, 70, 117, 110, 99, 116, 105, 111, 110, 84, 101, 115, 116, 46, 84, 101, 115, 116, 78, 97, 109, 101, 10]), DrainElement(stdout, [49, 47, 49, 32, 84, 101, 115, 116, 32, 35, 49, 58, 32, 70, 117, 110, 99, 116, 105, 111, 110, 84, 101, 115, 116, 46, 84, 101, 115, 116, 78, 97, 109, 101, 32, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 32, 32, 32, 80, 97, 115, 115, 101, 100, 32, 32, 32, 32, 48, 46, 48, 49, 32, 115, 101, 99, 10]), DrainElement(stdout, [10, 49, 48, 48, 37, 32, 116, 101, 115, 116, 115, 32, 112, 97, 115, 115, 101, 100, 44, 32, 48, 32, 116, 101, 115, 116, 115, 32, 102, 97, 105, 108, 101, 100, 32, 111, 117, 116, 32, 111, 102, 32, 49, 10, 10, 84, 111, 116, 97, 108, 32, 84, 101, 115, 116, 32, 116, 105, 109, 101, 32, 40, 114, 101, 97, 108, 41, 32, 61, 32, 32, 32, 48, 46, 48, 52, 32, 115, 101, 99, 10])]
trace: AnalyzeTestCmdForTestCase(false, [shell command './test.sh':[FunctionTest.TestName]]) -> UpdateAndResetAliveMutants([shell command './test.sh':[FunctionTest.TestName]]) [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]trace: UpdateAndResetAliveMutants([shell command './test.sh':[FunctionTest.TestName]]) -> RetestOldMutant() [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
trace: RetestOldMutant() -> IncreaseFilePrio() [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
trace: IncreaseFilePrio() -> CheckMutantsLeft(false) [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
trace: CheckMutantsLeft(false) -> MeasureTestSuite(false) [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
info: Measuring the runtime of the test command(s):
shell command './test.sh' [dextool.plugin.mutate.backend.test_mutant.TestDriver.opCall:1336]
shell command './test.sh' | false | [DrainElement(stdout, [84, 69, 83, 84, 73, 78, 71, 46, 46, 46, 10]), DrainElement(stdout, [82, 117, 110, 110, 105, 110, 103, 32, 116, 101, 115, 116, 115, 46, 46, 46, 10]), DrainElement(stdout, [84, 101, 115, 116, 32, 112, 114, 111, 106, 101, 99, 116, 32, 47, 119, 111, 114, 107, 47, 101, 120, 97, 109, 112, 108, 101, 47, 98, 117, 105, 108, 100, 10]), DrainElement(stdout, [32, 32, 32, 32, 83, 116, 97, 114, 116, 32, 49, 58, 32, 70, 117, 110, 99, 116, 105, 111, 110, 84, 101, 115, 116, 46, 84, 101, 115, 116, 78, 97, 109, 101, 10]), DrainElement(stdout, [49, 47, 49, 32, 84, 101, 115, 116, 32, 35, 49, 58, 32, 70, 117, 110, 99, 116, 105, 111, 110, 84, 101, 115, 116, 46, 84, 101, 115, 116, 78, 97, 109, 101, 32, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 32, 32, 32, 80, 97, 115, 115, 101, 100, 32, 32, 32, 32, 48, 46, 48, 49, 32, 115, 101, 99, 10]), DrainElement(stdout, [10, 49, 48, 48, 37, 32, 116, 101, 115, 116, 115, 32, 112, 97, 115, 115, 101, 100, 44, 32, 48, 32, 116, 101, 115, 116, 115, 32, 102, 97, 105, 108, 101, 100, 32, 111, 117, 116, 32, 111, 102, 32, 49, 10, 10, 84, 111, 116, 97, 108, 32, 84, 101, 115, 116, 32, 116, 105, 109, 101, 32, 40, 114, 101, 97, 108, 41, 32, 61, 32, 32, 32, 48, 46, 48, 52, 32, 115, 101, 99, 10])]
info: 0: Measured test command runtime 109 ms, 837 μs, and 6 hnsecs [dextool.plugin.mutate.backend.test_mutant.measureTestCommand:191]
info: Test command runtime: 108 ms, 945 μs, and 8 hnsecs [dextool.plugin.mutate.backend.test_mutant.TestDriver.opCall:1357]
trace: MeasureTestSuite(false) -> Coverage(false, false) [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
info: Coverage information is up to date [dextool.plugin.mutate.backend.test_mutant.TestDriver.opCall:1565]
trace: Coverage(false, false) -> UpdateTimeout() [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
trace: UpdateTimeout() -> OverloadCheck(false) [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
trace: OverloadCheck(false) -> ContinuesCheckTestSuite(false) [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
trace: ContinuesCheckTestSuite(true) -> Cleanup() [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
trace: Cleanup() -> NextSchemata() [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
trace: NextSchemata() -> SchemataTest(false, StopTesting(false)) [dextool.plugin.mutate.backend.test_mutant.TestDriver.this.__lambda10:507]
trace: Opening database /work/example/dextool_mutate.sqlite3 [dextool.plugin.mutate.backend.database.schema.initializeDB:171]
trace: Timeout Scale Factor: 2 [dextool.plugin.mutate.backend.test_mutant.schemata.spawnSchema.init_:201]
trace: loadctrl output: 19.9261 [dextool.plugin.mutate.backend.test_mutant.schemata.load.spawnLoadCtrlActor.tick:61]
trace: Opening database /work/example/dextool_mutate.sqlite3 [dextool.plugin.mutate.backend.database.schema.initializeDB:171]
error: exiting... [application.app_main.rmain:188]
joakim-brannstrom commented 5 months ago

Hi, thank you for the bug report. I have finally gotten time to check it out. The example.zip was very helpful.

I managed to reproduce it. The bug has to do with schema and the "example" being very small. I guess no "schema" is being generated but it still try to use one thus "segfault". With your example I can fix the bug.

Until it is fixed you can change the configuration "use = true" to "false" in the section "[schema]".

It should work on larger program just fine because that is how it is used. There is something that happens when the example is minimalistic.

joakim-brannstrom commented 5 months ago

Should be fixed in #1734

pigniczkim commented 5 months ago

Thank you for the info. This solution seems to work fine.