coverage definition, only DUT/REF, interesting corpus

[Gloria-cpu]

您好，

我现在已经跑通这个rocketchip的案例：https://github.com/OpenXiangShan/difftest/blob/master/.github/workflows/main.yml#L267， 我目前有三个问题想请教一下：

1）这个案例是rocketchip和spike的联合仿真，那覆盖率的计算方式是什么？DUT和REF的覆盖都算在里面吗？

2）可以单独跑一个模型进行模糊测试吗？比如只跑rocket chip或spike进行模糊测试

3）在模糊测试结束后，期间变异产生的有效corpus被保存在了哪里？

Hello,

I have now run through this rocketchip case: https://github.com/OpenXiangShan/difftest/blob/master/.github/workflows/main.yml#L267, I currently have three questions to ask:

1) This case is a joint simulation of rocketchip and spike, so how is the coverage calculated? Are the coverage of DUT and REF included?

2) Can I run a model alone for fuzz testing? For example, only run rocket chip or spike for fuzz testing

3) After the fuzz test is completed, where is the valid corpus generated by the mutation during the period saved?

[poemonsense]

1) The coverage used in the demo is the C++ branch coverage of Spike (instrumented by the SANCOV=1 when building the spike-so with command make -C riscv-isa-sim/difftest CPU=ROCKET_CHIP SANCOV=1 -j2)

Other coverage metrics are supported as well. See the README of xfuzz for how to instrument the DUT.

Also, multiple coverage metrics can be concatenated. There is a UnionCoverage class in C++. You may refer to the source code to see how it can be used to combine multiple metrics, such as DUT+REF.

Fuzzing the Spike REF (without DUT) is also supported. Please try https://github.com/OpenXiangShan/riscv-isa-sim/tree/difftest/spike_fuzz

2) This depends on the simulation framework. I think set NO_DIFF will remove the REF.

However, without the REF, the fuzzer does not know whether bugs are triggered.

3) ./build/fuzzer --help shows the full command list. You may also find them in the source code: https://github.com/OpenXiangShan/xfuzz/blob/main/src/lib.rs#L21-L49

Specifically for the valid corpus, using --corpus-output /path/to/output/dir will save the corpus to disk.

[Gloria-cpu]

Hi,

Thank you for the previous response; it has been really helpful, and I have managed to resolve the issues I faced earlier. I now have a new question:

I would like to perform a comprehensive run with xfuzz in the difftest framework. This includes running the DUT and REF, taking snapshots, comparing the results of the DUT and the REF, and any other relevant operations. Could you provide guidance on how to go about this?

Thank you!

[poemonsense]

The rocketchip demo contains running the DUT/REF with comparison between their results. You may try inject a bug into rocket (for example change the CSR logic) and test whether the fuzzer can find it.

What do you mean by taking snapshots? Is it LightSSS in difftest? That has no relation with xfuzz.

[Gloria-cpu]

Thank you for your previous response. It helped me resolve the issues I faced earlier.

Now I have a new question: Does xfuzz and rocketchip support ELF format for corpus_input? And, does the xfuzz is support with vcs, such as the Verilator Build with VCS in Nutshell(https://github.com/OpenXiangShan/difftest/blob/503f829ea8990551e7e5ba5f35b6428b90a18f1e/.github/workflows/main.yml#L310C15-L310C39).

[poemonsense]

Now I have a new question: Does xfuzz and rocketchip support ELF format for corpus_input?

Basically, No. The format should be supported by the simulation framework as a valid input. Since DiffTest does not accept elf files, xfuzz currently does not support it.

However, xfuzz is a fuzzer and the answer should be Yes. For example, if you have some other simulation environments that accept ELF files, then it should work perfectly with xfuzz. The xfuzz fuzzer does not know whether the input is ELF or RAW.

And, does the xfuzz is support with vcs, such as the Verilator Build with VCS in Nutshell(https://github.com/OpenXiangShan/difftest/blob/503f829ea8990551e7e5ba5f35b6428b90a18f1e/.github/workflows/main.yml#L310C15-L310C39).

Theorically yes, however, I'm not sure whether it works in the current form. As I wrote in the README, the required interfaces between Rust and C/C++ modules are listed exclusively at the harness file. If you (or difftest) has these interfaces when using VCS for simulation, then xfuzz should work.

I think these questions are for the simulation environment instead of this fuzzer. Basically the fuzzer relies only on these interfaces at the harness.rs. To support ELF or VCS, we should check the difftest repo or other simulation frameworks you are willing to use.