This is the RISC-V torture test generator and framework. This repository contains three sub-projects that build upon one another. The first, [generator], is used to create a single random torture test. The second, [testrun], is used to run a particular test on particular simulators, diffing the resulting signature with the ISA simulator and optionally creating a derivative test subset that pinpoints the divergence. The third, [overnight], wraps testrun, allowing tests to be run repeatedly for a given duration or until a failure count.
Modify "config/default.config" to set the parameters desired for building tests (e.g., setting which instructions to use and in which ratio).
Modify "Makefile" as desired to execute the C simulator or RTL simulator of your choice, and to set the other parameters as you require.
To build a single test and test it on Spike:
$ make igentest
To build single test and run it on the C simulator or RTL simulator, use "make cgentest" or "make rgentest".
To run overnight tests, you can use "make cnight" and "make rnight".
Torture works by dumping the register state out to memory at the end of the test program execution. This output is then compared against the output from the Spike ISA simulator.
The torture program writes the register state to the memory address specified by "xreg_output_data", which is located in the memory section ".global begin_signature". The Spike ISA simulator will write out the data found in the "begin_signature" section on exit if provided with the "+signature=" argument:
$ spike +signature=my_spike_signature.txt test_binary
The Rocket-chip infrastructure uses the "riscv-fesvr" program to control the execution of the C and RTL simulators. The "riscv-fesvr" also accepts the +signature argument too.
$ ./csim-rocket-chip +signature=my_rocket_signature.txt test_binary
A simple diff between the Spike and chip simulator signatures will tell you if any errors have occurred.
$ diff my_spike_signature.txt my_rocket_signature.txt
PORTING TORTURE TO YOUR OWN RISC-V PROCESSOR:
If you would like to use riscv-torture with your own RISC-V processor, you will need to provide a way to dump the "begin_signature" section to a file.
Some basic use cases are illustrated here (note the Makefile abstracts this for you).
Make a single test: % ./sbt generator/run % cd output % make % spike +signature=test.sig test
Take an existing test and diff the signatures of ISA and C simulators: % ./sbt 'testrun/run -a output/test.S -c /path/to/reference-chip/emulator/emulator'
Currently, due to the limiation of scala process library, you cannot torture the RTL simulator
Run tests for 30 minutes, email hcook when done, and save failures to dir: % ./sbt 'overnight/run -m 30 -e hcook@eecs.berkeley.edu -p dir'
% git submodule update --init
This framework utilizes both the test runner and test generator to perform a long terms serach for failing test cases. It takes the following command line arguments:
Usage: overnight/run [options]
-C
You can only generate tests with one instruction mix at a time (based on the setting in the config file). It doesn't matter what simulator you use with the -r and -c flags, they just determines the name used to describe whose diff failed.
This utility compares the signatures generated by passing the -testsig flag to the specified simulators. If it encounters a difference, it subdivides the test into many subtests and searches for which exact program segment reveals the failure. It takes the following command line arguments:
Usage: testrun/run [options]
-C
If you don't specify a asm file, a random one will be generated for you. You can only generate tests with one instruction mix at a time (based on the setting in the config file). It doesn't matter what simulator you use with the -r and -c flags, they just determines the name used to describe whose diff failed. By default, a failed diff will result in the subtest sweep occuring, but this search can be diasbled. Note that the pseg ID reported is actually the pseg following the pseg containing the error. You can optionally dump mistmatched signatures to the dir containing the asm file under test.
To generate a random test, the torture test generator randomly generates many test sequences from a set of test sequences that are written by hand, performs a random register allocation for all test sequences, and finally randomly interleaves instructions from these test sequences. To extend the set of tests or coverage, the programmer needs to write new test sequences. It takes the following command line arguments:
Usage: generator/run [options]
-o
The following sections describe adding new functionality to the generator.
Before we talk about how to write a test sequence, let's look at a very simple example. The following example is a test sequence, which emits an add instruction.
class SeqADD extends Seq { val src1 = reg_read_any() val src2 = reg_read_any() val dest = reg_write(src1, src2) insts += ADD(dest, src1, src2) }
As I hinted in the overview that the test generator will do register allocation you don't write a string of instructions with architectural registers. You request a virtual registers (i.e., registers that are yet tied down to architectural registers) when you need them, save them in scala values, and use them when you need to (e.g., in an instruction).
Hidden (position dependent registers): Registers that will have different values when the code is positioned at a different address. An example is registers that hold addresses. Registers that are hidden should be excluded from the output signature.
Visible (position independent registers): Registers that are not hidden, therefore will have the same values when the code is positioned at a different address. These registers should be included as part of the output signature.
Use the following functions to request a register, and generate a string of instructions (look at Inst.scala to see what instructions are available) that uses these virtual registers, and add them to the insts array.
Note that the torture test framework is written in scala, you can use any scala functionality to generate instructions. Look at SeqALU.scala, SeqMem.scala, and SeqBranch.scala to get inspired.
provide support for loops
generate statistics of a test to get a sense of coverage
complete floating point tests
complete vector tests
code refactoring