This is the gem5 simulator for Xiangshan (XS-GEM5), which currently scores similar with Kunminghu on SPEC CPU 2006.
XS-GEM5 is not as easy to use as official GEM5, because it only supports full-system simulation with Xiangshan's specific formats, refer to Workflows for more details.
XS-GEM5 is enhanced with
Because XS-GEM5 is currently under internal development, we have several branches for different purposes:
The typical flow for running workloads is similar for NEMU, XS-GEM5, and Xiangshan processor. All of them only support full-system simulation. To prepare workloads for full-system simulation, users need to either build a baremetal app or running user programs in an operating system.
graph TD;
am["Build a baremetal app with AM"]
linux["Build a Linux image containing user app"]
baremetal[/"Image of baremetal app or OS"/]
run["Run image with NEMU, XS-GEM5, or Xiangshan processor"]
am-->baremetal
linux-->baremetal
baremetal-->runBecause most of the enterprise users and researchers are more interested in running larger workloads, like SPECCPU, on XS-GEM5. To reduce the simulation time of detailed simulation, NEMU serves as a checkpoint producer. The flow for producing and running checkpoints is as follows.
graph TD;
linux["Build a Linux image containing NEMU trap app and user app"]
bin[/"Image containing Linux and app"/]
profiling["Boot image with NEMU with SimPoint profiling"]
bbv[/"SimPoint BBV, a .gz file"/]
cluster["Cluster BBV with SimPoint"]
points[/"SimPoint sampled points and weights"/]
take_cpt["Boot image with NEMU to produce checkpoints"]
checkpoints[/"Checkpoints, several .gz files of memory image"/]
run["Run checkpoints with XS-GEM5"]
linux-->bin
bin-->profiling
profiling-->bbv
bbv-->cluster
cluster-->points
points-->take_cpt
take_cpt-->checkpoints
checkpoints-->run
As described above, XS-GEM5 either takes a baremetal app or a checkpoint as input.
To build baremetal app compatible with XS-GEM5, we use Abstract Machine as a light-weight baremetal library. Common simple apps like coremark and dhrystone can be built with Abstract Machine.
To obtain checkpoints of large applications, please follow the doc to build Linux to pack a image, and follow the checkpoint tutorial for Xiangshan to produce checkpoints.
The process to produce SimPoint checkpoints includes 3 individual steps
If you have problem generating SPECCPU checkpoints, following links might help you.
Install dependencies as official GEM5 tutorial says:
If compiling gem5 on Ubuntu 22.04, or related Linux distributions, you may install all these dependencies using APT:
sudo apt install build-essential git m4 scons zlib1g zlib1g-dev \
libprotobuf-dev protobuf-compiler libprotoc-dev libgoogle-perftools-dev \
python3-dev libboost-all-dev pkg-config libsqlite3-dev zstd libzstd-devIf compiling gem5 on Ubuntu 20.04, or related Linux distributions, you may install all these dependencies using APT:
sudo apt install build-essential git m4 scons zlib1g zlib1g-dev \
libprotobuf-dev protobuf-compiler libprotoc-dev libgoogle-perftools-dev \
python3-dev python-is-python3 libboost-all-dev pkg-config libsqlite3-dev zstd libzstd-devRefer to The readme for DRAMSim3 to install DRAMSim3.
Notes:
cd GEM5
scons build/RISCV/gem5.opt --gold-linker -j8
export gem5_home=`pwd`Press enter if you saw
You're missing the gem5 style or commit message hook. These hooks help
to ensure that your code follows gem5's style rules on git commit.
This script will now install the hook in your .git/hooks/ directory.
Press enter to continue, or ctrl-c to abort:Users must properly prepare workloads before running GEM5, plz read Workflows first.
The example running script contains the default command for simulate XS-GEM5. The example batch running script shows an example to simulate multiple workloads in parallel.
Users should set the following environment variables before running GEM5:
.so file of NEMU or spike.These files can be found in the release page. Users can also opt to build them from source (Difftest with NEMU and Build GCPT restorer). A tested working matrix of repos & revisions is here:
| Checkpoint Type | reference design | GCPT restorer |
|---|---|---|
| RV64GCB | NEMU master + riscv64-gem5-ref_defconfig | NEMU master |
| RV64GCBV | NEMU master + riscv64-gem5-ref_defconfig | NEMU gcpt_new_mem_layout |
| RV64GCB multi-core | NEMU master + riscv64-gem5-multicore-ref_defconfig | Download Binary from release; Code release soon |
| RV64GCBV multi-core | NEMU master + riscv64-gem5-multicore-ref_defconfig |
If above branches are not working, you can try the following commits:
| Checkpoint Type | reference design | GCPT restorer |
|---|---|---|
| RV64GCB | NEMU 4332a525 + riscv64-gem5-ref_defconfig | NEMU 732e4ccd |
| RV64GCBV | NEMU 4332a525 + riscv64-gem5-ref_defconfig | NEMU b966d274 |
| RV64GCB multi-core | NEMU 4332a525 + riscv64-gem5-multicore-ref_defconfig | Download Binary from release; Code release soon |
| RV64GCBV multi-core | NEMU 4332a525 + riscv64-gem5-multicore-ref_defconfig |
NOTE:
configs/example/xiangshan.py and configs/common/XSConfig.py to disable it.
Simulation error without Difftest will NOT be responded.Firstly, one should ensure GEM5 is properly built and workloads are prepared by running a single workload:
mkdir util/xs_scripts/example
cd util/xs_scripts/example
bash ../kmh_6wide.sh /path/to/a/single/checkpoint.gzThen, for running multiple workloads in parallel, one can use the batch running script:
mkdir util/xs_scripts/example
cd util/xs_scripts/example
bash ../parallel_sim.sh `realpath ../kmh_6wide.sh` $workloads_lst /top/dir/of/checkpoints a_fancy_simulation_tagIn this example, parallel_sim.sh will invoke kmh_6wide.sh with GNU parallel to run multiple workloads. Through this, parallel simulation infrastructure is decouple from the simulation script.
In order to be able to run scores on servers without root access, we provide a simple docker script to run xs-gem5. For more details see README about run in docker.
A line of workload_lst is a space-separated list of workload parameters.
For example, "hmmer_nph3_15858 hmmer_nph3/15858 0 0 20 20" represents the workload name, checkpoint path, skip insts (usually 0), functional warmup insts (usually 0),
detailed warmup insts (usually 20), and sample insts (usually 20), respectively.
parallel_sim.sh will find hmmer_nph3/15858/*.gz in the /top/dir/of/checkpoints to obtain the checkpoint gz file.
Then the gz file will be passed to kmh_6wide.sh to run the simulation.
More details can be found in comments and code of the example running scripts.
Arch DB is a database to store the micro-architectural trace of the program with SQLite. You can access it with Python or other languages. A Python example is given here.
git clone https://github.com/OpenXiangShan/NEMU.git
cd NEMU/resource/gcpt_restore
make
export GCB_RESTORER=`realpath build/gcpt.bin`If users want to build RVV version, run the following command:
git clone https://github.com/OpenXiangShan/NEMU.git -b gcpt_new_mem_layout
# Then similar as above
# ...
export GCBV_RESTORER=`realpath build/gcpt.bin`NEMU is used as a reference design for XS-GEM5. Typical workflow is as follows.
graph TD;
build["Build NEMU in reference mode"]
so[/"./build/riscv64-nemu-interpreter-so"/]
cosim["Run XS-GEM5 or Xiangshan processor, turn on difftest, specify riscv64-nemu-interpreter-so as reference design"]
build-->so
so-->cosimWe the gem5-ref-main branch of NEMU for difftest with XS-GEM5.
git clone https://github.com/OpenXiangShan/NEMU.git
cd NEMU
export NEMU_HOME=`pwd`
make riscv64-gem5-ref_defconfig
make -j 10Then the contents of build directory should be
build
|-- obj-riscv64-nemu-interpreter-so
| `-- src
`-- riscv64-nemu-interpreter-sothen use riscv64-nemu-interpreter-so as reference for GEM5,
export GCB_REF_SO=`realpath build/riscv64-nemu-interpreter-so`git clone https://github.com/OpenXiangShan/riscv-isa-sim.git -b gem5-ref spike
cd spike/difftest && make CPU=XIANGSHANThen use difftest/build/riscv64-spike-so similarly as NEMU.
export GCBV_REF_SO=`realpath difftest/build/riscv64-spike-so`If your machine has a Python with very high version, you may need to install a lower version of Python to avoid some compatibility issues. We recommend to use miniconda to install Python 3.8.
Installation command, copied from official miniconda website
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.shThen add conda to path in ~/.bashrc or ~/.zshrc. Note this will hide the system Python.
# for bash
~/miniconda3/bin/conda init bash
# for zsh
~/miniconda3/bin/conda init zshRestart your terminal, and you should be able to use conda. Then create a Python 3.8 env:
# create env
conda create --name py38 --file $gem5_home/ext/xs_env/gem5-py38.txt
# This is mudatory to avoid conda auto activate base env
conda config --set auto_activate_base falseEach time login, you need to activate the conda env before building GEM5:
conda activate py38In case that you don't like this or it causes problem, to completely remove Python and conda from your PATH, run:
# for bash
conda init bash --reverse
# for zsh
conda init zsh --reversePython not foundThis is often not Python missing, but other problems.
Because the build scripts (and scons) uses a strange way to find Python, see site_scons/gem5_scons/configure.py for more detail.
For example, when building with clang10, I encountered this problem:
Error: Check failed for Python.h header.
Two possible reasons:
1. Python headers are not installed (You can install the package python-dev on Ubuntu and RedHat)
2. SCons is using a wrong C compiler. This can happen if CC has the wrong value.
CC = clangThis is not becaues of Python, but because GCC and clang have different warning suppression flags. To fix it, I apply this path:
git apply ext/xs_env/clang-warning-suppress.patchBut Python complaints are also possible caused by other problems,
For similar errors, check build/RISCV/gem5.build/scons_config.log to get the real error message.
The README for official GEM5 is here: Original README