search

lowRISC / ibex

Ibex is a small 32 bit RISC-V CPU core, previously known as zero-riscy.
https://www.lowrisc.org
Apache License 2.0
1.34k stars 520 forks source link

Compile_tb.py generate options not recogised by xrun #2205

Open cern143 opened 4 weeks ago

cern143 commented 4 weeks ago

Observed Behavior

compile_tb_stdstreams.log

Expected Behavior

Steps to reproduce the issue

My Environment

EDA tool and version:

Xcelium 23.09-s001 **Operating system:** Centos 7.9 **Version of the Ibex source code:** 668233699df9ec2a40413e69e0de0a5b10185980
rswarbrick commented 4 weeks ago

To save other people having to download the log file (which is only 3 lines long!), it contains:

xrun: *SE,BDOPT: Unknown option -rpath,/lib.
Running compile_tb command :
xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker -rpath,/lib -lstdc++
rswarbrick commented 4 weeks ago

I think this form of the -rpath argument is coming from _get_iss_pkgconfig_flags in compile_tb.py. Pinging @hcallahan-lowrisc, who might be able to help (given that he wrote the code there)

hcallahan-lowrisc commented 4 weeks ago

Hi @cern143

There has been a few issues with CentOS7 support in the past, so this could possibly be related to that. I can't reproduce this locally right now. (I am using NixOS, but I can also confirm it works fine on Ubuntu). But could I just check a few things with you first before we go down that route...

The xrun invocation I get locally looks as follows:

$ make --keep-going IBEX_CONFIG=opentitan SIMULATOR=xlm ISS=spike ITERATIONS=1 SEED=1 TEST=riscv_arithmetic_basic_test WAVES=0 COV=0

# compile_tb_stdstreams.log
<...>
xrun -64bit -q -f /home/harry/projects/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/harry/projects/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/harry/projects/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/harry/projects/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/lib -I/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/include -I/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/include/softfloat -I/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/include -Wld,-Xlinker,-rpath,/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/lib -lstdc++
<...>

Comparing it to your logfile, this suggests to me that your build has not been able to correctly use pkg-config to resolve the build-time flags pointing at your local spike build. Everywhere my build contains the path /nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0 (the path to my spike build), yours has no path there at all.

Also, strangely the -rpath argument looks to have been formed differently between our systems. Mine resolved to -Wld,-Xlinker,-rpath,<spike_path>/lib, whereas yours resolved to -Wld,-Xlinker -rpath,/lib, not only missing the resolved spike_path but also there is now a space instead of a comma between -Xlinker and -rpath.

To get a bit more information, I'll prepare a branch which prints out a bit more information from the build. In the meantime, could you just confirm a few things.

  1. What your version of pkg-config is?

    [harry@neptune:~/projects/ibex/dv/uvm/core_ibex]$ pkg-config --version
0.29.2

  2. From the build instructions page, you have built spike and setup the PKG_CONFIG_PATH correctly, pointing to a path that exists. e.g.

    [harry@neptune:~/projects/ibex/dv/uvm/core_ibex]$ echo $PKG_CONFIG_PATH | tr : '\n' | grep spike
/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/lib/pkgconfig

Thanks

cern143 commented 4 weeks ago

Hi @hcallahan-lowrisc Thank for your reply. When i check pkg-config version, i got:

 pkg-config --version
0.27.1

and for PKG_CONFIG_PATH:

 echo $PKG_CONFIG_PATH
/usr/lib/pkgconfig

I can notice our environments are different. Is that ok? Best regard

hcallahan-lowrisc commented 4 weeks ago

Okay I see the issue. You need to follow the build instructions page, which describes how to build spike and setup your environment variables appropriately. A part of this is setting PKG_CONFIG_PATH to point towards the spike model you have just built. Without doing this, it is not possible to run the simulations, as we need to link against the spike model to perform cosimulation. Thanks!

cern143 commented 4 weeks ago

Hi @hcallahan-lowrisc, I rebuilt it in a different directory:

echo $PKG_CONFIG_PATH
/opt/lib/pkgconfig

And still get the same result:

xrun: *SE,BDOPT: Unknown option -rpath,/opt/lib.
Running compile_tb command :
xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/opt/lib -I/opt/include -I/opt/include/softfloat -Wld,-Xlinker -rpath,/opt/lib -lstdc++

Im pretty sure i didn't specify the prefix at the first time i built it (which is not a smart move), but .pc files content remain the same. For more information, i also check the version of riscv-isa-sim

 ❯ git rev-parse HEAD
39612f93837122a980395487f55b2d97d29d70c1
❯ git rev-parse --abbrev-ref HEAD
ibex_cosim

Everything seem fine. I also tried to edit compile.py to use -Wld option of xcelium and it seem to work, howerver the default g++ of xcelium can't understand -rpath option:

ake[2]: Entering directory `/home/usr14/ibex/dv/uvm/core_ibex'
building library run.so
g++: error: unrecognized command line option '-rpath'
make[2]: *** [/home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/xcelium.d/run.lnx8664.23.09.d/librun.so] Error 1
make[2]: Leaving directory `/home/usr14/ibex/dv/uvm/core_ibex'

Can you tell me what to do next? Best regard.

hcallahan-lowrisc commented 3 weeks ago

Could you try running the command directly, except with a comma between -Xlinker and -rpath instead of a space? See the message above where I pasted the command generated on my system for an example. I believe the xrun tool is parsing the CLI invocation differently due to this change, and hence g++ is being fed different arguments. This suggests to me that pkg-config on our systems are somehow returning different outputs. I'll try to put together a branch today which prints some logging output to confirm this.

cern143 commented 3 weeks ago

It is still the same problem:

xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker -rpath,/lib -lstdc++
xrun: *SE,BDOPT: Unknown option -rpath,/lib.

I couldn't understand why that doesn't work

hcallahan-lowrisc commented 3 weeks ago

Hey @cern143 Could you try just running this command directly, and let me know what happens?

xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker,-rpath,/lib -lstdc++

(I just copied the command from your message, but changed the space for a comma before the -rpath argument) Thanks

cern143 commented 3 weeks ago

I tried your command @hcallahan-lowrisc, and xrun seem ok with those parameters:

❯ xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker,-rpath,/lib -lstdc++
xrun: *E,FILEMIS: Cannot find the provided file /dv/uvm/core_ibex/common/prim/prim_pkg.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_assert.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_util_pkg.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_count_pkg.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_count.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_secded_pkg.sv.
...

There are more log line but i these are enough to point out the problem, xrun can't find file listed in ibex_dv_cosim_dpi.f

cern143 commented 3 weeks ago

Hey @hcallahan-lowrisc, seem like we fixed it. Just need to edit a line in compile.py (remove space between xlinker and formated string): 

_flags = "'-Xlinker{}'".format(_flags.replace('-Wl,', ','))

and it will be ok:

make IBEX_CONFIG=opentitan SIMULATOR=xlm ISS=spike ITERATIONS=1 SEED=1 TEST=riscv_arithmetic_basic_test WAVES=0 COV=0
Build metadata already exists, not recreating from scratch.
Building RTL testbench
Generating core configuration file
Building randomized test generator
Running randomized test generator to create assembly file /home/usr14/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/test.S
Compiling riscvdv test assembly to create binary at /home/usr14/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/test.bin
Running RTL simulation at /home/usr14/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1
Collecting simulation results and checking logs of testcase at /home/usr14/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/trr.yaml
Collecting up results of tests into report regr.log
100.00% PASS 1 PASSED, 0 FAILED
hcallahan-lowrisc commented 2 weeks ago

Thanks for letting me know @cern143, glad to hear its working now. I've just opened #2209, which should fix this a bit more generally. Would you mind testing it out and confirming that it works for you?