QSFP Simulation Fails for X400

[raefYoussef]

Issue Description

I'm trying to run a simulation of the X400 QSFP. I found an extensive testbench called 'x4xx_qsfp_wrapper' but it fails. The error I get is the following:

ERROR: [VRFC 10-2063] Module not found while processing module instance [

/uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/eth_100g_bd/eth_100g_bd/ip/eth_100g_bd_cmac_usplus_0_0/eth_100g_bd_cmac_usplus_0_0.v:695] ERROR: [VRFC 10-2063] Module not found while processing module instance [/uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/eth_100g_bd/eth_100g_bd/sim/eth_100g_bd.v:499] ERROR: [VRFC 10-2063] Module not found while processing module instance [/init/uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/eth_100g_bd/eth_100g_bd/sim/eth_100g_bd.v:501] ERROR: [VRFC 10-2865] module 'x4xx_qsfp_wrapper_tb(TEST_NAME="100GbE_128S",PROTOCOL0=5,CHDR_W=128)' ignored due to previous errors [/init/uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/x4xx_qsfp_wrapper_tb.sv:23] ERROR: [VRFC 10-2865] interface 'AxiIf_v(DATA_WIDTH=128,ADDR_WIDTH=49)' ignored due to previous errors [/init/uhd4/fpga/usrp3/lib/axi4_sv/AxiIf.sv:237] ERROR: [VRFC 10-2865] interface 'AxiLiteIf_v(DATA_WIDTH=32,ADDR_WIDTH=40)' ignored due to previous errors [/init/uhd4/fpga/usrp3/lib/axi4lite_sv/AxiLiteIf.sv:211] ERROR: [VRFC 10-2865] module 'x4xx_qsfp_wrapper_temp(PROTOCOL0=5,CHDR_W=512)' ignored due to previous errors [/init/uhd4/fpga/usrp3/top/x400/x4xx_qsfp_wrapper_temp.sv:27] ERROR: [VRFC 10-2865] interface 'AxiIf(DATA_WIDTH=128,ADDR_WIDTH=49)' ignored due to previous errors [/init/uhd4/fpga/usrp3/lib/axi4_sv/AxiIf.sv:25] ERROR: [XSIM 43-3322] Static elaboration of top level Verilog design unit(s) in library work failed.

Setup Details

UHD Version: 4.4.0.0 Git Hash: 106afff8890a4161e00bcebca0ed673890d2fdc0 OS: Ubuntu 20.04.6 LTS Hardware: X400

Expected Behavior

I expect the simulation to run and get either pass/fail similar to unit tests.

Actual Behavior

Makefile crashes

Steps to reproduce the problem

[Optional] Apply diff file included to fix some bugs.
sim_diff.txt Run 'source 'uhd4/fpga/usrp3/top/x400/setupenv.sh' Navigate to 'uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper' Run 'make xsim'

Additional Information

I'm not sure if this is the correct file to simulate the QSFP. Please direct me to the correct way if it isn't.

[wordimont]

Simulation of this testbench is not supported under XSim. We have explicitly masked this test from our automated XSim CI:

https://github.com/EttusResearch/uhddev/blob/5ba4cd85a1ee2823ccfc0086752fe233eaa4a075/fpga/usrp3/tools/utils/testbenches.excludes#L16

We do however run the testbench in Modelsim as part of our CI automated tests.

I don't know the exact reason why it's not supported in XSim, but in the past XSim has been very buggy when it comes to simulation and sometimes it's not worth the effort to try to work around the simulation bugs. We'll retry this in a future Vivado version since their simulator seems to improve with each release.

[raefYoussef]

Thanks for the response worhimont. May I ask which Modelsim version you guys are using? Do you also require Questa or is Modelsim sufficient?

[wordimont]

Questa is not required. The CI uses ModelSim SE 2020.4. I recently used SE 2021.3. ModelSim DE should also work fine. I know some people have also used PE, but that's Windows only so it doesn't play as nicely with our Linux build flow. I don't think the version number matters much.

[raefYoussef]

I switched to ModelSim SE 2020.4 (Compiler 2020.10) but I'm still running into issues using both 'modelsim' and 'vsim' flags.

'modelsim' errors: Error: /uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/axi_interconnect_eth_bd/axi_interconnect_eth_bd/ip/axi_interconnect_eth_bd_xbar_0/sim/axi_interconnect_eth_bd_xbar_0.v(247): Module 'axi_crossbar_v2_1_25_axi_crossbar' is not defined. Error: /uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/axi_eth_dma_bd/axi_eth_dma_bd/sim/axi_eth_dma_bd.v(324): Module 'axi_eth_dma_bd_axi_eth_dma_0' is not defined. Error: /uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/axi_eth_dma_bd/axi_eth_dma_bd/sim/axi_eth_dma_bd.v(416): Module 'axi_eth_dma_bd_smartconnect_eth_dma_0' is not defined. Error: /uhd4/fpga/usrp3/top/x400/ip/axi_interconnect_dma_bd/axi_interconnect_dma.sv(30): Module 'axi_interconnect_dma_bd' is not defined. Error: /uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/eth_100g_bd/eth_100g_bd/ip/eth_100g_bd_cmac_usplus_0_0/eth_100g_bd_cmac_usplus_0_0.v(766): Module 'eth_100g_bd_cmac_usplus_0_0_wrapper' is not defined. Error: /uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/eth_100g_bd/eth_100g_bd/sim/eth_100g_bd.v(499): Module 'eth_100g_bd_tie_loopback_0' is not defined. Error: /uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/eth_100g_bd/eth_100g_bd/sim/eth_100g_bd.v(501): Module 'eth_100g_bd_tie_zero_0' is not defined. Error: /uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/xge_pcs_pma/xxv_ethernet_v4_0_0/xge_pcs_pma_wrapper.v(481): Module 'xge_pcs_pma_ultrascale_tx_userclk' is not defined. Error: /uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/xge_pcs_pma/xxv_ethernet_v4_0_0/xge_pcs_pma_wrapper.v(497): Module 'xge_pcs_pma_ultrascale_rx_userclk' is not defined. Error: /uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/xge_pcs_pma/xxv_ethernet_v4_0_0/xge_pcs_pma_wrapper.v(516): Module 'xge_pcs_pma_gt' is not defined. ** Error: /uhd4/fpga/usrp3/top/x400/sim/x4xx_qsfp_wrapper/build-ip/xczu28drffvg1517-1e/xge_pcs_pma/xxv_ethernet_v4_0_0/xge_pcs_pma_wrapper.v(771): Module 'xge_pcs_pma_top' is not defined.

'vsim' errors: Error: ../../../../../build-ip/xczu28drffvg1517-1e/axi_eth_dma_bd/axi_eth_dma_bd/sim/axi_eth_dma_bd.v(324): Module 'axi_eth_dma_bd_axi_eth_dma_0' is not defined. Error: ../../../../../build-ip/xczu28drffvg1517-1e/axi_eth_dma_bd/axi_eth_dma_bd/sim/axi_eth_dma_bd.v(416): Module 'axi_eth_dma_bd_smartconnect_eth_dma_0' is not defined. Error: ../../../../../build-ip/xczu28drffvg1517-1e/eth_100g_bd/eth_100g_bd/ip/eth_100g_bd_cmac_usplus_0_0/eth_100g_bd_cmac_usplus_0_0.v(766): Module 'eth_100g_bd_cmac_usplus_0_0_wrapper' is not defined. Error: ../../../../../build-ip/xczu28drffvg1517-1e/eth_100g_bd/eth_100g_bd/sim/eth_100g_bd.v(499): Module 'eth_100g_bd_tie_loopback_0' is not defined. ** Error: ../../../../../build-ip/xczu28drffvg1517-1e/eth_100g_bd/eth_100g_bd/sim/eth_100g_bd.v(501): Module 'eth_100g_bd_tie_zero_0' is not defined. ERROR: [USF-ModelSim-72] Failed to launch simulate.sh: ERROR: [Common 17-39] 'send_msg_id' failed due to earlier errors.

[wordimont]

Looks like you don't have the Xilinx simulation libraries compiled for ModelSim.

https://support.xilinx.com/s/article/53678

Please note also that due to a bug in Vivado, you might also need to use "METHOD 2" to install any patches being used. Otherwise Vivado will fail to compile some libraries.

https://support.xilinx.com/s/article/72095

[raefYoussef]

Thank you! I compiled the IP using this guide: https://files.ettus.com/manual/md_usrp3_sim_running_testbenches.html. The 'build_simlibs' command seemed to run error-free. Are you saying that having a Vivado patch can mess up with the simulation?

[wordimont]

If you have the patch installed using "method 1" then Vivado will skip some IP simulation libraries when running build_simlibs. So you will still run into the same error where ModelSim can find them, because they weren't actually compiled. So make sure you're using "method 2" to install Vivado patches as described in that article before you compile the simulation libraries.

[raefYoussef]

Thanks for the tip, that worked! However, the wildcard in the various Makefile do not expand properly. Not sure why but had to replace the expansions manually.