Closed vmayoral closed 1 year ago
Bumped into a build error while attempting to create IP for ZCU102:
INFO: [IP_Flow 19-2313] Loaded Vivado IP repository '/tools/Xilinx/Vivado/2022.1/data/ip'.
WARNING: [IP_Flow 19-3664] IP 'eth_xcvr_gt_full' generated file not found '/home/victor/verilog-ethernet/example/ZCU102/fpga/fpga/fpga.gen/sources_1/ip/eth_xcvr_gt_full/eth_xcvr_gt_full.dcp'. Please regenerate to continue.
WARNING: [IP_Flow 19-3664] IP 'eth_xcvr_gt_full' generated file not found '/home/victor/verilog-ethernet/example/ZCU102/fpga/fpga/fpga.gen/sources_1/ip/eth_xcvr_gt_full/eth_xcvr_gt_full_stub.v'. Please regenerate to continue.
WARNING: [IP_Flow 19-3664] IP 'eth_xcvr_gt_full' generated file not found '/home/victor/verilog-ethernet/example/ZCU102/fpga/fpga/fpga.gen/sources_1/ip/eth_xcvr_gt_full/eth_xcvr_gt_full_stub.vhdl'. Please regenerate to continue.
WARNING: [IP_Flow 19-3664] IP 'eth_xcvr_gt_full' generated file not found '/home/victor/verilog-ethernet/example/ZCU102/fpga/fpga/fpga.gen/sources_1/ip/eth_xcvr_gt_full/eth_xcvr_gt_full_sim_netlist.v'. Please regenerate to continue.
WARNING: [IP_Flow 19-3664] IP 'eth_xcvr_gt_full' generated file not found '/home/victor/verilog-ethernet/example/ZCU102/fpga/fpga/fpga.gen/sources_1/ip/eth_xcvr_gt_full/eth_xcvr_gt_full_sim_netlist.vhdl'. Please regenerate to continue.
WARNING: [IP_Flow 19-3664] IP 'eth_xcvr_gt_channel' generated file not found '/home/victor/verilog-ethernet/example/ZCU102/fpga/fpga/fpga.gen/sources_1/ip/eth_xcvr_gt_channel/eth_xcvr_gt_channel.dcp'. Please regenerate to continue.
WARNING: [IP_Flow 19-3664] IP 'eth_xcvr_gt_channel' generated file not found '/home/victor/verilog-ethernet/example/ZCU102/fpga/fpga/fpga.gen/sources_1/ip/eth_xcvr_gt_channel/eth_xcvr_gt_channel_stub.v'. Please regenerate to continue.
WARNING: [IP_Flow 19-3664] IP 'eth_xcvr_gt_channel' generated file not found '/home/victor/verilog-ethernet/example/ZCU102/fpga/fpga/fpga.gen/sources_1/ip/eth_xcvr_gt_channel/eth_xcvr_gt_channel_stub.vhdl'. Please regenerate to continue.
WARNING: [IP_Flow 19-3664] IP 'eth_xcvr_gt_channel' generated file not found '/home/victor/verilog-ethernet/example/ZCU102/fpga/fpga/fpga.gen/sources_1/ip/eth_xcvr_gt_channel/eth_xcvr_gt_channel_sim_netlist.v'. Please regenerate to continue.
WARNING: [IP_Flow 19-3664] IP 'eth_xcvr_gt_channel' generated file not found '/home/victor/verilog-ethernet/example/ZCU102/fpga/fpga/fpga.gen/sources_1/ip/eth_xcvr_gt_channel/eth_xcvr_gt_channel_sim_netlist.vhdl'. Please regenerate to continue.
# reset_run impl_1
# launch_runs -jobs 4 impl_1
ERROR: [Common 17-70] Application Exception: Failed to launch run 'impl_1' due to failures in the following run(s):
eth_xcvr_gt_full_synth_1
eth_xcvr_gt_channel_synth_1
These failed run(s) need to be reset prior to launching 'impl_1' again.
INFO: [Common 17-206] Exiting Vivado at Mon Feb 13 17:22:56 2023...
make[1]: *** [../common/vivado.mk:115: fpga.runs/impl_1/fpga_routed.dcp] Error 1
make[1]: Leaving directory '/home/victor/verilog-ethernet/example/ZCU102/fpga/fpga'
make: *** [Makefile:14: fpga] Error 2
I looked further into the logs as adviced and here's what I found:
victor@accelerationrobotics:~/verilog-ethernet/example/ZCU102/fpga$ du -a| grep log
4 ./fpga/fpga.runs/eth_xcvr_gt_full_synth_1/runme.log
4 ./fpga/fpga.runs/eth_xcvr_gt_channel_synth_1/runme.log
victor@accelerationrobotics:~/verilog-ethernet/example/ZCU102/fpga$ cat ./fpga/fpga.runs/eth_xcvr_gt_channel_synth_1/runme.log
*** Running vivado
with args -log eth_xcvr_gt_channel.vds -m64 -product Vivado -mode batch -messageDb vivado.pb -notrace -source eth_xcvr_gt_channel.tcl
****** Vivado v2022.1 (64-bit)
**** SW Build 3526262 on Mon Apr 18 15:47:01 MDT 2022
**** IP Build 3524634 on Mon Apr 18 20:55:01 MDT 2022
** Copyright 1986-2022 Xilinx, Inc. All Rights Reserved.
source eth_xcvr_gt_channel.tcl -notrace
Command: synth_design -top eth_xcvr_gt_channel -part xczu9eg-ffvb1156-2-e -incremental_mode off -mode out_of_context
Starting synth_design
Attempting to get a license for feature 'Synthesis' and/or device 'xczu9eg'
WARNING: [Common 17-348] Failed to get the license for feature 'Synthesis' and/or device 'xczu9eg'. Explanation: The license feature Synthesis could not be found.
Resolution: Check the status of your licenses in the Vivado License Manager. For debug help search Xilinx Support for "Licensing FAQ".
0 Infos, 1 Warnings, 0 Critical Warnings and 1 Errors encountered.
synth_design failed
ERROR: [Common 17-345] A valid license was not found for feature 'Synthesis' and/or device 'xczu9eg'. Please run the Vivado License Manager for assistance in determining
which features and devices are licensed for your system.
Resolution: Check the status of your licenses in the Vivado License Manager. For debug help search Xilinx Support for "Licensing FAQ". If you are using a license server, verify that the license server is up and running a version of the xilinx daemon that is compatible with the version of Xilinx software that you are using. Note: Vivado 2021.1 and later versions require upgrading your license server tools to the Flex 11.17.2.0 versions. Please confirm with your license admin that the correct version of the license server tools are installed.
INFO: [Common 17-206] Exiting Vivado at Mon Feb 13 20:31:35 2023...
It seems a license issue. Will re-try things once the license is set up appropriately.
Ah yep, looks like Vivado needs a license for that part. The Kira SoMs do not require a license, correct?
Ah yep, looks like Vivado needs a license for that part. The Kira SoMs do not require a license, correct?
It appears to be the case. Same as with the ZCU102, license is required. For completeness, ZCU102 built successfully after setting up a license:
Took a look at the KR260 schematic; it looks like porting the design from the ZCU102 should mainly be a matter of updating the pin constraints and deleting 3 of the SFP+ channels. Although, it looks like there isn't an external 125 MHz clock (the GTH transceivers need a free-running clock of some sort) so maybe you can borrow the setup I have for the VCU1525 to take the transceiver ref clock and convert that to 125 MHz, see https://github.com/alexforencich/verilog-ethernet/blob/master/example/VCU1525/fpga_10g/rtl/fpga.v#L364 . I suppose using a PLL to convert to 125 MHz is not strictly necessary, the GTH transceivers would probably work fine with using the 156.25 MHz ref clock directly, but I like putting that PLL in so things are consistent. You'll need to adjust the PLL settings for 156.25 MHz as the VCU1525 uses 161.1328125 MHz. Looks like 156.25 * 8/10 = 125, so D = 1 and M = 8 for Fvco 1250 MHz, output divide = 10 to get 125 MHz, and set the input period to 6.4.
Also, it looks like the 156.25 MHz oscillator is fixed, which is a little unfortunate. Fine for PTP, but something like sync-e or white rabbit would need an adjustable oscillator like an si570 or PLL chip with ppb-resolution adjustment capability.
Actually, on taking another look at the KR260 schematic, it looks like there are 25 MHz clocks provided to the PL on two different pins (see page 16 on the sch). So, you could feed that into a PLL and covert it to 125 MHz, instead of using the 156.25 MHz from the ref osc. Either method would work, but if you want to make a 1G design it might make more sense to use one of those 25 MHz clocks instead of the GTH ref clock.
All right, so spent the last few weeks understanding and porting the code for the KR260. The branch that currently reflects my latest status is https://github.com/vmayoral/verilog-ethernet/tree/kr260. I've gone through quite a few iterations while using the following resources:
After jumping over quite a few hurdles, I bumped into DRC errors about a week ago and have been fighting with them ever since. Dropping my struggles in here for completeness:
Struggle 1
Complete error dump at https://gist.github.com/vmayoral/fe294556fe89ee18737ac706ee07c100
Struggle 2
If I bypass the previous error root cause, I bump into the following error which affects most of the definitions in the XDC:
Complete error dump at https://gist.github.com/vmayoral/b31084a9d7f93812a24b19e98e62be44
ERROR: [DRC NSTD-1] Unspecified I/O Standard: 12 out of 18 logical ports use I/O standard (IOSTANDARD) value 'DEFAULT', instead of a user assigned specific value. This may cause I/O contention or incompatibility with the board power or connectivity affecting performance, signal integrity or in extreme cases cause damage to the device or the components to which it is connected. To correct this violation, specify all I/O standards. This design will fail to generate a bitstream unless all logical ports have a user specified I/O standard value defined. To allow bitstream creation with unspecified I/O standard values (not recommended), use this command: set_property SEVERITY {Warning} [get_drc_checks NSTD-1]. NOTE: When using the Vivado Runs infrastructure (e.g. launch_runs Tcl command), add this command to a .tcl file and add that file as a pre-hook for write_bitstream step for the implementation run. Problem ports: led[7:0], clk_125mhz_n, clk_125mhz_p, reset, and sfp0_tx_disable_b. ERROR: [DRC UCIO-1] Unconstrained Logical Port: 13 out of 18 logical ports have no user assigned specific location constraint (LOC). This may cause I/O contention or incompatibility with the board power or connectivity affecting performance, signal integrity or in extreme cases cause damage to the device or the components to which it is connected. To correct this violation, specify all pin locations. This design will fail to generate a bitstream unless all logical ports have a user specified site LOC constraint defined. To allow bitstream creation with unspecified pin locations (not recommended), use this command: set_property SEVERITY {Warning} [get_drc_checks UCIO-1]. NOTE: When using the Vivado Runs infrastructure (e.g. launch_runs Tcl command), add this command to a .tcl file and add that file as a pre-hook for write_bitstream step for the implementation run. Problem ports: led[7:0], clk_125mhz_n, clk_125mhz_p, reset, sfp_mgt_refclk_0_n, and sfp_mgt_refclk_0_p.
Looks like maybe some of these signals should be removed from the top-level verilog file, and the ones that are necessary should get LOC and IOSTANDARD constraints.
ERROR: [DRC BIVB-1] Bank IO standard Support: Bank 66 has incompatible IO(s) because: The LVCMOS33 I/O standard is not supported for banks of type High Performance. Move the following ports or change their properties: sfp0_tx_disable_b ERROR: [DRC PLHDIO-5] HDIO DRC Checks: The following IO terminals need to be placed in HIGH_DENSITY IO banks (based on their IO standards), but they are incorrectly locked to non-HIGH_DENSITY IO banks. Please review and update the LOC constraints: sfp0_tx_disable_b
From looking at the schematic, this one makes no sense at all. The pin is pulled up to +3v3, but it looks like it's wired to a pin with Vccio of 1.8v, without any level shifting? That looks like a rather serious board design mistake, tbh. But if Vccio is 1.8, then use LVCMOS18.
CRITICAL WARNING: [DRC AVAL-326] Hard_block_must_have_LOC: The hard block IBUFDS_GTE4 cell ibufds_gte4_sfp_mgt_refclk_0_inst is missing a valid LOC constraint for placement assignment, normally supplied by IP generation or manually assigned using the LOC property. Unguided placement of this block may cause problems in routing or other issues. Please check your design and set a valid LOC for this block to avoid these problems.
Looks like this didn't get any LOC constraints.
Thanks for all of the above @alexforencich,
I re-iterated based on this (and the rest of our discussion, so thanks!) resulting in:
sfp0_tx_disable_b
pulled up to +3v3, but it looks like it's wired to a pin with Vccio of 1.8V without level shifting. Using LVCMOS18
removed the error while building https://github.com/vmayoral/verilog-ethernet/commit/f2d3f071ea1a07b77070b29014d354d387c40761Final branch available at https://github.com/vmayoral/verilog-ethernet/tree/kr260. My expectation was for things to build just fine this time but still bumping into the following two errors:
ERROR: [DRC NSTD-1] Unspecified I/O Standard: 4 out of 11 logical ports use I/O standard (IOSTANDARD) value 'DEFAULT', instead of a user assigned specific value. This may cause I/O contention or incompatibility with the board power or connectivity affecting performance, signal integrity or in extreme cases cause damage to the device or the components to which it is connected. To correct this violation, specify all I/O standards. This design will fail to generate a bitstream unless all logical ports have a user specified I/O standard value defined. To allow bitstream creation with unspecified I/O standard values (not recommended), use this command: set_property SEVERITY {Warning} [get_drc_checks NSTD-1]. NOTE: When using the Vivado Runs infrastructure (e.g. launch_runs Tcl command), add this command to a .tcl file and add that file as a pre-hook for write_bitstream step for the implementation run. Problem ports: led[1:0], clk_25mhz_ref, and reset.
ERROR: [DRC UCIO-1] Unconstrained Logical Port: 6 out of 11 logical ports have no user assigned specific location constraint (LOC). This may cause I/O contention or incompatibility with the board power or connectivity affecting performance, signal integrity or in extreme cases cause damage to the device or the components to which it is connected. To correct this violation, specify all pin locations. This design will fail to generate a bitstream unless all logical ports have a user specified site LOC constraint defined. To allow bitstream creation with unspecified pin locations (not recommended), use this command: set_property SEVERITY {Warning} [get_drc_checks UCIO-1]. NOTE: When using the Vivado Runs infrastructure (e.g. launch_runs Tcl command), add this command to a .tcl file and add that file as a pre-hook for write_bitstream step for the implementation run. Problem ports: led[1:0], clk_25mhz_ref, reset, sfp_mgt_refclk_0_n, and sfp_mgt_refclk_0_p.
Further analyzing this:
To my uderstanding, the definitions in the fgpa.xdc
file should be enough as they refer to the K26 SOM (which sets the PACKAGE_PIN
for every one them)
All right, managed to address the hurdles above after various iterations. One of the major issues being experienced is that though I had modified the part number in various places, I had overlooked the variable in the Makefile
. So of course I was being driven nuts by pin conflicts. It was a different SoC after all. https://github.com/vmayoral/verilog-ethernet/commit/b084a8bf9e78d79e1b32c3d46e3bceacd41eea71 did it.
Bitstream builds just fine. However, when programming the PL (testing for now without SD card, to avoid any possible interference, will scale from there once functional) and testing the link:
victor@accelerationrobotics:~/verilog-ethernet/example/KR260/fpga/fpga$ sudo ethtool enp3s0
Settings for enp3s0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
This hints there's something wrong with the design, since I can see the link up perfectly fine with the ZCU102 reference design. No CRITICAL WARNING. Looking at the WARNING messages through the logs, the following caught my attention:
./fpga.runs/synth_1/fpga.vds:WARNING: [Synth 8-3917] design fpga has port sfp0_tx_disable_b driven by constant 1
however that doesn't seem to be the issue, as I can see the same being reported and shown in the ZCU102 design, which is known and confirmed to work:
KR260 design | ZCU102 design |
---|---|
![]() |
![]() |
Well, one minor thing that should be fixed, although it should make no difference when using a DAC: the TX disable connection on the ZCU102 is inverted (there is a transistor in line that provides the level translation) and as such has a _b in the name and is tied to 1. But on the KR260, it's a direct connection, so the _b should be dropped and it should be tied to 0.
Now, as for why it's not working, we'll have to work through a few things. No link reported at the link partner definitely rules out a lot of possibilities. Start with the pinout report and the clock summary from the timing report. Make sure that all of the clock frequencies in the timing report are correct, and that everything is actually using the correct pins.
ummmm, this is likely a problem:
Good news first, managed to get it working ✅ for the AMD KR260
! There's no a new reference design example for this board. Submitted a cleaned PR for your review @alexforencich at https://github.com/alexforencich/verilog-ethernet/pull/150.
As for the last struggles, It was indeed related to (thanks a lot!):
ummmm, this is likely a problem:
After fixing that to a logic 0
, worked like a charm. Reference echo back design example works like a charm 👍 💥 .
Now, into the not so exciting news. While further testing the PL with the PS enabled I noticed that:
@giuliocorradi and @jasvinderkhurana, any thoughts in here? Have you guys seen or experienced something similar?
Nice work!
That boot log seems to indicate that something may be wrong with the SD card?
My board (KR260) doesn't seem to be able to boot the PS normally any further. Regardless of whether I power-boot it, or after having programmed known designs to work previously. The PS doesn't fully boot. I inspected the boot process through serial and after a few attempts obtained the following:
That boot log seems to indicate that something may be wrong with the SD card?
Indeed related to the SD card @alexforencich 👍. KR260's JTAG and microSD conflict when used together. Avoiding JTAG usage while booting did the trick.
Following from that, I flashed the FPGA from the PS manually. This gets the bitstream into the PL appropriately and the IP to operate as expected (✅ confirmed 10G reference example works while PS is active), however, it leaves the PS unusable. After programming the PL from the PS the following happens:
Closing this ticket as the KR260 port was long ago validated. See https://github.com/alexforencich/verilog-ethernet/pull/150 for the upstream contribution attempt.
(This first comment will be updated as the porting efforts evolve and will summarize results so that others can benefit from it.)
Hi @alexforencich and all,
First off, congrats for an impresive piece. The project is not only well maintained but has grown a significant community around it. Very impressive. Congrats and kudos to @alexforencich. I'm currently involved into a project considering to port Corundum into the AMD KR260. This board leverages an AMD Zynq UltraScale+ SoC and exposes various Ethernet (Gb RGMII PL and PS) interfaces and an SFP+ connector which makes it a pretty target for benchmarking.
For now, our endeavour aims to bring ethernet-verilog up to speed into the KR260 and produce some latency performance benchmarking comparisons by leveraging a low overhead and real-time capable tracer (we'll be using LTTng on the PS side, and Vitis Profiler on the PL side).
By doing so, we hope to contribute to this project by
1)
bringing a usable port of ethernet-verilog (Corundum) for a popular FPGA SoC SOM,2)
provide a fair and reproducible latency benchmark that compares UDP/IP networking performance (L1-L2-L3-L4) across various setups.After reading through [^1], it appears that creating example designs for verilog-ethernet is the first recommended step. This ticket will thereby track progress on this regard, porting verilog-ethernet to KR260. The list below aims to lay out some of the milestones neccessary to achieve the porting goals.
Sources considered
- Source code: https://github.com/alexforencich/verilog-ethernet - Paper https://cseweb.ucsd.edu/~snoeren/papers/corundum-fccm20.pdf - Future of corundum https://github.com/corundum/corundum/wiki/Future-of-Corundum - Corundum overview https://docs.corundum.io/en/latest/modules/overview.html - Corundum porting guide https://docs.corundum.io/en/latest/porting.html - Corundum operations, describes reception and transmission operations https://docs.corundum.io/en/latest/operations.html (:warning: useful yet complex to follow) - [K26 SOM XDC file](https://www.xilinx.com/member/forms/download/design-license.html?cid=29e0261a-9532-4a47-bb06-38c83bbbb8c0&filename=xtp685-kria-k26-som-xdc.zip)Related issues and past experiences
- https://github.com/alexforencich/verilog-ethernet/issues/16 - https://github.com/alexforencich/verilog-ethernet/issues/106 - https://github.com/alexforencich/verilog-ethernet/issues/64Environment used
2022.1
master
Contributions
Hardware used
Milestones:
10G BASE-R PHY IP
example10G BASE-R PHY IP core
)10G BASE-R PHY
10G BASE-R PHY
example to the SFP+ connectorBring up a(skipped, focusing instead on the RGMII PHYs)1G PHY
example to the SFP+ connector1G MAC
example into one of the PL Ethernet PHYs10G BASE-R PHY
example to the SFP+ connectorBenchmark(skipped, focusing instead on the RGMII PHYs)1G PHY
example to the SFP+ connector1G MAC
example into one of the PL Ethernet PHYs[^1]: Corundum porting guide https://docs.corundum.io/en/latest/porting.html