IPbus for VC707 - SGMII PHY

[ixiotidi]

Hi,

I am trying to port the IPbus fw into a VC707 board, by following the example of the KCU105, but instead of configuring the gig_ethernet_pcs_pma_16_01 core into 1000BASE-X, I am configuring it to SGMII in order to create an GMII to SGMII bridge. Apparently, this is not so straight forward and issues arise with clocking. Is there any example design you may have developed, because by a fast online check it seems that people were able to port IPbus on a Virtex-7 board. Thanks in advace

Kind Regards --Giannis

[dmnewbold]

I don't think we have a stock SGMII design for V7, but there is an example for V6 in the repository.

What exactly is the issue you see? It should be as simple as a change of configuration of the PHY core.

[ixiotidi]

Hi,

thanks for the reply, well the V6 example design follows a different approach since it configures the Ethernet input with a Xilinx Hard-Ethernet Wrapper core, which has SGMII I/O on default. This is not present in the V7 and vivado therefore you need to connect the pcs/pma or sgmii core with the temac (configured as internal). The clock you need to have the pcs/pma core running is a 125 MHz clock which has to arrive faster to the core than the free-running ethernet clock on the board (because it's used in the mmcm for the ipb_clk). Therefore you need either to convert the sysclk to 125 MHz (with an mmcm) which causes an issue with the CDC in vivado or change the mmcm to accept as an input the 200MHz clock and convert it to the ipb_clk, which for some reason (I am trying to investigate) it generated the bitstream normally but when I try to write or read a register with the software the connection times out.

Cheers --G.

[dmnewbold]

Not sure I understand. All you should need to do is to take the basex V7 design and change the PCS/PMA core to use SGMII protocol instead of 1000base-x. You can do this in the core configuration GUI, and then alter the .dep files to refer to a new core description file.

The 125MHz free-running clock is typically generated from tapping off the transceiver reference clock, and it's there before the FPGA global reset is released, independent of whether ethernet is running. The board clock is usually only used to drive some reset state machine, though of course you can use it to generate the ipbus clock, if you want.

The CDC might occur if you changed the clock structure but didn't update the TCL file to make the correct derived clocks asynchronous.

[ixiotidi]

Hi,

thanks for the immediate responses. I assume you mean the K7 basex firmware (since there is not V7 example design at all) ? Ok, I will try it and report the results on it.

G.

[dmnewbold]

There's no difference between Kintex and Virtex for this purpose, the architecture is identical.

Let me know what happens.

[dmnewbold]

I made a branch of the repo (enhancement/46) and attempted to put a design eth_7s_sgmii.vhd into it, with a corresponding core. Almost certainly it won't work, but might make a useful starting point.

[ixiotidi]

Hi,

thanks a lot, I checked the design you uploaded and seems similar to the way I configured the cores, the only thing that is different is the speedis10100 and speedis100 signals which in the pcs/pma core have to be present and connected to the temac output ones, but even if you add those this shouldn't be an issue. Ok I finished the design and assigned all the pins in the fpga. (I found that there are a lot of signals with high severity in the timing check, but the same happens also for the KC705 design), unfortunately it doesn't work again... I get the bit-file but when I try to read/write a register with the software the connection times out. I am sure that the software is working properly because I checked it with the kc705 board (I had to borrow) and the setup worked fine there. Would it make sense if I send you my design ?

[dmnewbold]

I can take a look, but probably not for a couple of days. I don't have any board with an SGMII interface to test on, unfortunately.

The quickest way to debug in hardware is probably to take Chipscope and monitor the signals out of the ethernet block. If you see packets coming in at that level, it's an ipbus issue. If you don't, then it's an ethernet problem.

One common issue with SGMII is auto-negotiation with the PHY. Is your switch / NIC telling you that it successfully achieved negotiation with the PHY on the VC707? Is the PHY LED lit on the board side? Can you ping the board?

[ixiotidi]

Well, since you have no hw it doesn't make much sense of sending you the fw (unless you thing there are tests you can run).

I integrated the ILA core in Vivado for the ethernet probes and I found that the ethernet modules is not getting any data from the ethernet socket. So I think the problem is the connection between the MAC and the pcs/pma core. For instance the speedis100 and speedis10100 do not have to be connected because the refer to different configuration of the MAC and the pcs/pma core.

Well the board Leds close to the ethernet socket are on even when the firmware is not implemented which leads me to the outcome that it achieved negotiation but the funny thing is that I am not able to ping into the board.

Do you have further suggestions ??

Thanks

[dmnewbold]

LEDs on means the PHY has negotiated with the other end; this is normal.

OK, here are some suggestions:

• There is a port on the PCS/PMA core called 'status vector'. Can you attach your ILA to that and see what it says?

• Can you double-check your constraints file and the pin report from vivado, to make sure that you have the MGT refclk and tx / rx on the right pins?

• Are you resetting the PHY at FPGA startup? This is sometimes necessary to start negotiation with the PCS/PMA core. Make sure you respect the minimum reset pulse width, this has proven to be an issue on other boards.

[Ed-Flaherty]

Hi, I had a similar problem in creating RGMII for Virtex7. Here's what I did:

1) I configured the Xilinx MAC IP core for RGMII and then generated the example design. The Example design has a traffic generator in it so that you can test it using Wireshark.

2) I then removed the traffic generator from the design, and replaced it with a FIFO. The input to the FIFO was the MAC Rx path and the output was connected to the MAC Tx path where I just removed the traffic generator. In this configuration, it re-transmits anything it receives. I tested this by using the Ostinato traffic generator on the server to transmit traffic to the FPGA. I also used Wireshark to see that everything I sent was in fact echoed back.

3) I then disconnected the FIFO output and reconnected it to the IPBUS Rx input. I then connected the IPBUS Tx output to the TX data port on the MAC.
IPBus porting to RGMII.pdf

That was it. I was able to keep all of the Xilinx RGMII constraints from the example design which made it much easier.

Hope this helps, Ed

[ixiotidi]

Hi,

thanks both for the information, I checked the value of the status vector and apparently all the values are 0 which I think means that the connection, is not set up properly. I checked also the gmii links from the pcs/pma and saw that when I try to pink I receive packages but all of them seem asynch, and therefore nothing is pushed towards the MAC. Now I am trying to configure the two cores again, (maybe I missed something) but I dunno if this is going to have any difference. Ed, thanks a lot of the information about the RGMII, but what is not clear to me is how did you converted the SGMII from the chips input to RGMII for further processing ?? I checked with wireshark my software side and it is transmitting the packages normally, the only issue (I think due to what I mention above) is that nothing is being transmitted back from the FPGA side. Apparently, SGMII conversion to GMII is more tricky than expected...

Cheers --G.

[dmnewbold]

All zeroes probably means the core is in reset or the physical pins are wrong. Did you check the core is actually using the transceiver you think it is? What did you set the config vector and auto-neg vector to?

If you send the code I can take a look.

[Ed-Flaherty]

Sorry, I should have said: My PHY is RGMII (not the VC707). So, I really didn't do anything in the SGMII domain. I generated the MAC core and the Xilinx example design for RGMII.

If you want to do the same, the idea is that you start with whatever format your PHY is, and then build upon that from start to finish (no conversions). The example design gives you the right V7 constraints.

It sounds like you're already close to making it work, so you may not want to start again with this approach unless you feel you're not getting anywhere.

As dmnewbold mentioned, I also had to add a reset for the PHY.

[dpcsankey]

Sorry Ed, my bad there. I forgot that you were RGMII rather than SGMII.

Worth noting that when you get the thing working, first test can be ping (and first traffic will be the ARP request and response prior to the ping).

[ixiotidi]

Hi All,

I am really thankful for the support and the information from everyone. Regarding the questions: I checked the pins and they are the corresponding pins coming for the SGMII input. I obtain the 125MHz clock from the free-running ethernet clock and I am assign the tx-rx pins to the ones in the pin according to the User Guide of VC707. Also, the configuration vectors I am using the default values from the IPbus core (the ones in the eth_7s_1000basex.vhd file). About, the auto-negotiation vector, I generated the core with auto-neg disabled, which was the simplest core configuration I could think about. By, checking the gmii output from the pcs/pma or sgmii core I found that when I try to ping the gmii_rxd data get some values but not the gmii_rx_dv or gmii_rx_er signals show anything (if that makes sense). How do you want me to send you the code ? Do you want the complete project or just the ethernet interface module ?? Also, if we manage to solve it then I think you can add it also in the git hub for further use of other people. Thanks again

Cheers --G.

[ixiotidi]

Hi everybody,

I managed to solve the issue after some time of debugging and looking through the specification document of the VC707 board. The main problem was the auto-negotiation, the Alaska Ultra 88E1111 Physical Layer on the board is configured to auto-negotiation mode, and that had to be taken into account in the PCS/PMA or SGMII core of Xilinx by enabling the auto-negotiation vector. When you follow the instruction on the user guide how to configure the vector then the data flow is happening normally. Thank you everybody for the support.

Cheers --G.