Access to the ha1588 fifos from the driver

[mtetrault]

I've successfully used the driver with the ha1588 hdl core and embedded Zynq-Linux, thanks! I would now like to also use the ha1588 fifos in the hw core through the driver, which I believe is not yet in the driver (registers 0x40-0x5C and 0x60-0x7C). Is this planned for addition in the driver? My hope is to have it work with standard PTP daemons such as ptp4l.

If there are no current plans, what would be the recommended path for this? Or suggested driver templates to be inspired from?

For minimal compatibility, I can probably hack into the ptp4l code (did a little already), but if I can have it through the driver, either through existing code or a contribution from my end, it would likely be cleaner and I could commit here.

[btgoodwin]

Good to hear!

At this time, we have not discussed adding the FIFOs for the TSU since in our case we didn't need it. The 1/2.5G AXI Ethernet Subsystem handled it for us once we got around the Vivado automatic configuration issues (enable CONFIG_XILINX_AXI_EMAC_HWTSTAMP, etc. in the kernel). With both drivers active, we were able to use an unmodified ptp4l v2.0 without having to specify the specific ethernet interface or PTP clock to use (though that is how we initially ran ptp4l).

Can you give more specifics from your design?

Do you have the PTP portion of this driver enabled as well as the RTC (i.e., do you see a /dev/rtc0 and /dev/ptp0)? It's the default per the ha1588.cfg; you should see CONFIG_PTP_1588_CLOCK enabled in your kernel .config.

Cheers,

Thomas

[mtetrault]

Hi Thomas,

Thanks for the info.

Our design is based on the Zynq-7000, with a second eth port connected to the PL. The 1/2.5G AXI Ethernet subsystem wizard greys out the PTP, so it doesn't seem available. But beyond that, our goal is to sync the PTP clock to an existing system-wide timer running at 100 MHz.

PTP normally runs on the 125 MHz reference clock, correct? In that case, I could maybe derive the 125 from the ref 100 MHz, but I'd rather have the PTP and our project-specific reference counter run in lock- step. The ha1588 core seems to be quite flexible for this.

For the driver, both portions of the driver are enabled. Temporarily, I hacked ptp4l to use /dev/ptp0 to read the PTP time instead of kernel clock in software timestamping mode, but I'd like to push it to use hw mode and avoid maintaining this hacked version.

What do you mean by "with both drivers active"? If the AXI core's ptp is enabled, then shouldn't the xilinx eth driver take care of everything?

Thanks!

On Tue, 2019-08-06 at 04:41 -0700, Thomas Goodwin wrote:

Good to hear! At this time, we have not discussed adding the FIFOs for the TSU since in our case we didn't need it. The 1/2.5G AXI Ethernet Subsystem handled it for us once we got around the Vivado automatic configuration issues (enable CONFIG_XILINX_AXI_EMAC_HWTSTAMP, etc. in the kernel). With both drivers active, we were able to use an unmodified ptp4l v2.0 without having to specify the specific ethernet interface or PTP clock to use (though that is how we initially ran ptp4l). Can you give more specifics from your design? Do you have the PTP portion of this driver enabled as well as the RTC (i.e., do you see a /dev/rtc0 and /dev/ptp0)? It's the default per the ha1588.cfg; you should see CONFIG_PTP_1588_CLOCK enabled in your kernel .config. Cheers, Thomas — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[btgoodwin]

If I'm understanding you right, you selected the interface speed as 1000BASE-X and the wizard still does not show enabling 1588-2008 as an option? What version of Vivado?

FWIW, we ran our PTP only at 40 MHz because of other related logic. Just be sure your device tree period value is set correctly if you decide to go that route (pseudo-code is in the blog post at the bottom).

Yes, if the core is configured for 1588-2008 support and that part of the Xilinx driver is enabled (previous post, CONFIG_XILINX_AXI_EMAC_HWTSTAMP) you should see hardware time stamping work.

[mtetrault]

1- Correct. In the AXI 1.0/2.5G Ethernet Subsystem IP (v7.1) customization GUI, the IEEE 1588 option is greyed out for my device. I tried changing some of the other options to see if it would allow it, to no avail. I'm using Vivado 2018.3. Note, this is not the PCS/PMA core.

I also remember several documentations saying that IEEE 1588 is only supported on Zynq ultra-scale. I know the Zynq-7000 GEM has faulty 1588, hence the lock-out, but I'm surprised they locked it out of the PL cores as well. Maybe easier to maintain...

Other design note : I'm using the Zynq as PTP Master.

2- Device-tree: yup, I found the device-tree entry for that and set it for 100 MHz in my case.

3- I'll have to try that. This might take a bit of time since I don't have the development board for the Ethernet expander with me right now (to have a PHY on the PL side). I re-compiled the kernel with the extra flag and did not get any error.

Question: with this modification, does the Xilinx driver create another /dev/ptpX separate from the ha1588 one? I would then redirect to the correct one using ptp4l configuration or command line arguments. If not, is the fifo access possible because the ha1588 uses standardized registry mapping and offsets?

Question 2: any idea if this would also work with the Zynq GEM + GMII2RGMII core module? They seem to now share the same driver code. If not, it's not an issue to use the AXI core instead, plenty of logic available.

Thanks

On Tue, 2019-08-06 at 09:34 -0700, Thomas Goodwin wrote:

If I'm understanding you right, you selected the interface speed as 1000BASE-X and the wizard still does not show enabling 1588-2008 as an option? What version of Vivado? FWIW, we ran our PTP only at 40 MHz because of other related logic. Just be sure your device tree period value is set correctly if you decide to go that route (pseudo-code is in the blog post at the bottom). Yes, if the core is configured for 1588-2008 support and that part of the Xilinx driver is enabled (previous post, CONFIG_XILINX_AXI_EMAC_HWTSTAMP) you should see hardware time stamping work. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[btgoodwin]

The note about the Zynq-7000 GEM having a faulty 1588 device is referring to the PS-attached GEMs. Any PL-attached ones support 1588 only if it's implemented in the firmware (e.g., the AXI Ethernet Subsystem we've each been using). We were using a 7030 Picozed on the FMCv2 carrier which has a PL-attached SFP cage. What hardware are you using?

For question 1, I'm not sure I follow. The Xilinx AXI Ethernet driver only registers a /dev/ptpX if you're using their TSN Subsystem as well because at that point it has the whole time stamping unit in it (which is a whole extra set of kernel flags and IP core licenses). The flag we're setting lets the AXI Ethernet driver interact with the kernel's time stamping subsystem with the assumption that something else is providing the PTP interface to a reference clock plugged into the AXI Ethernet Subsystem. So at the other end, the ha1588 provides the PTP interface to that subsystem as well as the RTC core (i.e., the reference clock). The loop is closed by having the ptp4l daemon interact with that subsystem by using the ethX and /dev/ptpY together.

I'm guessing at this one, but I think if you were to add in the FIFO accesses for the time stamping portion of the ha1588 core, you would then need to integrate that with a MAC to get those packets into the queue (but I honestly don't know how you would do that with the Xilinx cores, instead I think you would use the whole ha1588 reference design rather than only the RTC as we did).

Question 2, I won't be much of a help on that one; I don't have any familiarity with that module.

[mtetrault]

I've been using the ZC706 board with the Avnet AES-FMC-NETW1-G expansion board. It has two 10/100/1000 ethernet PHY with RJ45 connectors. Unfortunately the ZC706 is not in the supported board list for the expansion board, and its been working on and off (I think because of trace length mismatch to the FMC connector). We have a collaborator that owns a Zedboard, so when it becomes free it should work normally and I'll be able to test more easily.

Yes, I'm aware the faulty 1588 is for the PS part. The 1588 is probably greyed out in the AXI core because of this note in the AXI subsystem product guide, Appendix A (pg138-axi-ethernet.pdf):

IEEE 1588 hardware timestamping support is available only for the 1000BASE-X or SGMII

And I'm targetting GMII/RGMII interface.

Question 1: you answered the question with this last part, thanks!

The loop is closed by having the ptp4l daemon interact with that subsystem by using the ethX and /dev/ptpY together.

If you are using a SFP cage, are you using sgmii? In that case, how did you link the AXI Ethernet with the ha1588 in the PL part?

[btgoodwin]

I agree with you on why it's greyed out. We used the 1000BASE-X interface on our design since we had the SFP cage. When you add the AXI Ethernet Subsystem with 1000BASE-X and time stamping enabled, it exposes some clock inputs (systemtimer_*) that match up with the ha1588 RTC core which makes the integration relatively easy.

If you were to expose the TSU fifos of the HA1588 core and put it in-line with your GMII/RGMII interface like the HA1588 diagrams show, and you tried to use the AXI Ethernet driver, I think you would need to patch the driver to not only permit RGMII and CONFIG_XILINX_AXI_EMAC_HWTSTAMP at the same time, but also to utilize your registers and device rather than its own. I imagine that would be pretty challenging.

If it's not too late for a part change, maybe the AES-FM-S14 is a better route (realizing of course it costs 3x as much and has 4 ports vs. 2)?

[mtetrault]

Unfortunately (for me), we really need a RJ45 connector, because the Zynq is acting as master clock for a purchased piece of equipment with RJ45-only connectivity. Since there is only one of these, it doesn't make sense to add a PTP-capable switch to the system just for that.

On the bright side, the software timestamping hybrid modification to ptp4l is sufficient for our application (where I get the time from the /dev/ptpX instead of the kernel time). It's relatively simple, so not too bad to maintain. 

Going the driver route would probably be cleaner but more complicated, as you mention. My naive thinking would be to expose a dummy /dev/ptpX for /dev/ethX through the driver, and call ptp4l with the existing flag redirecting to /dev/ptpY which would be from ha1588, with the logic wired to catch and timestamp the PTP packets.

Hopefully I'll find time to explore this solution, and then get the full HW timestamping path to work (a part of this being personal satisfaction!).

On Wed, 2019-08-07 at 07:49 -0700, Thomas Goodwin wrote:

I agree with you on why it's greyed out. We used the 1000BASE-X interface on our design since we had the SFP cage. When you add the AXI Ethernet Subsystem with 1000BASE-X and time stamping enabled, it exposes some clock inputs (systemtimer_*) that match up with the ha1588 RTC core which makes the integration relatively easy. If you were to expose the TSU fifos of the HA1588 core and put it in- line with your GMII/RGMII interface like the HA1588 diagrams show, and you tried to use the AXI Ethernet driver, I think you would need to patch the driver to not only permit RGMII and CONFIG_XILINX_AXI_EMAC_HWTSTAMP at the same time, but also to utilize your registers and device rather than its own. I imagine that would be pretty challenging. If it's not too late for a part change, maybe the AES-FM-S14 is a better route (realizing of course it costs 3x as much and has 4 ports vs. 2)? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[btgoodwin]

I see. Our design requirements changed from fiber to copper on this design; we purchased base-T modules and continued on. Even with copper, our results were very close.

Best of luck! If you get good results in the software mod route that you can share, I'd like to hear them. :)

[mtetrault]

Base-T could be a working solution, they don't look too expensive. We'll look into it as well.

Thanks for the feedback and suggestions! I'll keep you posted.

Update: Hi Thomas, the copper SFP solution is quite appealing. I've ordered a unit and will give it a run. The custom PCB design has been pushed off so I will have time to test this alternative. Thanks!

On Wed, 2019-08-07 at 09:29 -0700, Thomas Goodwin wrote:

I see. Our design requirements changed from fiber to copper on this design; we purchased base-T modules and continued on. Even with copper, our results were very close. Best of luck! If you get good results in the software mod route that you can share, I'd like to hear them. :) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.