search

EttusResearch / uhd

The USRP™ Hardware Driver Repository
http://uhd.ettus.com
Other
1k stars 666 forks source link

X440 Not advertising 100GbE supported link modes. Receiver cannot auto-negotiate 100GbE #765

Open RyanChaosInc opened 4 months ago

RyanChaosInc commented 4 months ago

Issue Description

X440 configured with CG_400 FPGA Flavor does not properly advertise supported link modes which does not allow for receiving NIC or switches to auto-negotiate to 100 GbE Full Duplex via QSFP28.

Once auto-negotiate was DISABLED on 100 GbE switch, then link was able to be detected and established. While auto-negation is enabled on the SDR side, the SDR does not advertise necessary data to enable the other side of the connection to establish the link.

ethtool on the X400 configured with CG_400 confirms this.
In the output below, the external switch has been configured with auto-negotiate=false to enable the link to be established. Therefore, Link detected: yes below is expected w/ 100GbE tranceiver/cables:

root@SDR15:~# ethtool sfp1
Settings for sfp1:
        Supported ports: [ MII ]
        Supported link modes:   Not reported
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 100000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Link detected: yes

Setup Details

X400 configured w/ CG_400

I tried 5 different brands of QSFP28 transceivers & cables.
All but one worked once I disabled once I turned off auto-negotiation on the switch's QSFP28 port connected to the SDR.

If configured, with other FPGA flavors using 4 x 10GbE and a QSFP28 splitter, 10GbE auto-negotiates fine. The problem is when trying to connect QSFP28 to QSFP28 at 100GbE using CG_400.

root@SDR15:~# uhd_usrp_probe --args addr=127.0.0.1
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.6.0.0-0-g50fa3baa
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=127.0.0.1,type=x4xx,product=x440,serial=336B2D6,name=SDR15,fpga=CG_400,claimed=False,addr=127.0.0.1
[INFO] [MPM.PeriphManager] init() called with device args 'fpga=CG_400,mgmt_addr=127.0.0.1,name=SDR15,product=x440,clock_source=internal,time_source=internal,initializing=True'.
  _____________________________________________________
 /
|       Device: X400-Series Device
|     _____________________________________________________
|    /
|   |       Mboard: SDR15
|   |   module_pid: 42048
|   |   module_rev: 4
|   |   module_serial: 336B2D6
|   |   pid: 1088
|   |   rev: 6
|   |   rev_compat: 6
|   |   serial: 336A367
|   |   MPM Version: 5.3
|   |   FPGA Version: 8.2
|   |   FPGA git hash: e547a6b.clean
|   |   RFNoC capable: Yes
|   |
|   |   Time sources:  internal, external, qsfp0, gpsdo
|   |   Clock sources: mboard, internal, external, nsync, gpsdo
|   |   Sensors: ref_locked, fan0, fan1, temp_fpga, temp_main_power, temp_scu_internal, gps_alarm, gps_enabled, gps_gpgga, gps_locked, gps_phase_lock, gps_sky, gps_survey, gps_time, gps_tpv, gps_warmup
|     _____________________________________________________
|    /
|   |       RFNoC blocks on this device:
|   |
|   |   * 0/Radio#0
|   |   * 0/Radio#1
|     _____________________________________________________
|    /
|   |       Static connections on this device:
|   |
|   |   * 0/SEP#0:0==>0/Radio#0:0
|   |   * 0/Radio#0:0==>0/SEP#0:0
|   |   * 0/SEP#1:0==>0/Radio#0:1
|   |   * 0/Radio#0:1==>0/SEP#1:0
|   |   * 0/SEP#2:0==>0/Radio#0:2
|   |   * 0/Radio#0:2==>0/SEP#2:0
|   |   * 0/SEP#3:0==>0/Radio#0:3
|   |   * 0/Radio#0:3==>0/SEP#3:0
|   |   * 0/SEP#4:0==>0/Radio#1:0
|   |   * 0/Radio#1:0==>0/SEP#4:0
|   |   * 0/SEP#5:0==>0/Radio#1:1
|   |   * 0/Radio#1:1==>0/SEP#5:0
|   |   * 0/SEP#6:0==>0/Radio#1:2
|   |   * 0/Radio#1:2==>0/SEP#6:0
|   |   * 0/SEP#7:0==>0/Radio#1:3
|   |   * 0/Radio#1:3==>0/SEP#7:0
|     _____________________________________________________
|    /
|   |       TX Dboard: 0/Radio#0
|   |     _____________________________________________________
|   |    /
|   |   |       TX Frontend: 0
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, CAL_LOOPBACK
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       TX Frontend: 1
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, CAL_LOOPBACK
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       TX Frontend: 2
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, CAL_LOOPBACK
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       TX Frontend: 3
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, CAL_LOOPBACK
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|     _____________________________________________________
|    /
|   |       RX Dboard: 0/Radio#0
|   |     _____________________________________________________
|   |    /
|   |   |       RX Frontend: 0
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, RX1, SYNC_INT, SYNC_EXT, CAL_LOOPBACK, TERM
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       RX Frontend: 1
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, RX1, SYNC_INT, SYNC_EXT, CAL_LOOPBACK, TERM
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       RX Frontend: 2
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, RX1, SYNC_INT, SYNC_EXT, CAL_LOOPBACK, TERM
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       RX Frontend: 3
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, RX1, SYNC_INT, SYNC_EXT, CAL_LOOPBACK, TERM
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|     _____________________________________________________
|    /
|   |       TX Dboard: 0/Radio#1
|   |     _____________________________________________________
|   |    /
|   |   |       TX Frontend: 0
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, CAL_LOOPBACK
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       TX Frontend: 1
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, CAL_LOOPBACK
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       TX Frontend: 2
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, CAL_LOOPBACK
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       TX Frontend: 3
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, CAL_LOOPBACK
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|     _____________________________________________________
|    /
|   |       RX Dboard: 0/Radio#1
|   |     _____________________________________________________
|   |    /
|   |   |       RX Frontend: 0
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, RX1, SYNC_INT, SYNC_EXT, CAL_LOOPBACK, TERM
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       RX Frontend: 1
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, RX1, SYNC_INT, SYNC_EXT, CAL_LOOPBACK, TERM
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       RX Frontend: 2
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, RX1, SYNC_INT, SYNC_EXT, CAL_LOOPBACK, TERM
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No
|   |     _____________________________________________________
|   |    /
|   |   |       RX Frontend: 3
|   |   |   Name: FBX
|   |   |   Antennas: TX/RX0, RX1, SYNC_INT, SYNC_EXT, CAL_LOOPBACK, TERM
|   |   |   Sensors: temperature, rfdc_rate, lo_locked, nco_locked
|   |   |   Freq range: 1.000 to 4000.000 MHz
|   |   |   Gain range all: 0.0 to 0.0 step 0.0 dB
|   |   |   Bandwidth range: 1600000000.0 to 1600000000.0 step 0.0 Hz
|   |   |   Connection Type: IQ
|   |   |   Uses LO offset: No

Expected Behavior

Auto-negotiate should be advertised for SFP0 and SFP1 when using CG_400 and therefore supported at 100GbE, consistent with 10GbE and 1GbE on other Ettus SDRs..

Actual Behaviour

Link cannot be established to a receiving NIC or switch using auto-negotiation. However, if same cable is connected to 2 X440s w/ CG_400 a link light is established.

Steps to reproduce the problem

Configure X440 with CG_400. Use QSFP28 cable to connect to a QSFP28 port on a switch or NIC set to auto-negotiate. ethtool sfp0 will show link detected: no and no link light will show. ip a will show no IP address.

Additional Information

Many other parameters were tried on both the SDR and Switch side and ultimately it was narrowed down to lack of advertising of the link mode by the SDR to enable auto-negotiation.

In doing so:

  1. 5 brands of QSFP28 transceivers & cables were purchased and tried. All but one worked, once this issue was figured out. Therefore, the root cause is not the transceivers or cables.
  2. ethtool is found to not implement a large amount of options specified by ethtool -h. Many of the options, such as -s are not supported. It appears via ethtool -i sfp0 that the nixge driver is very minimal. Therefore ethtool wasn't able to override or set missing config.
wordimont commented 4 months ago

Apologies for the confusion. The X440 does not support auto-negotiation on the QSFP ports. It only supports the mode described by the bitstream flavor (CG = 100 GbE). You have to switch bitstreams to change modes. This is intentional to save resources on the FPGA for other uses. I believe this is the same behavior as the SFP ports on other USRPs, but you may get lucky with auto-negotion enabled on the NIC.

From reading your description, it sounds like it's behaving as expected. Given it's not supported, is there something you saw that still seems incorrect? Or was there a mistake in the documentation?

RyanChaosInc commented 4 months ago

I understand that and agree that we must change the bitstream flavor to match the desired speed. No issues there.

I still believe there is a bug with the driver in 100GbE configurations.

Ethtool should report the correct and accurate information so the other network device can correctly and automatically connect.

Supported link modes, supports auto-negotiation, advertised link modes, and advertised auto-negotiation must be accurate to enable NICs to connect to it normally w/o special configuration.

100GbE:

root@SDR15:~# ethtool sfp1
Settings for sfp1:
        Supported ports: [ MII ]
        Supported link modes:   Not reported
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 100000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Link detected: yes

For reference, the 10GbE driver correctly reports required data and doesn't show them as "Not reported". 10GbE:

root@SDR4:~# ethtool sfp1
Settings for sfp1:
        Supported ports: [ ]
        Supported link modes:   10000baseER/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseER/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 4
        Transceiver: internal
        Auto-negotiation: off
        Link detected: yes
wordimont commented 4 months ago

Thank you for creating this issue! I agree it looks like some of the information being reported is wrong.

You also mentioned that you got "all but one" of the cables/transceivers to work. Can you share the part numbers for the items that didn't work?

RyanChaosInc commented 4 months ago

Thank you for creating this issue! I agree it looks like some of the information being reported is wrong.

You also mentioned that you got "all but one" of the cables/transceivers to work. Can you share the part numbers for the items that didn't work?

This is the one that didn't work, but I didn't have time to chase down why. https://www.amazon.com/dp/B0CJ8BWMB3