search

open-switch / opx-nas-interface

https://openswitch.net
2 stars 13 forks source link

Fail to bring up all interface between s5148 and s4248-fbl after fanout #51

Open waliulislam opened 6 years ago

waliulislam commented 6 years ago

I have two 100 gig link between S5148 and S4248-fbl. Media type is QSFP28 100GBASE LR.

s5148 s4248-fbl link 1: e101-053-0 -------------------------e101-048-0 link 2: e101-054-0--------------------------e101-047-0

Both ports are configure to be fanout to 4x1 10 g mode in dn_nas_fanout_confg.init file. At reboot of s4200, I run a script to bring up all interface. For link 1 opx-ethtool shows all four ports are UP in both side but for link 2 I have seen some link are UP and some are down and some are one side shows up and the peer device that shows down.

root@TR1:~# opx-ethtool e101-054-1
Settings for e101-054-1:
    Channel ID:   1
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55840019
    Qualified: Yes
    Administrative State: UP
    Operational State: DOWN
    Supported Speed (in Mbps):  [25000, 10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : 0
    Duplex   : full
root@AG2:~# opx-ethtool e101-047-1
Settings for e101-047-1:
    Channel ID:   1
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55830005
    Qualified: Yes
    Administrative State: UP
    Operational State: DOWN
    Supported Speed (in Mbps):  [10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : 0
    Duplex   : full

root@TR1:~# opx-ethtool e101-054-2
Settings for e101-054-2:
    Channel ID:   2
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55840019
    Qualified: Yes
    Administrative State: UP
    Operational State: UP
    Supported Speed (in Mbps):  [25000, 10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : False
    Duplex   : full
root@AG2:~# opx-ethtool e101-047-2
Settings for e101-047-2:
    Channel ID:   2
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55830005
    Qualified: Yes
    Administrative State: UP
    Operational State: DOWN
    Supported Speed (in Mbps):  [10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : 0
    Duplex   : full

root@TR1:~# opx-ethtool e101-054-3
Settings for e101-054-3:
    Channel ID:   3
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55840019
    Qualified: Yes
    Administrative State: UP
    Operational State: UP
    Supported Speed (in Mbps):  [25000, 10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : False
    Duplex   : full
root@AG2:~# opx-ethtool e101-047-3
Settings for e101-047-3:
    Channel ID:   3
    Transceiver Status: Enable
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55830005
    Qualified: Yes
    Administrative State: UP
    Operational State: UP
    Supported Speed (in Mbps):  [10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : False
    Duplex   : full

root@TR1:~# opx-ethtool e101-054-4
Settings for e101-054-4:
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55840019
    Qualified: Yes
    Administrative State: UP
    Operational State: UP
    Supported Speed (in Mbps):  [25000, 10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : False
    Duplex   : full
root@AG2:~# opx-ethtool e101-047-4
Settings for e101-047-4:
    Media Type: QSFP28 100GBASE LR4
    Part Number: GMFC5
    Serial Number: CN0GMFC55830005
    Qualified: Yes
    Administrative State: UP
    Operational State: DOWN
    Supported Speed (in Mbps):  [10000]
    Auto Negotiation : off
    Configured Speed   : auto
    Operating Speed   : 0
    Duplex   : full
Hardware configuration:
S5148:
xpShell):linkMgr)print_all_port_status 0

==================================================================================================================================================================
ChassisPort# CNX-Port# :[PTGNum ]:   Link     Fault    PTG-signalOK       MAC-Mode         FEC-Mode     serdesId:(Loopback, signalOk, MediaStatus)  LB
==================================================================================================================================================================
--------------------
--------------------
    53/1:     CNX- 36  :[PTG: 9 ]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [9  :(EXT, OK       ,  C) Tx(Inv) : Rx(Inv)]  ---
    53/2:     CNX- 37  :[PTG: 9 ]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [10 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    53/4:     CNX- 38  :[PTG: 9 ]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [11 :(EXT, OK       ,  C) Tx(Inv) : Rx(Inv)]  ---
    53/3:     CNX- 39  :[PTG: 9 ]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [12 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    51/1:     CNX- 40  :[PTG: 10]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [13 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    51/2:     CNX- 41  :[PTG: 10]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [14 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    51/4:     CNX- 42  :[PTG: 10]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [15 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)]  ---
    51/3:     CNX- 43  :[PTG: 10]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [16 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)]  ---
    49/1:     CNX- 44  :[PTG: 11]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [17 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    49/2:     CNX- 45  :[PTG: 11]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [18 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    49/4:     CNX- 46  :[PTG: 11]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [19 :(EXT, OK       ,  C) Tx(Inv) : Rx(Inv)]  ---
    49/3:     CNX- 47  :[PTG: 11]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [20 :(EXT, OK       ,  C) Tx(Inv) : Rx(Inv)]  ---
    40/1:     CNX- 48  :[PTG: 12]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [21 :(EXT, NO_SIGNAL, NC) Tx      : Rx(Inv)]  ---
    47/1:     CNX- 49  :[PTG: 12]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [22 :(EXT, NO_SIGNAL, NC) Tx      : Rx     ]  ---
    34/1:     CNX- 50  :[PTG: 12]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [23 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    33/1:     CNX- 51  :[PTG: 12]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [24 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    54/2:     CNX- 52  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [25 :(EXT, OK       ,  C) Tx      : Rx     ]  ---
    54/1:     CNX- 53  :[PTG: 13]:   DOWN   FAULT(R)             OK    MAC_MODE_4X10GB    DISABLE      [26 :(EXT, OK       ,  C) Tx      : Rx     ]  ---
    54/3:     CNX- 54  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [27 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    54/4:     CNX- 55  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [28 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    52/2:     CNX- 56  :[PTG: 14]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [29 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    52/1:     CNX- 57  :[PTG: 14]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      Error: Mac serdes signalOK = 0 for portNum 2, Can not tune serdes.
[30 :(EXT, NO_SIGNAL, NC) Tx      : Rx     ]  ---
    52/3:     CNX- 58  :[PTG: 14]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [31 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    52/4:     CNX- 59  :[PTG: 14]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X10GB    DISABLE      [32 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    50/2:     CNX- 60  :[PTG: 15]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [33 :(EXT, OK       ,  C) Tx      : Rx     ]  ---
    50/1:     CNX- 61  :[PTG: 15]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [34 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    50/4:     CNX- 62  :[PTG: 15]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [35 :(EXT, OK       ,  C) Tx(Inv) : Rx(Inv)]  ---
    50/3:     CNX- 63  :[PTG: 15]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [36 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    41/1:     CNX- 64  :[PTG: 16]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [37 :(EXT, NO_SIGNAL, NC) Tx      : Rx(Inv)]  ---
    43/1:     CNX- 65  :[PTG: 16]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [38 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)]  ---
    44/1:     CNX- 66  :[PTG: 16]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [39 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    48/1:     CNX- 67  :[PTG: 16]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [40 :(EXT, NO_SIGNAL, NC) Tx      : Rx(Inv)]  ---
    39/1:     CNX- 68  :[PTG: 17]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [41 :(EXT, NO_SIGNAL, NC) Tx      : Rx(Inv)]  ---
    45/1:     CNX- 69  :[PTG: 17]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [42 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---
    46/1:     CNX- 70  :[PTG: 17]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [43 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx(Inv)]  ---
    42/1:     CNX- 71  :[PTG: 17]:   DOWN      FAULT      NO_SIGNAL    MAC_MODE_4X25GB    DISABLE      [44 :(EXT, NO_SIGNAL, NC) Tx(Inv) : Rx     ]  ---

(xpShell):linkMgr) 

S4248-fbl:
    Duplex   : full
root@AG2:~# hshell -c "ps"
Entry not found
0:_bcm_dpp_gport_to_hw_resources: Entry not found
0:bcm_petra_port_discard_set: Entry not found
                 ena/    speed/ link auto    STP                  lrn  inter   max  loop
           port  link    duplex scan neg?   state   pause  discrd ops   face frame  back
-----------------
-----------------
      xe65( 65)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe66( 66)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe67( 67)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe68( 68)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe69( 69)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe70( 70)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe71( 71)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe72( 72)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028

Even though all links are down on hardware in s4248 device, yet opx-ethtool shows one link is up. As those links are part of the bond and karnel try to bring up those interface , I have seen a different result when I ran the ps for second time.

      xe65( 65)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe66( 66)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe67( 67)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe68( 68)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe69( 69)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe70( 70)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe71( 71)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe72( 72)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe65( 65)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe66( 66)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe67( 67)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe68( 68)  down   10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe69( 69)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe70( 70)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe71( 71)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028
      xe72( 72)  up     10G  FD   SW  No   Forward          None   FA    XFI  9028

After some times all links are up in hardware on S5148 side

    54/2:     CNX- 52  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [25 :(EXT, OK       ,  C) Tx      : Rx     ]  ---
    54/1:     CNX- 53  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [26 :(EXT, OK       ,  C) Tx      : Rx     ]  ---
    54/3:     CNX- 54  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [27 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
    54/4:     CNX- 55  :[PTG: 13]:     UP         OK             OK    MAC_MODE_4X10GB    DISABLE      [28 :(EXT, OK       ,  C) Tx(Inv) : Rx     ]  ---
jeff-yin commented 6 years ago

@waliulislam does flapping the interfaces have any effect?

jeff-yin commented 6 years ago

This might be a service file dependency issue.

atanu-mandal commented 6 years ago

There was a recent fix in service dependency in opx-nas-interface. Please verify if the issue is reproducible having upgraded the package.

GarrickHe commented 5 years ago

@waliulislam

Any updates on this?