search

opennetworkinglab / stratum-onos-demo

Stratum+ONOS demo @ ONF Connect 2019 & EuroP4 '19
Apache License 2.0
13 stars 7 forks source link

Fix missing spine routes #66

Closed pudelkoM closed 5 years ago

ccascone commented 5 years ago

@pudelkoM can you elaborate a bit more on the issue and the fix?

ccascone commented 5 years ago

@pudelkoM I understand this improves the previous approach, as instead of iterating for all leaf devices when updating a given spine routes, now we update routes only for the leaf associated with a link event. However, I'm not sure how the previous approach could be causing missing spine routes.

pudelkoM commented 5 years ago

The root issue was missing leafs in deviceService.getDevices(). When called from a debugger leaf2 and leaf 3 are present. But inside the onos app they are missing in the return set in this for loop: for (Device device : deviceService.getDevices()) in setUpSpineRoutes. We then never create any spine flows for both leafs.

Pier suggested some kind of concurrency issue inside onos code or the eventual consisted hash map. But that's still weird, since logging from a breakpoint on the exact call returned the correct set.

Here is some debug output from inside the for loop. First line is the current device for the loop iteration, second line are all devices as listed by deviceService.getDevices(). leaf2 and 3 are in the list/set, but the loop never gets to them.

device:spine3
[DefaultDevice{id=device:leaf1, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=Wedge100BF-32X (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:spine3, type=SWITCH, manufacturer=Stordis, hwVersion=Stordis (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:spine1, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D7032Q28B (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:spine2, type=SWITCH, manufacturer=Delta Electronics, Inc., hwVersion=AG9064v1 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:leaf4, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D5254 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:leaf5, type=SWITCH, manufacturer=Dell Inc., hwVersion=Z9100 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf2, type=SWITCH, manufacturer=Quanta Cloud Technology, hwVersion=T7032-IX1 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf3, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=AS7712-32X (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}]
device:spine1
[DefaultDevice{id=device:leaf1, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=Wedge100BF-32X (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:spine3, type=SWITCH, manufacturer=Stordis, hwVersion=Stordis (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:spine1, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D7032Q28B (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:spine2, type=SWITCH, manufacturer=Delta Electronics, Inc., hwVersion=AG9064v1 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:leaf4, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D5254 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:leaf5, type=SWITCH, manufacturer=Dell Inc., hwVersion=Z9100 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf2, type=SWITCH, manufacturer=Quanta Cloud Technology, hwVersion=T7032-IX1 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf3, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=AS7712-32X (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}]
device:spine2
[DefaultDevice{id=device:leaf1, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=Wedge100BF-32X (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:spine3, type=SWITCH, manufacturer=Stordis, hwVersion=Stordis (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:spine1, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D7032Q28B (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:spine2, type=SWITCH, manufacturer=Delta Electronics, Inc., hwVersion=AG9064v1 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:leaf4, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D5254 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:leaf5, type=SWITCH, manufacturer=Dell Inc., hwVersion=Z9100 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf2, type=SWITCH, manufacturer=Quanta Cloud Technology, hwVersion=T7032-IX1 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf3, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=AS7712-32X (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}]
device:leaf4
[DefaultDevice{id=device:leaf1, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=Wedge100BF-32X (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:spine3, type=SWITCH, manufacturer=Stordis, hwVersion=Stordis (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:spine1, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D7032Q28B (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:spine2, type=SWITCH, manufacturer=Delta Electronics, Inc., hwVersion=AG9064v1 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:leaf4, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D5254 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:leaf5, type=SWITCH, manufacturer=Dell Inc., hwVersion=Z9100 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf2, type=SWITCH, manufacturer=Quanta Cloud Technology, hwVersion=T7032-IX1 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf3, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=AS7712-32X (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}]
device:leaf5
[DefaultDevice{id=device:leaf1, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=Wedge100BF-32X (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:spine3, type=SWITCH, manufacturer=Stordis, hwVersion=Stordis (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:spine1, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D7032Q28B (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:spine2, type=SWITCH, manufacturer=Delta Electronics, Inc., hwVersion=AG9064v1 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:leaf4, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D5254 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:leaf5, type=SWITCH, manufacturer=Dell Inc., hwVersion=Z9100 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf2, type=SWITCH, manufacturer=Quanta Cloud Technology, hwVersion=T7032-IX1 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf3, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=AS7712-32X (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}]
device:leaf1
[DefaultDevice{id=device:leaf1, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=Wedge100BF-32X (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:spine3, type=SWITCH, manufacturer=Stordis, hwVersion=Stordis (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:spine1, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D7032Q28B (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:spine2, type=SWITCH, manufacturer=Delta Electronics, Inc., hwVersion=AG9064v1 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.mavericks}, DefaultDevice{id=device:leaf4, type=SWITCH, manufacturer=Inventec Corp., hwVersion=D5254 (BFN Tofino), swVersion=stratum_bf, serialNumber=unknown, driver=stratum-tofino:org.stratumproject.fabric-demo.montara}, DefaultDevice{id=device:leaf5, type=SWITCH, manufacturer=Dell Inc., hwVersion=Z9100 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf2, type=SWITCH, manufacturer=Quanta Cloud Technology, hwVersion=T7032-IX1 (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}, DefaultDevice{id=device:leaf3, type=SWITCH, manufacturer=Edge-Core Networks Corp., hwVersion=AS7712-32X (BCM Tomahawk), swVersion=stratum_bcm, serialNumber=unknown, driver=stratum-fpm:org.stratumproject.fabric-demo.fpm}]
ccascone commented 5 years ago

I agree such a race condition between link events and device store is weird... anyways LGTM. As I said before this is an improvement over the previous algorithm.

ccascone commented 5 years ago

Feel free to merge this whenever you are ready

pudelkoM commented 5 years ago

Had to get info from Yi about the RouteService in the latest commit