Closed italovalcy closed 2 weeks ago
It may be related to https://github.com/atlanticwave-sdx/pce/issues/123
I noticed that _vlan_tags_table
only gets updated when adding the topology, not when updating the topology as per:
and the add_topology is only called when the SDX-LC is sending a topology for a unknown domain (not saved on the database). So if probably if you restart the SDX-Controller and sends again the topology for a domain (which was saved to the DB before the controller to restart), it seems like this issue could be triggered. This seems to happen at least the first time you restart SDX-Controller.
Hello, team. Did you guys have the opportunity to check this issue? Ccing @YufengXin @sajith and @congwang09
I'm writing now because once again I was running tests and faced this issue. The SDX controller logs say this:
$ docker logs sdx-controller -t > logs-sdx-controller
$ tail -n 10 logs-sdx-controller
2024-08-28T09:08:01.282953671Z WARNING:sdx_pce.topology.temanager:Can't find a mapping for domain:urn:sdx:topology:tenet.ac.za port:urn:sdx:port:tenet.ac.za:Tenet03:50
2024-08-28T09:08:01.282997534Z WARNING:sdx_pce.topology.temanager:Port urn:sdx:port:tenet.ac.za:Tenet01:41 not found in port map, it's a user port, by default uses the OXP vlan
2024-08-28T09:08:01.283029531Z WARNING:sdx_pce.topology.temanager:Port urn:sdx:port:tenet.ac.za:Tenet03:50 not found in port map, it's a user port, by default uses the OXP vlan
2024-08-28T09:08:01.283043683Z INFO:sdx_pce.topology.temanager:VLAN reservation: domain: urn:sdx:topology:tenet.ac.za, ingress_vlan: None, egress_vlan: None
2024-08-28T09:08:01.283552127Z ERROR:sdx_controller.handlers.connection_handler:Error when generating/publishing breakdown: Not implemented - Traceback (most recent call last):, File "/opt/venv/lib/python3.9/site-packages/sdx_controller/handlers/connection_handler.py", line 157, in place_connection, breakdown = te_manager.generate_connection_breakdown(, File "/opt/venv/lib/python3.9/site-packages/sdx_pce/topology/temanager.py", line 620, in generate_connection_breakdown, tagged_breakdown = self._reserve_vlan_breakdown(, File "/opt/venv/lib/python3.9/site-packages/sdx_pce/topology/temanager.py", line 792, in _reserve_vlan_breakdown, self._unreserve_vlan(domain, ingress_port, ingress_vlan), File "/opt/venv/lib/python3.9/site-packages/sdx_pce/topology/temanager.py", line 933, in _unreserve_vlan, assert False, "Not implemented", AssertionError: Not implemented,
2024-08-28T09:08:01.283590780Z INFO:sdx_controller.controllers.l2vpn_controller:place_connection result: ID: 15e2dae0-34cc-4256-af5d-e178aa85e07e reason='Error: Not implemented', code=400
2024-08-28T09:08:01.283654623Z DEBUG:connexion.apis.abstract:Getting data and status code
2024-08-28T09:08:01.283825341Z DEBUG:connexion.apis.abstract:Prepared body and status code (400)
2024-08-28T09:08:01.283903271Z DEBUG:connexion.apis.abstract:Got framework response
2024-08-28T09:08:01.284573954Z INFO: 192.168.0.1:50760 - "POST /SDX-Controller/l2vpn/1.0 HTTP/1.1" 400 Bad Request
Which indicates the issue was missing vlan table on TENET domain.
Looking on how TENET domain had its topology imported, we have:
$ egrep 'MQ received message:.* "urn:sdx:topology:tenet.ac.za"' logs-sdx-controller
2024-08-28T09:02:10.937524682Z DEBUG:sdx_controller.messaging.rpc_queue_consumer:MQ received message:b'{"name": "Tenet-OXP", "id": "urn:sdx:topology:tenet.ac.za", "version": 3, "timestamp": "2024-08-28T09:02:07Z", "model_version": "2.0.0", "nodes": [{"name": "Tenet02", "id": "urn:sdx:node:tenet.ac.za:Tenet02", "location": {"address": "", "latitude": 0.0, "longitude": -30.0, "iso3166_2_lvl4": "", "private": []}, "ports": [{"id": "urn:sdx:port:tenet.ac.za:Tenet02:41", "name": "Tenet02-eth41", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet02:1", "name": "Tenet02-eth1", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet02:50", "name": "Tenet02-eth50", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}], "status": "up", "state": "enabled"}], "links": [], "services": ["l2vpn-ptp"]}'
2024-08-28T09:02:10.937726512Z INFO:sdx_controller.handlers.lc_message_handler:MQ received message:b'{"name": "Tenet-OXP", "id": "urn:sdx:topology:tenet.ac.za", "version": 3, "timestamp": "2024-08-28T09:02:07Z", "model_version": "2.0.0", "nodes": [{"name": "Tenet02", "id": "urn:sdx:node:tenet.ac.za:Tenet02", "location": {"address": "", "latitude": 0.0, "longitude": -30.0, "iso3166_2_lvl4": "", "private": []}, "ports": [{"id": "urn:sdx:port:tenet.ac.za:Tenet02:41", "name": "Tenet02-eth41", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet02:1", "name": "Tenet02-eth1", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet02:50", "name": "Tenet02-eth50", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}], "status": "up", "state": "enabled"}], "links": [], "services": ["l2vpn-ptp"]}'
2024-08-28T09:02:28.245063093Z DEBUG:sdx_controller.messaging.rpc_queue_consumer:MQ received message:b'{"name": "Tenet-OXP", "id": "urn:sdx:topology:tenet.ac.za", "version": 10, "timestamp": "2024-08-28T09:02:24Z", "model_version": "2.0.0", "nodes": [{"name": "Tenet02", "id": "urn:sdx:node:tenet.ac.za:Tenet02", "location": {"address": "Johanesburgo", "latitude": -26.0, "longitude": 28.0, "iso3166_2_lvl4": "ZA-GP", "private": []}, "ports": [{"id": "urn:sdx:port:tenet.ac.za:Tenet02:41", "name": "Tenet02-eth41", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "urn:sdx:port:sax.net:Sax02:41", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet02:1", "name": "Tenet02-eth1", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet02:50", "name": "Tenet02-eth50", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}], "status": "up", "state": "enabled"}, {"name": "Tenet03", "id": "urn:sdx:node:tenet.ac.za:Tenet03", "location": {"address": "EastLondon", "latitude": -33.0, "longitude": 27.0, "iso3166_2_lvl4": "ZA-EC", "private": []}, "ports": [{"id": "urn:sdx:port:tenet.ac.za:Tenet03:2", "name": "Tenet03-eth2", "node": "urn:sdx:node:tenet.ac.za:Tenet03", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "name": "Tenet03-eth50", "node": "urn:sdx:node:tenet.ac.za:Tenet03", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}], "status": "up", "state": "enabled"}, {"name": "Tenet01", "id": "urn:sdx:node:tenet.ac.za:Tenet01", "location": {"address": "CapeTown", "latitude": -33.0, "longitude": 18.0, "iso3166_2_lvl4": "ZA-WC", "private": []}, "ports": [{"id": "urn:sdx:port:tenet.ac.za:Tenet01:41", "name": "Tenet01-eth41", "node": "urn:sdx:node:tenet.ac.za:Tenet01", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "urn:sdx:port:sax.net:Sax01:41", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet01:1", "name": "Tenet01-eth1", "node": "urn:sdx:node:tenet.ac.za:Tenet01", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet01:2", "name": "Tenet01-eth2", "node": "urn:sdx:node:tenet.ac.za:Tenet01", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet01:50", "name": "Tenet01-eth50", "node": "urn:sdx:node:tenet.ac.za:Tenet01", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}], "status": "up", "state": "enabled"}], "links": [{"name": "Tenet01/1_Tenet02/1", "id": "urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1", "ports": ["urn:sdx:port:tenet.ac.za:Tenet02:1", "urn:sdx:port:tenet.ac.za:Tenet01:1"], "type": "intra", "bandwidth": 10, "residual_bandwidth": 100, "latency": 0, "packet_loss": 0, "availability": 0, "status": "up", "state": "enabled", "private": ["packet_loss"]}, {"name": "Tenet01/2_Tenet03/2", "id": "urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2", "ports": ["urn:sdx:port:tenet.ac.za:Tenet03:2", "urn:sdx:port:tenet.ac.za:Tenet01:2"], "type": "intra", "bandwidth": 10, "residual_bandwidth": 100, "latency": 0, "packet_loss": 0, "availability": 0, "status": "up", "state": "enabled", "private": ["packet_loss"]}], "services": ["l2vpn-ptp"]}'
2024-08-28T09:02:28.245691969Z INFO:sdx_controller.handlers.lc_message_handler:MQ received message:b'{"name": "Tenet-OXP", "id": "urn:sdx:topology:tenet.ac.za", "version": 10, "timestamp": "2024-08-28T09:02:24Z", "model_version": "2.0.0", "nodes": [{"name": "Tenet02", "id": "urn:sdx:node:tenet.ac.za:Tenet02", "location": {"address": "Johanesburgo", "latitude": -26.0, "longitude": 28.0, "iso3166_2_lvl4": "ZA-GP", "private": []}, "ports": [{"id": "urn:sdx:port:tenet.ac.za:Tenet02:41", "name": "Tenet02-eth41", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "urn:sdx:port:sax.net:Sax02:41", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet02:1", "name": "Tenet02-eth1", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet02:50", "name": "Tenet02-eth50", "node": "urn:sdx:node:tenet.ac.za:Tenet02", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}], "status": "up", "state": "enabled"}, {"name": "Tenet03", "id": "urn:sdx:node:tenet.ac.za:Tenet03", "location": {"address": "EastLondon", "latitude": -33.0, "longitude": 27.0, "iso3166_2_lvl4": "ZA-EC", "private": []}, "ports": [{"id": "urn:sdx:port:tenet.ac.za:Tenet03:2", "name": "Tenet03-eth2", "node": "urn:sdx:node:tenet.ac.za:Tenet03", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "name": "Tenet03-eth50", "node": "urn:sdx:node:tenet.ac.za:Tenet03", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}], "status": "up", "state": "enabled"}, {"name": "Tenet01", "id": "urn:sdx:node:tenet.ac.za:Tenet01", "location": {"address": "CapeTown", "latitude": -33.0, "longitude": 18.0, "iso3166_2_lvl4": "ZA-WC", "private": []}, "ports": [{"id": "urn:sdx:port:tenet.ac.za:Tenet01:41", "name": "Tenet01-eth41", "node": "urn:sdx:node:tenet.ac.za:Tenet01", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "urn:sdx:port:sax.net:Sax01:41", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet01:1", "name": "Tenet01-eth1", "node": "urn:sdx:node:tenet.ac.za:Tenet01", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet01:2", "name": "Tenet01-eth2", "node": "urn:sdx:node:tenet.ac.za:Tenet01", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}, {"id": "urn:sdx:port:tenet.ac.za:Tenet01:50", "name": "Tenet01-eth50", "node": "urn:sdx:node:tenet.ac.za:Tenet01", "type": "10GE", "status": "up", "state": "enabled", "mtu": 1500, "nni": "", "services": {"l2vpn-ptp": {"vlan_range": [[1, 4095]]}}, "private": ["status"]}], "status": "up", "state": "enabled"}], "links": [{"name": "Tenet01/1_Tenet02/1", "id": "urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1", "ports": ["urn:sdx:port:tenet.ac.za:Tenet02:1", "urn:sdx:port:tenet.ac.za:Tenet01:1"], "type": "intra", "bandwidth": 10, "residual_bandwidth": 100, "latency": 0, "packet_loss": 0, "availability": 0, "status": "up", "state": "enabled", "private": ["packet_loss"]}, {"name": "Tenet01/2_Tenet03/2", "id": "urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2", "ports": ["urn:sdx:port:tenet.ac.za:Tenet03:2", "urn:sdx:port:tenet.ac.za:Tenet01:2"], "type": "intra", "bandwidth": 10, "residual_bandwidth": 100, "latency": 0, "packet_loss": 0, "availability": 0, "status": "up", "state": "enabled", "private": ["packet_loss"]}], "services": ["l2vpn-ptp"]}'
As you can see above, the first topology event does not contains the full topology. Other nodes and links are added on subsequent events. This is normal behavior and can happen in real-world scenarios (beside tests like the one above). Since the _update_vlan_tags_table_from_links
is only called when the topology is added, rather than when is added AND updated the problem above can happen.
Thanks, Italo, for this case, very good one! I added #123 to the meta issue. Sajith is in the process of managing these vlan related issues.
Hi,
After creating and removing a few VLANs (one of them being the case reported on https://github.com/atlanticwave-sdx/sdx-controller/issues/309), eventually we get the following error:
The request above was basically: