bug: overwritten missing sink flow with multiple paths

[viniarck]

I stumbled upon this, when testing flows with both failover paths and also with backup_path, a common occurrence with multiple paths were certain sink flows, with certain svlans despite not having deletions were missing, so there's a unexpected overwritten overlap:

• 1) Either is a mistake in the mapping data transformations
• 2) Or indeed it's a conflict in the svlan with the external loop

Obs: I've also tested it with completely diff svlans on both paths and clans and the problem didn't manifest, so apparently it's related to svlan. I'll keep analyzing and I'll fix this when I have the chance to collect more evidence and data on INT Lab.

kytos $> 2024-06-13 13:23:05,183 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,184 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12118466772469484105, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 11}, 'instr
uctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-13 13:23:05,185 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,186 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12118466772469484105, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 11}, 'instr
uctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 22}]}]}]}
2024-06-13 13:23:05,186 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,187 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12118466772469484105, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 14}, 'instr
uctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-13 13:23:05,187 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,188 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12118466772469484105, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 14}, 'instr
uctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 22}]}]}]}
2024-06-13 13:23:05,188 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,190 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12118528141760518473, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 29}, 'instr
uctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-13 13:23:05,190 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,191 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12118528141760518473, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 29}, 'instr
uctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 22}]}]}]}
2024-06-13 13:23:05,191 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,192 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12118528141760518473, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 24}, 'instr
uctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-13 13:23:05,192 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,194 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12118528141760518473, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 24}, 'instr
uctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 22}]}]}]}
2024-06-13 13:23:05,194 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,195 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12126747551721894980, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 17}, 'instr
uctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-13 13:23:05,195 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,197 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12126747551721894980, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 17}, 'instr
uctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 22}]}]}]}
2024-06-13 13:23:05,197 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,198 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12127002169108930374, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 3}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-13 13:23:05,198 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,199 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12127002169108930374, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 3}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 22}]}]}]}
2024-06-13 13:23:05,199 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,200 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12140170741733595468, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 7}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-13 13:23:05,200 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,201 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12140170741733595468, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 7}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 22}]}]}]}
2024-06-13 13:23:05,202 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,203 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12140170741733595468, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 1}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-13 13:23:05,203 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,204 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12140170741733595468, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 1}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 22}]}]}]}
2024-06-13 13:23:05,204 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,205 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12147479697815360579, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 4}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-13 13:23:05,205 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,206 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12147479697815360579, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 4}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 22}]}]}]}
2024-06-13 13:23:05,207 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,208 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12154345372829405772, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 2}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-13 13:23:05,208 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Consistency check: missing flow on switch 00:00:00:00:00:00:00:06.
2024-06-13 13:23:05,209 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_30) Flow forwarded to switch 00:00:00:00:00:00:00:06 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12154345372829405772, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 26, 'dl_vlan': 2}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 22}]}]}]}

[viniarck]

In some cases, it was observed it was cycling, it can be a cookie issue with a similar match, I'll about to go a vacation, I'll continue from here when I come back:

2024-06-14 12:21:02,734 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_34) Consistency check: missing flow on switch 00:00:00:00:00:00:00:01.
2024-06-14 12:21:02,736 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_34) Flow forwarded to switch 00:00:00:00:00:00:00:01 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12122554414778792004, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 18, 'dl_vlan': 5}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-14 12:21:02,736 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_34) Consistency check: missing flow on switch 00:00:00:00:00:00:00:01.
2024-06-14 12:21:02,738 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_34) Flow forwarded to switch 00:00:00:00:00:00:00:01 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12122554414778792004, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 18, 'dl_vlan': 5}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 15}]}]}]}

2024-06-14 12:24:02,888 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_9) Consistency check: missing flow on switch 00:00:00:00:00:00:00:01.
2024-06-14 12:24:02,889 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_9) Flow forwarded to switch 00:00:00:00:00:00:00:01 to be installed. Flow: {'flows': [{'table_id': 0, 'o
wner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12122554414778792004, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 18, 'dl_vlan': 10}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-14 12:24:02,890 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_9) Consistency check: missing flow on switch 00:00:00:00:00:00:00:01.
2024-06-14 12:24:02,893 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_9) Flow forwarded to switch 00:00:00:00:00:00:00:01 to be installed. Flow: {'flows': [{'table_id': 2, 'o
wner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12122554414778792004, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 18, 'dl_vlan': 10}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 15}]}]}]}

2024-06-14 12:27:02,801 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_41) Consistency check: missing flow on switch 00:00:00:00:00:00:00:01.
2024-06-14 12:27:02,804 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_41) Flow forwarded to switch 00:00:00:00:00:00:00:01 to be installed. Flow: {'flows': [{'table_id': 0, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12122554414778792004, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 18, 'dl_vlan': 5}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'send_report'}]}, {'instruction_type': 'goto_table', 'table_id': 2}]}]}
2024-06-14 12:27:02,804 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_41) Consistency check: missing flow on switch 00:00:00:00:00:00:00:01.
2024-06-14 12:27:02,807 - INFO [kytos.napps.kytos/flow_manager] (thread_pool_sb_41) Flow forwarded to switch 00:00:00:00:00:00:00:01 to be installed. Flow: {'flows': [{'table_id': 2, '
owner': 'telemetry_int', 'table_group': 'evpl', 'priority': 20000, 'cookie': 12122554414778792004, 'idle_timeout': 0, 'hard_timeout': 0, 'match': {'in_port': 18, 'dl_vlan': 5}, 'instru
ctions': [{'instruction_type': 'apply_actions', 'actions': [{'action_type': 'pop_int'}, {'action_type': 'pop_vlan'}, {'action_type': 'output', 'port': 15}]}]}]}

[viniarck]

It turns out that the issue was indeed related to ii) "it's a conflict in the svlan with the external loop". The root cause problem is two-fold:

1) of_core and flow_manager were incorrectly including cookie in the match_id https://github.com/kytos-ng/of_core/issues/136, this will be fixed 2) The external loop "extends" the terminating end of an EVC that becomes the same incoming port, so when EVCs get extended to go other the loop, if for some reason different EVCs (different cookies) end with a similar s-vlan on either the primary or secondary path then they can overlap (this is rare to happen, but it can, I first saw this conflict when I was creating multiple EVCs), here's an example of sink flows from different EVCs, but they ended up allocating a similar s_vlan that was either in one of their primary or secondary paths (this is not a leak since indeed the the primary path and backup path might use the same s-vlan):

               +--------+                  
               |        |                  
               |        |                  
               |        |                  
           17  |     18 |                  
          +----+--------+----+             
          |                  | x (NNI)    
          |                  +-------------
          |                  |             
15 (UNI)  |      sw1         |             
----------+                  |             
          |                  | y (NNI)     
          |                  +-------------
          |                  |             
          +------------------+             

[
  {
    "priority": 20000,
    "cookie": 10,
    "table_id": 0,
    "match": {
      "in_port": 18,
      "dl_vlan": 3
    },
    "instructions": [
      {
        "instruction_type": "apply_actions",
        "actions": [
          {
            "action_type": "send_report"
          }
        ]
      },
      {
        "instruction_type": "goto_table",
        "table_id": 2
      }
    ]
  },
  {
    "priority": 20000,
    "cookie": 10,
    "table_id": 2,
    "match": {
      "in_port": 18,
      "dl_vlan": 3
    },
    "instructions": [
      {
        "instruction_type": "apply_actions",
        "actions": [
          {
            "action_type": "pop_int"
          },
          {
            "action_type": "pop_vlan"
          },
          {
            "action_type": "output",
            "port": 15
          }
        ]
      }
    ]
  },
  {
    "priority": 20000,
    "cookie": 20,
    "table_id": 0,
    "match": {
      "in_port": 18,
      "dl_vlan": 3
    },
    "instructions": [
      {
        "instruction_type": "apply_actions",
        "actions": [
          {
            "action_type": "send_report"
          }
        ]
      },
      {
        "instruction_type": "goto_table",
        "table_id": 2
      }
    ]
  },
  {
    "priority": 20000,
    "cookie": 20,
    "table_id": 2,
    "match": {
      "in_port": 18,
      "dl_vlan": 3
    },
    "instructions": [
      {
        "instruction_type": "apply_actions",
        "actions": [
          {
            "action_type": "pop_int"
          },
          {
            "action_type": "pop_vlan"
          },
          {
            "action_type": "output",
            "port": 15
          }
        ]
      }
    ]
  }
]

Notice above that the only difference is the cookie, but these flows overlap, in this case this external loop was for the same UNI, but if this were used to a different UNI the output port would be different and consequently it would break the current path of one of the EVCs. So, the current spec has this major restriction that in the worst case you can only use external loops for a different UNIs if your EVCs have multiple paths, unless you always use disjoint svlans in the paths (which for a production network it might not always be viable).

In prod will we able able to afford to use an external loop for each UNI or will they be shared? If they're shared we'll need to reassess the options here.