mef_eline high concurrency EVC deletion can leak flows (leave them installed)

[viniarck]

Problem:

• When deleting an EVC, mef_eline makes a request to delete flows, but if the concurrency limit is achieved flow_manager will return 503 as expected. In this case mef_eline mef_eline was also returning a successful request response incorrectly.

How to reproduce:

• Create 100 EVCs and try to delete them at once concurrently.

Potential solution(s) to discuss:

• Handle and propagate the return back without deleting the EVC
• Use event based flows deletions instead of a request and make sure that no out of order events would be problematic

2024-07-24 11:16:40,793 - INFO [uvicorn.access] (MainThread) 127.0.0.1:60326 - "DELETE /api/kytos/mef_eline/v2/evc/3bbeb10cdbf448 HTTP/1.1" 200
2024-07-24 11:16:40,765 - ERROR [kytos.napps.kytos/mef_eline] (AnyIO worker thread) Error removing flows from switch 00:00:00:00:00:00:00:02 forEVC EVC(f340b351579043, evc28): Service 
Unavailable
2024-07-24 11:16:40,794 - INFO [uvicorn.access] (MainThread) 127.0.0.1:60448 - "DELETE /api/kytos/mef_eline/v2/evc/5db72196c3384a HTTP/1.1" 200
2024-07-24 11:16:40,765 - ERROR [kytos.napps.kytos/mef_eline] (AnyIO worker thread) Error removing flows from switch 00:00:00:00:00:00:00:02 forEVC EVC(e1f3f38809a444, evc63): Service 
Unavailable
2024-07-24 11:16:40,795 - INFO [uvicorn.access] (MainThread) 127.0.0.1:60386 - "DELETE /api/kytos/mef_eline/v2/evc/547b9c7ad51446 HTTP/1.1" 200
2024-07-24 11:16:40,767 - ERROR [kytos.napps.kytos/mef_eline] (AnyIO worker thread) Error removing flows from switch 00:00:00:00:00:00:00:02 forEVC EVC(e078a4a911bc4e, evc53): Service 
Unavailable

rs0 [direct: primary] napps> db.evcs.find({"active": true}).count()
0
rs0 [direct: primary] napps> db.evcs.find({"active": false}).count()
101
rs0 [direct: primary] napps> db.evcs.find({"archived": false}).count()
0
rs0 [direct: primary] napps> 

rs0 [direct: primary] napps> db.evcs.find({"archived": true}).count()
101
rs0 [direct: primary] napps> db.flows.find({"state": "installed", "flow.owner": "mef_eline"}).count()
579
rs0 [direct: primary] napps> 

Here's a full example of flows of an EVC that has been deleted but still has installed flows:

rs0 [direct: primary] napps> db.flows.find({"flow.cookie": Decimal128("12318260544696455235"), "flow.owner": "mef_eline"})
[
  {
    _id: '5221e2d8b660b94151acd3a56cad8b67',
    flow: {
      table_id: 0,
      owner: 'mef_eline',
      table_group: 'evpl',
      priority: 20000,
      cookie: Decimal128("12318260544696455235"),
      idle_timeout: 0,
      hard_timeout: 0,
      match: { in_port: 15, dl_vlan: 28 },
      actions: [
        { action_type: 'push_vlan', tag_type: 's' },
        { action_type: 'set_vlan', vlan_id: 25 },
        { action_type: 'output', port: 11 }
      ]
    },
    flow_id: '8d754da3570c455ba1210f85570e3bfd',
    id: '5221e2d8b660b94151acd3a56cad8b67',
    inserted_at: ISODate("2024-07-24T14:02:00.077Z"),
    state: 'installed',
    switch: '00:00:00:00:00:00:00:01',
    updated_at: ISODate("2024-07-24T14:02:00.302Z")
  },
  {
    _id: 'b3dc0b5ad473e40e6c73a19a4d6feb5c',
    flow: {
      table_id: 0,
      owner: 'mef_eline',
      table_group: 'evpl',
      priority: 20000,
      cookie: Decimal128("12318260544696455235"),
      idle_timeout: 0,
      hard_timeout: 0,
      match: { in_port: 11, dl_vlan: 25 },
      actions: [
        { action_type: 'pop_vlan' },
        { action_type: 'output', port: 15 }
      ]
    },
    flow_id: '3e20da49cf091688cfaad769f9e682c0',
    id: 'b3dc0b5ad473e40e6c73a19a4d6feb5c',
    inserted_at: ISODate("2024-07-24T14:02:00.077Z"),
    state: 'installed',
    switch: '00:00:00:00:00:00:00:01',
    updated_at: ISODate("2024-07-24T14:02:00.302Z")
  },
  {
    _id: 'c5eb456ff69daf08dc8791cd9ffd584b',
    flow: {
      table_id: 0,
      owner: 'mef_eline',
      table_group: 'evpl',
      priority: 20000,
      cookie: Decimal128("12318260544696455235"),
      idle_timeout: 0,
      hard_timeout: 0,
      match: { in_port: 22, dl_vlan: 28 },
      actions: [
        { action_type: 'push_vlan', tag_type: 's' },
        { action_type: 'set_vlan', vlan_id: 25 },
        { action_type: 'output', port: 11 }
      ]
    },
    flow_id: 'a28784c8d43dae78faa5a5a90da1ed54',
    id: 'c5eb456ff69daf08dc8791cd9ffd584b',
    inserted_at: ISODate("2024-07-24T14:02:00.171Z"),
    state: 'installed',
    switch: '00:00:00:00:00:00:00:06',
    updated_at: ISODate("2024-07-24T14:02:00.444Z")
  },
  {
    _id: '3750f80d3d992982fa7bcc1e69a0b992',
    flow: {
      table_id: 0,
      owner: 'mef_eline',
      table_group: 'evpl',
      priority: 20000,
      cookie: Decimal128("12318260544696455235"),
      idle_timeout: 0,
      hard_timeout: 0,
      match: { in_port: 11, dl_vlan: 25 },
      actions: [
        { action_type: 'pop_vlan' },
        { action_type: 'output', port: 22 }
      ]
    },
    flow_id: '00427e8c00c1efe902402c5d639818af',
    id: '3750f80d3d992982fa7bcc1e69a0b992',
    inserted_at: ISODate("2024-07-24T14:02:00.171Z"),
    state: 'installed',
    switch: '00:00:00:00:00:00:00:06',
    updated_at: ISODate("2024-07-24T14:02:00.444Z")
  },
  {
    _id: '978ab79b6d2b1b5d2b4973875b33e141',
    flow: {
      table_id: 0,
      owner: 'mef_eline',
      table_group: 'evpl',
      priority: 20000,
      cookie: Decimal128("12318260544696455235"),
      idle_timeout: 0,
      hard_timeout: 0,
      match: { in_port: 2, dl_vlan: 24 },
      actions: [
        { action_type: 'set_vlan', vlan_id: 24 },
        { action_type: 'output', port: 1 }
      ]
    },
    flow_id: 'd0aae95432c1b2e128af5cd242dd62af',
    id: '978ab79b6d2b1b5d2b4973875b33e141',
    inserted_at: ISODate("2024-07-24T14:02:00.513Z"),
    state: 'installed',
    switch: '00:00:00:00:00:00:00:02',
    updated_at: ISODate("2024-07-24T14:02:01.039Z")
  },
  {
    _id: '7d37e35dca5485d35009c008a251b913',
    flow: {
      table_id: 0,
      owner: 'mef_eline',
      table_group: 'evpl',
      priority: 20000,
      cookie: Decimal128("12318260544696455235"),
      idle_timeout: 0,
      hard_timeout: 0,
      match: { in_port: 1, dl_vlan: 24 },
      actions: [
        { action_type: 'set_vlan', vlan_id: 24 },
        { action_type: 'output', port: 2 }
      ]
    },
    flow_id: '801eab507325c0e83ed18e1250f4945b',
    id: '7d37e35dca5485d35009c008a251b913',
    inserted_at: ISODate("2024-07-24T14:02:00.513Z"),
    state: 'installed',
    switch: '00:00:00:00:00:00:00:02',
    updated_at: ISODate("2024-07-24T14:02:01.039Z")
  },
  {
    _id: '7ac48e12681eea583608006f69c09f7c',
    flow: {
      table_id: 0,
      owner: 'mef_eline',
      table_group: 'evpl',
      priority: 20000,
      cookie: Decimal128("12318260544696455235"),
      idle_timeout: 0,
      hard_timeout: 0,
      match: { in_port: 2, dl_vlan: 24 },
      actions: [
        { action_type: 'pop_vlan' },
        { action_type: 'output', port: 15 }
      ]
    },
    flow_id: '6ad907279cc7ba83d1a8130992ab65c6',
    id: '7ac48e12681eea583608006f69c09f7c',
    inserted_at: ISODate("2024-07-24T14:02:01.082Z"),
    state: 'installed',
    switch: '00:00:00:00:00:00:00:01',
    updated_at: ISODate("2024-07-24T14:02:01.427Z")
  },
  {
    _id: '74394ac622ee705da66e77be3da259ca',
    flow: {
      table_id: 0,
      owner: 'mef_eline',
      table_group: 'evpl',
      priority: 20000,
      cookie: Decimal128("12318260544696455235"),
      idle_timeout: 0,
      hard_timeout: 0,
      match: { in_port: 5, dl_vlan: 24 },
      actions: [
        { action_type: 'pop_vlan' },
        { action_type: 'output', port: 22 }
      ]
    },
    flow_id: 'ebb6b4db63a6b3d690f71c0323558649',
    id: '74394ac622ee705da66e77be3da259ca',
    inserted_at: ISODate("2024-07-24T14:02:01.325Z"),
    state: 'installed',
    switch: '00:00:00:00:00:00:00:06',
    updated_at: ISODate("2024-07-24T14:02:01.587Z")
  }
]
rs0 [direct: primary] napps> db.evcs.find({"name": "evc28"})
[
  {
    _id: 'f340b351579043',
    active: false,
    archived: true,
    backup_links: [],
    backup_path: [],
    bandwidth: 0,
    circuit_scheduler: [],
    creation_time: ISODate("2024-07-24T14:01:59.000Z"),
    current_path: [],
    dynamic_backup_path: true,
    enabled: false,
    execution_rounds: 0,
    failover_path: [],
    id: 'f340b351579043',
    inserted_at: ISODate("2024-07-24T14:01:59.833Z"),
    metadata: {
      telemetry_request: {},
      telemetry: {
        enabled: false,
        status: 'DOWN',
        status_reason: [ 'disabled' ],
        status_updated_at: '2024-07-24T14:16:41'
      }
    },
    name: 'evc28',
    primary_constraints: {},
    primary_links: [],
    primary_path: [],
    queue_id: -1,
    request_time: ISODate("2024-07-24T14:01:59.000Z"),
    secondary_constraints: {},
    service_level: 0,
    start_date: ISODate("2024-07-24T14:01:59.000Z"),
    uni_a: {
      tag: { tag_type: 'vlan', value: 28 },
      interface_id: '00:00:00:00:00:00:00:01:15'
    },
    uni_z: {
      tag: { tag_type: 'vlan', value: 28 },
      interface_id: '00:00:00:00:00:00:00:06:22'
    },
    updated_at: ISODate("2024-07-24T14:16:40.957Z")
  }
]
rs0 [direct: primary] napps> 

[Alopalao]

This is issue is very related to issue #476. If this bug happens and EVCs are created, kytos will eventually create duplicated flows. If we deal with leftover flows and assure the complete deletion of them, we can close both issues.