bug: `pymongo.errors.InvalidOperation: No operations to execute` when trying to enable 200+ EVCs at once

viniarck commented 3 months ago

It manifested when I tried to enable 200+ EVCs at once, I'll investigate, this is a well known bug on https://github.com/kytos-ng/flow_manager/issues/104, but telemetry_int shouldn't be sending an empty flow list, so I'll investigate the root cause to see if it was a concurrency issue or data issue when getting/mapping the flows:

2024-06-12 12:46:37,441 - ERROR [kytos.core.helpers] (thread_pool_app_2) listen_to handler: <function Main.on_flows_install_delete at 0x7fcc745d8180>, args: (<Main(flow_manager, stoppe
d 140511792850624)>, KytosEvent('kytos.flow_manager.flows.install', {'dpid': '00:00:00:00:00:00:00:02', 'force': True, 'flow_dict': {'flows': []}}, 0)) traceback: Traceback (most recen
t call last):,   File "/home/viniarck/repos/kytos/kytos/core/helpers.py", line 141, in handler_context,     result = handler(*args),              ^^^^^^^^^^^^^^,   File "/home/viniarck
/repos/napps/napps/kytos/flow_manager/main.py", line 558, in on_flows_install_delete,     self.handle_flows_install_delete(event),   File "/home/viniarck/repos/napps/napps/kytos/flow_m
anager/main.py", line 595, in handle_flows_install_delete,     self._install_flows(command, flow_dict, [switch], reraise_conn=not force),   File "/home/viniarck/repos/napps/napps/kytos
/flow_manager/main.py", line 732, in _install_flows,     self.flow_controller.upsert_flows(,   File "/home/viniarck/repos/kytos/.direnv/python-3.11/lib/python3.11/site-packages/tenacit
y/__init__.py", line 289, in wrapped_f,     return self(f, *args, **kw),            ^^^^^^^^^^^^^^^^^^^^,   File "/home/viniarck/repos/kytos/.direnv/python-3.11/lib/python3.11/site-pac
kages/tenacity/__init__.py", line 379, in __call__,     do = self.iter(retry_state=retry_state),          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^,   File "/home/viniarck/repos/kytos/.direnv
/python-3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 314, in iter,     return fut.result(),            ^^^^^^^^^^^^,   File "/usr/lib/python3.11/concurrent/futures/_ba
se.py", line 449, in result,     return self.__get_result(),            ^^^^^^^^^^^^^^^^^^^,   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result,     ra
ise self._exception,   File "/home/viniarck/repos/kytos/.direnv/python-3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 382, in __call__,     result = fn(*args, **kwargs),
              ^^^^^^^^^^^^^^^^^^^,   File "/home/viniarck/repos/kytos/kytos/core/retry.py", line 25, in decorated,     return func(*args, **kwargs),            ^^^^^^^^^^^^^^^^^^^^^,  
 File "/home/viniarck/repos/napps/napps/kytos/flow_manager/controllers/__init__.py", line 99, in upsert_flows,     return self.db.flows.bulk_write(ops).upserted_ids,            ^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^,   File "/home/viniarck/repos/kytos/.direnv/python-3.11/lib/python3.11/site-packages/pymongo/_csot.py", line 107, in csot_wrapper,     return func(self, *args, *
*kwargs),            ^^^^^^^^^^^^^^^^^^^^^^^^^^^,   File "/home/viniarck/repos/kytos/.direnv/python-3.11/lib/python3.11/site-packages/pymongo/collection.py", line 569, in bulk_write,  
   bulk_api_result = blk.execute(write_concern, session),                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^,   File "/home/viniarck/repos/kytos/.direnv/python-3.11/lib/python3.
11/site-packages/pymongo/bulk.py", line 558, in execute,     raise InvalidOperation("No operations to execute"), pymongo.errors.InvalidOperation: No operations to execute,

When enabling these 200+ in batches of 10, it didn't reproduce, it might be concurrency related. We'll see.

viniarck commented 3 months ago

I've found the root causes were multiple things:

1) Empty payload when trying to remove flows before enabling
2) Flow manager sending flows via events is also susceptible to out of order events. We'll need another handler there with pool="dynamic_single" to keep it single threaded for NApps like telemetry that don't want the request overhead but also is sending deletions/installations with similar matches.

viniarck commented 3 months ago

The item number 2 has been fixed on this PR here https://github.com/kytos-ng/flow_manager/pull/186

viniarck commented 3 months ago

The item number 1 has been fixed on this PR here https://github.com/kytos-ng/telemetry_int/pull/104

kytos-ng / telemetry_int

bug: `pymongo.errors.InvalidOperation: No operations to execute` when trying to enable 200+ EVCs at once #105