[PERF] [BUG] Potential _populate_cache Exhaustion on Large Switch Automation Changes

[sadsfae]

On latest 1.1.7 QUADS I am sometimes seeing the following traceback after processing a large amount of switchport changes via /opt/quads/quads/tools/verify_switchconf.py

This will correct itself after subsequent runs but wanted to open a tracking issue. We might look into the _populate_cache function of the python MongoEngine library and see if there's a way to increase it.

This is likely less important going towards QUADS 2.0 where we're switching to PostgreSQL and SQLAlchemy ORM but worth investigating nonetheless. @grafuls

Traceback (most recent call last):
  File "/opt/quads/quads/tools/verify_switchconf.py", line 157, in <module>
    verify(args.cloud, args.host, args.change)
  File "/opt/quads/quads/tools/verify_switchconf.py", line 43, in verify
    for _host_obj in hosts:
  File "/usr/lib/python3.7/site-packages/mongoengine/queryset/queryset.py", line 99, in _iter_results
    self._populate_cache()
  File "/usr/lib/python3.7/site-packages/mongoengine/queryset/queryset.py", line 118, in _populate_cache
    self._result_cache.append(next(self))
  File "/usr/lib/python3.7/site-packages/mongoengine/queryset/base.py", line 1486, in __next__
    raw_doc = next(self._cursor)
  File "/usr/lib64/python3.7/site-packages/pymongo/cursor.py", line 1189, in next
    if len(self.__data) or self._refresh():
  File "/usr/lib64/python3.7/site-packages/pymongo/cursor.py", line 1126, in _refresh
    self.__send_message(g)
  File "/usr/lib64/python3.7/site-packages/pymongo/cursor.py", line 982, in __send_message
    helpers._check_command_response(first)
  File "/usr/lib64/python3.7/site-packages/pymongo/helpers.py", line 152, in _check_command_response
    raise CursorNotFound(errmsg, code, response)
pymongo.errors.CursorNotFound: cursor id 32073715851 not found
(root@quads) - (15:16) - (~)

[sadsfae]

Since the move to 2.0 and 2.1 coming shortly let's watch big moves and see if this still occurs in the logs.