search

theLaborInVain / kdm-manager-api

The API used by https://kdm-manager.com and related Kingdom Death: Monster utilities.
Other
3 stars 0 forks source link

MongoDB-org compatibility issues #80

Closed toconnell closed 8 months ago

toconnell commented 8 months ago

The first world run versus the new MongoDB threw a bunch of errors:

[2024-02-28 07:55:21] DEBUG:    Updated 'latest_settlement' asset in mdb.
[2024-02-28 07:55:22] ERROR:    exception caught while refreshing 'top_survivor_names' asset!
[2024-02-28 07:55:22] ERROR:    no such command: 'group', full error: {'ok': 0.0, 'errmsg': "no such command: 'group'", 'code': 59, 'codeName': 'CommandNotFound'}
Traceback (most recent call last):
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 220, in refresh_asset
    self.update_asset_dict(asset_dict)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 276, in update_asset_dict
    fresh_results = update_method()
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 1087, in top_survivor_names
    return self.get_top("survivors", "name")
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 633, in get_top
    results = utils.mdb[collection].group(
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/collection.py", line 2640, in group
    return self._command(sock_info, cmd, slave_ok,
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/collection.py", line 238, in _command
    return sock_info.command(
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/pool.py", line 683, in command
    return command(self, dbname, spec, slave_ok,
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/network.py", line 159, in command
    helpers._check_command_response(
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/helpers.py", line 160, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: no such command: 'group', full error: {'ok': 0.0, 'errmsg': "no such command: 'group'", 'code': 59, 'codeName': 'CommandNotFound'}
[2024-02-28 07:55:23] ERROR:    exception caught while refreshing 'top_settlement_names' asset!
[2024-02-28 07:55:23] ERROR:    no such command: 'group', full error: {'ok': 0.0, 'errmsg': "no such command: 'group'", 'code': 59, 'codeName': 'CommandNotFound'}
Traceback (most recent call last):
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 220, in refresh_asset
    self.update_asset_dict(asset_dict)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 276, in update_asset_dict
    fresh_results = update_method()
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 1090, in top_settlement_names
    return self.get_top("settlements", "name")
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 633, in get_top
    results = utils.mdb[collection].group(
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/collection.py", line 2640, in group
    return self._command(sock_info, cmd, slave_ok,
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/collection.py", line 238, in _command
    return sock_info.command(
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/pool.py", line 683, in command
    return command(self, dbname, spec, slave_ok,
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/network.py", line 159, in command
    helpers._check_command_response(
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/helpers.py", line 160, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: no such command: 'group', full error: {'ok': 0.0, 'errmsg': "no such command: 'group'", 'code': 59, 'codeName': 'CommandNotFound'}
[2024-02-28 07:55:24] ERROR:    exception caught while refreshing 'top_causes_of_death' asset!
[2024-02-28 07:55:24] ERROR:    no such command: 'group', full error: {'ok': 0.0, 'errmsg': "no such command: 'group'", 'code': 59, 'codeName': 'CommandNotFound'}
Traceback (most recent call last):
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 220, in refresh_asset
    self.update_asset_dict(asset_dict)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 276, in update_asset_dict
    fresh_results = update_method()
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 1093, in top_causes_of_death
    return self.get_top("survivors", "cause_of_death", limit=10)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 633, in get_top
    results = utils.mdb[collection].group(
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/collection.py", line 2640, in group
    return self._command(sock_info, cmd, slave_ok,
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/collection.py", line 238, in _command
    return sock_info.command(
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/pool.py", line 683, in command
    return command(self, dbname, spec, slave_ok,
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/network.py", line 159, in command
    helpers._check_command_response(
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.8/site-packages/pymongo/helpers.py", line 160, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: no such command: 'group', full error: {'ok': 0.0, 'errmsg': "no such command: 'group'", 'code': 59, 'codeName': 'CommandNotFound'}
[2024-02-28 07:55:25] DEBUG:    Updated 'top_innovations' asset in mdb.
[2024-02-28 07:55:26] DEBUG:    Updated 'principle_selection_rates' asset in mdb.
[2024-02-28 07:55:29] DEBUG:    Updated 'settlement_popularity_contest_expansions' asset in mdb.
[2024-02-28 07:55:30] WARNING:  Attempting to initialize app.assets.campaigns asset from handle 'People of the Lantern' which does not appear to be an asset handle!
[2024-02-28 07:55:30] ERROR:    exception caught while refreshing 'settlement_popularity_contest_campaigns' asset!
[2024-02-28 07:55:30] ERROR:    local variable 'c_name' referenced before assignment
Traceback (most recent call last):
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 220, in refresh_asset
    self.update_asset_dict(asset_dict)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 276, in update_asset_dict
    fresh_results = update_method()
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 1228, in settlement_popularity_contest_campaigns
    if c_name in list(popularity_contest.keys()):
UnboundLocalError: local variable 'c_name' referenced before assignment
[2024-02-28 07:55:31] DEBUG:    Updated 'current_hunt' asset in mdb.
[2024-02-28 07:55:32] DEBUG:    Updated 'new_users_last_30' asset in mdb.
[2024-02-28 07:55:36] INFO:     Refreshed 54/54 assets.
toconnell commented 8 months ago

Rewrote the world/init.py get_top method to use a pipeline and remove some ugly post-query sorting:

-            results = utils.mdb[collection].group(
-                [attrib], query, {"count": 0}, "function(o, p){p.count++}"
-            )
+#            results = utils.mdb[collection].group(
+#                [attrib], query, {"count": 0}, "function(o, p){p.count++}"
+#            )
+
+            pipeline = [
+                {
+                    '$match': query  # Add a $match stage if necessary based on your existing 'query'
+                },
+                {
+                    '$group': {
+                        '_id': '$' + attrib,  # Group by the 'attrib' field
+                        'count': {'$sum': 1}  # Increment count for each group
+                    }
+                },
+                {
+                    '$sort': {'count': -1}  # Sort by 'count' field in descending order
+                }
+            ]
+
+            sorted_list = list(utils.mdb[collection].aggregate(pipeline))
+            if self.query_debug:
+                self.logger.debug('MDB results: %s', sorted_list)

-            sorted_list = sorted(results, key=lambda k: k["count"], reverse=True)
-            for i in sorted_list:
-                i["value"] = i[attrib]
-                i["count"] = int(i["count"])
+#            sorted_list = sorted(results, key=lambda k: k["count"], reverse=True)
+#            for i in sorted_list:
+#                i["value"] = i[attrib]
+#                i["count"] = int(i["count"])
toconnell commented 8 months ago

Methods to validate:

toconnell commented 8 months ago

Validations look good; all assets refresh. This is resolved.