xme commented 5 months ago

Prerequisites

Please answer the following questions for yourself before submitting an issue.

[x] I am running the latest version
[x] I did read the README!
[x] I checked the documentation and found no answer
[x] I checked to make sure that this issue has not already been filed
[x] I'm reporting the issue to the correct repository (for multi-repository projects)
[x] I have read and checked all configs (with all optional parts)

Expected Behavior

MongoDB report is enabled but it crashes since I installed a new guest VM. Reports are not available.

Context

Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions. Operating system version, bitness, installed software versions, test sample details/hash/binary (if applicable).

Question	Answer
Git commit	d46b1352cf67f69fcf84cd470052a29e13cc23c7
OS version	Ubuntu 22.04, Windows 10 (guest)

Failure Logs

2024-04-30 11:11:24,012 [Task 28] [lib.cuckoo.core.plugins] ERROR: Failed to run the reporting module "MongoDB": BitField.__new__() missing 3 required positional arguments: 'type_name', 'object_info', and 'base_type'
Traceback (most recent call last):
  File "/opt/CAPEv2/utils/../lib/cuckoo/core/plugins.py", line 738, in process
    current.run(self.results)
  File "/opt/CAPEv2/utils/../modules/reporting/mongodb.py", line 110, in run
    report = get_json_document(results, self.analysis_path)
  File "/opt/CAPEv2/utils/../modules/reporting/report_doc.py", line 60, in get_json_document
    report = copy.deepcopy(results)
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 206, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.10/copy.py", line 265, in _reconstruct
    y = func(*args)
  File "/usr/lib/python3.10/copyreg.py", line 101, in __newobj__
    return cls.__new__(cls, *args)
TypeError: BitField.__new__() missing 3 required positional arguments: 'type_name', 'object_info', and 'base_type'

doomedraven commented 5 months ago

Hey, what is the version of mongodb and pymongo?

xme commented 5 months ago

Here we go!

# /usr/bin/mongod --version
db version v7.0.9
Build Info: {
    "version": "7.0.9",
    "gitVersion": "3ff3a3925c36ed277cf5eafca5495f2e3728dd67",
    "openSSLVersion": "OpenSSL 3.0.2 15 Mar 2022",
    "modules": [],
    "allocator": "tcmalloc",
    "environment": {
        "distmod": "ubuntu2204",
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}
# sudo -u cape poetry run pip3 list|grep mongo
pymongo                4.6.3

doomedraven commented 5 months ago

i guess is has something to do with v7 changes, im on 6.0.9. i just returned from PTO so i first need to sync with my dayjob and then i will try to see what changed

doomedraven commented 5 months ago

i just installed the same version, i don-t have any problem with 7.0.9, do you some something custom?

doomedraven commented 5 months ago

FYI i do have problems, different but works on pymongo 4.5.0 but fails on latest 4.7.2, fixed my problem with 4.7.2. if you don-t have any custom stuff, can you share sample that generates that problem?

xme commented 5 months ago

Hi, sorry for the late reply, also busy on my side. I upgraded pymongo to 4.7.2 but still the same. It's not related to a sample, it's for all of them :(

[Edit] I'm running MongoDB 7.0.9 and pymongo 4.5.0 like you. Now, I've this error:

2024-05-14 21:19:54,703 [Task 34] [lib.cuckoo.core.plugins] ERROR: Failed to run the reporting module "MongoDB":
Traceback (most recent call last):
  File "/opt/CAPEv2/utils/../lib/cuckoo/core/plugins.py", line 738, in process
    current.run(self.results)
  File "/opt/CAPEv2/utils/../modules/reporting/mongodb.py", line 110, in run
    report = get_json_document(results, self.analysis_path)
  File "/opt/CAPEv2/utils/../modules/reporting/report_doc.py", line 60, in get_json_document
    report = copy.deepcopy(results)
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 206, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.10/copy.py", line 161, in deepcopy
    rv = reductor(4)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/volatility3/framework/objects/__init__.py", line 184, in __getnewargs_ex__
    kwargs["new_value"] = self.__new_value
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/volatility3/framework/interfaces/objects.py", line 136, in __getattr__
    raise AttributeError
AttributeError

[Edit2] I tried a full upgrade of all packages & dependencies. Now on pymongo 4.7.2 and got this error: CAPE does not start at all:

May 14 21:31:45 cape python3[1085227]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/logger.py", line 96, in _info_log
May 14 21:31:45 cape python3[1085227]:     logger.info(LogMessage(**fields))
May 14 21:31:45 cape python3[1085227]:   File "/usr/lib/python3.10/logging/__init__.py", line 1477, in info
May 14 21:31:45 cape python3[1085227]:     self._log(INFO, msg, args, **kwargs)
May 14 21:31:45 cape python3[1085227]:   File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
May 14 21:31:45 cape python3[1085227]:     self.handle(record)
May 14 21:31:45 cape python3[1085227]:   File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
May 14 21:31:45 cape python3[1085227]:     self.callHandlers(record)
May 14 21:31:45 cape python3[1085227]:   File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
May 14 21:31:45 cape python3[1085227]:     hdlr.handle(record)
May 14 21:31:45 cape python3[1085227]:   File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
May 14 21:31:45 cape python3[1085227]:     self.emit(record)
May 14 21:31:45 cape python3[1085227]:   File "/opt/CAPEv2/lib/cuckoo/core/log.py", line 72, in emit
May 14 21:31:45 cape python3[1085227]:     if "analysis procedure completed" in record.msg:
May 14 21:31:45 cape python3[1085227]: TypeError: argument of type 'LogMessage' is not iterable

doomedraven commented 5 months ago

could you share configs? it should be an plugin thaat im not using, as it happens on results copy

xme commented 5 months ago

In the meantime, I disabled "memory" in reporting and can manually regenerate the reports via process.py. But CAPE does not start (always the "LogMessage" error. It seems to be related to indexed in Mongodb!?

May 14 21:05:39 cape python3[1057758]: Traceback (most recent call last):
May 14 21:05:39 cape python3[1057758]:   File "/opt/CAPEv2/cuckoo.py", line 137, in <module>
May 14 21:05:39 cape python3[1057758]:     cuckoo_init(quiet=args.quiet, debug=args.debug, artwork=args.artwork, test=args.test)
May 14 21:05:39 cape python3[1057758]:   File "/opt/CAPEv2/cuckoo.py", line 79, in cuckoo_init
May 14 21:05:39 cape python3[1057758]:     check_webgui_mongo()
May 14 21:05:39 cape python3[1057758]:   File "/opt/CAPEv2/lib/cuckoo/core/startup.py", line 101, in check_webgui_mongo
May 14 21:05:39 cape python3[1057758]:     mongo_create_index("analysis", "info.id", name="info.id_1")
May 14 21:05:39 cape python3[1057758]:   File "/opt/CAPEv2/dev_utils/mongodb.py", line 55, in wrapper
May 14 21:05:39 cape python3[1057758]:     return mongo_op_func(*args, **kwargs)
May 14 21:05:39 cape python3[1057758]:   File "/opt/CAPEv2/dev_utils/mongodb.py", line 99, in mongo_create_index
May 14 21:05:39 cape python3[1057758]:     getattr(results_db, collection).create_index(index, background=background, name=name)
May 14 21:05:39 cape python3[1057758]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/collection.py", line 2120, in create_index
May 14 21:05:39 cape python3[1057758]:     return self.__create_indexes([index], session, **cmd_options)[0]
May 14 21:05:39 cape python3[1057758]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/_csot.py", line 108, in csot_wrapper
May 14 21:05:39 cape python3[1057758]:     return func(self, *args, **kwargs)
May 14 21:05:39 cape python3[1057758]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/collection.py", line 1973, in __create_indexes
May 14 21:05:39 cape python3[1057758]:     with self._conn_for_writes(session, operation=_Op.CREATE_INDEXES) as conn:
May 14 21:05:39 cape python3[1057758]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/collection.py", line 258, in _conn_for_writes
May 14 21:05:39 cape python3[1057758]:     return self.__database.client._conn_for_writes(session, operation)
May 14 21:05:39 cape python3[1057758]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1333, in _conn_for_writes
May 14 21:05:39 cape python3[1057758]:     server = self._select_server(writable_server_selector, session, operation)
May 14 21:05:39 cape python3[1057758]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1316, in _select_server
May 14 21:05:39 cape python3[1057758]:     server = topology.select_server(
May 14 21:05:39 cape python3[1057758]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/topology.py", line 369, in select_server
May 14 21:05:39 cape python3[1057758]:     server = self._select_server(
May 14 21:05:39 cape python3[1057758]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/topology.py", line 347, in _select_server
May 14 21:05:39 cape python3[1057758]:     servers = self.select_servers(
May 14 21:05:39 cape python3[1057758]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/topology.py", line 254, in select_servers
May 14 21:05:39 cape python3[1057758]:     server_descriptions = self._select_servers_loop(
May 14 21:05:39 cape python3[1057758]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/topology.py", line 309, in _select_servers_loop
May 14 21:05:39 cape python3[1057758]:     _info_log(
May 14 21:05:39 cape python3[1057758]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/logger.py", line 96, in _info_log
May 14 21:05:39 cape python3[1057758]:     logger.info(LogMessage(**fields))

conf.tgz

doomedraven commented 5 months ago

did you update your cape? i have fixed this logging last week when i upgraded pymongo. ok so the problem comes from memory, which i guess is volatility, is ages since i dont use that, so it will take some time to prepare everything for proper testing

xme commented 5 months ago

Yes, did a full upgrade yesterday.

doomedraven commented 5 months ago

can you try git pull again i just spot another place where that pymongo log was processed, now after git pull and systemctl restart cape-processor it should works

xme commented 5 months ago

I disabled some modules, apply the last update and it seems to work pretty nicely now! \o/ I will re-enable modules one by one and see which ones are messing up the whole setup. Tx for the great help!

doomedraven commented 5 months ago

thanks that will help to spot problematic one

alb3rt0-cyb3r commented 3 months ago

Hello, I've some problems with reporting and volatility. It seems that there is the same error log:

cape-error1

Hi, sorry for the late reply, also busy on my side. I upgraded pymongo to 4.7.2 but still the same. It's not related to a sample, it's for all of them :(

[Edit] I'm running MongoDB 7.0.9 and pymongo 4.5.0 like you. Now, I've this error:

2024-05-14 21:19:54,703 [Task 34] [lib.cuckoo.core.plugins] ERROR: Failed to run the reporting module "MongoDB":
Traceback (most recent call last):
  File "/opt/CAPEv2/utils/../lib/cuckoo/core/plugins.py", line 738, in process
    current.run(self.results)
  File "/opt/CAPEv2/utils/../modules/reporting/mongodb.py", line 110, in run
    report = get_json_document(results, self.analysis_path)
  File "/opt/CAPEv2/utils/../modules/reporting/report_doc.py", line 60, in get_json_document
    report = copy.deepcopy(results)
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 206, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.10/copy.py", line 161, in deepcopy
    rv = reductor(4)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/volatility3/framework/objects/__init__.py", line 184, in __getnewargs_ex__
    kwargs["new_value"] = self.__new_value
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/volatility3/framework/interfaces/objects.py", line 136, in __getattr__
    raise AttributeError
AttributeError

[Edit2] I tried a full upgrade of all packages & dependencies. Now on pymongo 4.7.2 and got this error: CAPE does not start at all:

May 14 21:31:45 cape python3[1085227]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pymongo/logger.py", line 96, in _info_log
May 14 21:31:45 cape python3[1085227]:     logger.info(LogMessage(**fields))
May 14 21:31:45 cape python3[1085227]:   File "/usr/lib/python3.10/logging/__init__.py", line 1477, in info
May 14 21:31:45 cape python3[1085227]:     self._log(INFO, msg, args, **kwargs)
May 14 21:31:45 cape python3[1085227]:   File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
May 14 21:31:45 cape python3[1085227]:     self.handle(record)
May 14 21:31:45 cape python3[1085227]:   File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
May 14 21:31:45 cape python3[1085227]:     self.callHandlers(record)
May 14 21:31:45 cape python3[1085227]:   File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
May 14 21:31:45 cape python3[1085227]:     hdlr.handle(record)
May 14 21:31:45 cape python3[1085227]:   File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
May 14 21:31:45 cape python3[1085227]:     self.emit(record)
May 14 21:31:45 cape python3[1085227]:   File "/opt/CAPEv2/lib/cuckoo/core/log.py", line 72, in emit
May 14 21:31:45 cape python3[1085227]:     if "analysis procedure completed" in record.msg:
May 14 21:31:45 cape python3[1085227]: TypeError: argument of type 'LogMessage' is not iterable

doomedraven commented 3 months ago

if you check last 3 lines from last code block, the code line there and in cape doesn't match https://github.com/kevoreilly/CAPEv2/blob/master/lib/cuckoo/core/log.py#L72

are you on latest cape?

kevoreilly / CAPEv2

MongoDB reporting exception? #2094

Prerequisites

Expected Behavior

Context

Failure Logs