search

kevoreilly / CAPEv2

Malware Configuration And Payload Extraction
https://capesandbox.com/analysis/
Other
2.02k stars 427 forks source link

capev2 physical machine sqlalchemy errors #2258

Closed marsomx closed 2 months ago

marsomx commented 4 months ago

Expected Behavior

lunch analysis to physical machine -> complete analysis -> reimage physical machine -> got result of analysis

Current Behavior

after i updated and upgraded my machine:
PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)"

and updated Capev2 lo last version, i got some errors due to sqlalchemy, after the analysis is completed and the image is deployed on physical machine.

Failure Information (for bugs)

this'is the log:

2024-08-02 08:48:18,095 [lib.cuckoo.core.guest] INFO: Task #41: Guest is running CAPE Agent 0.17 (id=physical01, ip=192.168.1.11)
2024-08-02 08:48:20,402 [lib.cuckoo.core.guest] INFO: Task #41: Uploading script files to guest (id=physical01, ip=192.168.1.11)
2024-08-02 08:48:28,948 [lib.cuckoo.core.resultserver] INFO: Task 41: Process 4832 (parent 1324): download.exe, path C:\Users\sam\AppData\Local\Temp\download.exe
2024-08-02 08:48:30,131 [lib.cuckoo.core.resultserver] INFO: Task 41: Process 8996 (parent 4832): RegSvcs.exe, path C:\Windows\Microsoft.NET\Framework\v4.0.30319\RegSvcs.exe
2024-08-02 08:51:56,073 [lib.cuckoo.core.guest] INFO: Task #41: Analysis completed successfully (id=physical01, ip=192.168.1.101)
2024-08-02 08:51:56,222 [lib.cuckoo.core.analysis_manager] INFO: Task #41: Disabled route 'internet'
2024-08-02 09:05:21,001 [lib.cuckoo.core.analysis_manager] ERROR: Task #41: failure in AnalysisManager.run
Traceback (most recent call last):
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 496, in run
    self.launch_analysis()
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 458, in launch_analysis
    success = self.perform_analysis()
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 442, in perform_analysis
    with self.machine_running(), self.result_server(), self.network_routing(), self.run_auxiliary():
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 334, in machine_running
    self.machinery_manager.machinery.release(self.machine)
  File "/opt/CAPEv2/lib/cuckoo/common/abstracts.py", line 270, in release
    return self.db.unlock_machine(machine)
  File "/opt/CAPEv2/lib/cuckoo/core/database.py", line 978, in unlock_machine
    self.session.add(machine)
  File "<string>", line 2, in add
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2648, in add
    self._save_or_update_state(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2672, in _save_or_update_state
    self._save_or_update_impl(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3289, in _save_or_update_impl
    self._update_impl(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3278, in _update_impl
    self.identity_map.add(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/identity.py", line 151, in add
    raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Can't attach instance <Machine at 0x7617691b01f0>; another instance with key (<class 'lib.cuckoo.core.database.Machine'>, (89,), None) is already present in this session.

Steps to Reproduce

Please provide detailed steps for reproducing the issue.

  1. update and upgrade Ubuntu 22.04.4 LTS
  2. update capev2 to last version
  3. lunch analysis on physical machine
  4. complete analysis and re-image physical machine (automatic task)
  5. error

Context

i noticed some issue due to sqlalchemy version. anyway i checked all packages are syncronized and sqlalchemy version is

name : sqlalchemy
version : 1.4.50
description : Database Abstraction Library

dependencies

required by

thanks in advance for support

marsomx commented 3 months ago

also tried to clean all tasks and samples but got same errors

2024-08-03 12:14:19,171 [lib.cuckoo.core.analysis_manager] ERROR: Task #1: failure in AnalysisManager.run
Traceback (most recent call last):
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 496, in run
    self.launch_analysis()
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 458, in launch_analysis
    success = self.perform_analysis()
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 442, in perform_analysis
    with self.machine_running(), self.result_server(), self.network_routing(), self.run_auxiliary():
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 334, in machine_running
    self.machinery_manager.machinery.release(self.machine)
  File "/opt/CAPEv2/lib/cuckoo/common/abstracts.py", line 270, in release
    return self.db.unlock_machine(machine)
  File "/opt/CAPEv2/lib/cuckoo/core/database.py", line 978, in unlock_machine
    self.session.add(machine)
  File "<string>", line 2, in add
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2648, in add
    self._save_or_update_state(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2672, in _save_or_update_state
    self._save_or_update_impl(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3289, in _save_or_update_impl
    self._update_impl(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3278, in _update_impl
    self.identity_map.add(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/identity.py", line 151, in add
    raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Can't attach instance <Machine at 0x77a5bc1c63e0>; another instance with key (<class 'lib.cuckoo.core.database.Machine'>, (3,), None) is already present in this session.
doomedraven commented 3 months ago

Enable SQL logging in cuckoo.conf, and rerun the command, post the output, but plz use markdown code escale for that blov

marsomx commented 3 months ago

Enable SQL logging in cuckoo.conf, and rerun the command, post the output, but plz use markdown code escale for that blov

thanks for reply.. here the log: capev2-sqldebug.txt below last part of the log 


2024-08-03 15:45:08,787 [sqlalchemy.engine.Engine] INFO: SELECT tasks.id AS tasks_id, tasks.target AS tasks_target, tasks.category AS tasks_category, tasks.cape AS tasks_cape, tasks.timeout AS tasks_timeout, tasks.priority AS tasks_priority, tasks.custom AS tasks_custom, tasks.machine AS tasks_machine, tasks.package AS tasks_package, tasks.route AS tasks_route, tasks.tags_tasks AS tasks_tags_tasks, tasks.options AS tasks_options, tasks.platform AS tasks_platform, tasks.memory AS tasks_memory, tasks.enforce_timeout AS tasks_enforce_timeout, tasks.clock AS tasks_clock, tasks.added_on AS tasks_added_on, tasks.started_on AS tasks_started_on, tasks.completed_on AS tasks_completed_on, tasks.status AS tasks_status, tasks.dropped_files AS tasks_dropped_files, tasks.running_processes AS tasks_running_processes, tasks.api_calls AS tasks_api_calls, tasks.domains AS tasks_domains, tasks.signatures_total AS tasks_signatures_total, tasks.signatures_alert AS tasks_signatures_alert, tasks.files_written AS tasks_files_written, tasks.registry_keys_modified AS tasks_registry_keys_modified, tasks.crash_issues AS tasks_crash_issues, tasks.anti_issues AS tasks_anti_issues, tasks.analysis_started_on AS tasks_analysis_started_on, tasks.analysis_finished_on AS tasks_analysis_finished_on, tasks.processing_started_on AS tasks_processing_started_on, tasks.processing_finished_on AS tasks_processing_finished_on, tasks.signatures_started_on AS tasks_signatures_started_on, tasks.signatures_finished_on AS tasks_signatures_finished_on, tasks.reporting_started_on AS tasks_reporting_started_on, tasks.reporting_finished_on AS tasks_reporting_finished_on, tasks.timedout AS tasks_timedout, tasks.sample_id AS tasks_sample_id, tasks.machine_id AS tasks_machine_id, tasks.shrike_url AS tasks_shrike_url, tasks.shrike_refer AS tasks_shrike_refer, tasks.shrike_msg AS tasks_shrike_msg, tasks.shrike_sid AS tasks_shrike_sid, tasks.parent_id AS tasks_parent_id, tasks.tlp AS tasks_tlp, tasks.user_id AS tasks_user_id, tasks.username AS tasks_username, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name, guests_1.id AS guests_1_id, guests_1.status AS guests_1_status, guests_1.name AS guests_1_name, guests_1.label AS guests_1_label, guests_1.platform AS guests_1_platform, guests_1.manager AS guests_1_manager, guests_1.started_on AS guests_1_started_on, guests_1.shutdown_on AS guests_1_shutdown_on, guests_1.task_id AS guests_1_task_id, errors_1.id AS errors_1_id, errors_1.message AS errors_1_message, errors_1.task_id AS errors_1_task_id 
FROM tasks LEFT OUTER JOIN (tasks_tags AS tasks_tags_1 JOIN tags AS tags_1 ON tags_1.id = tasks_tags_1.tag_id) ON tasks.id = tasks_tags_1.task_id LEFT OUTER JOIN guests AS guests_1 ON tasks.id = guests_1.task_id LEFT OUTER JOIN errors AS errors_1 ON tasks.id = errors_1.task_id 
WHERE tasks.status = %(status_1)s AND tasks.options NOT LIKE %(options_1)s ORDER BY tasks.priority DESC, tasks.added_on FOR UPDATE OF tasks
2024-08-03 15:45:08,788 INFO sqlalchemy.engine.Engine [cached since 1057s ago] {'status_1': 'pending', 'options_1': '%node=%'}
2024-08-03 15:45:08,788 [sqlalchemy.engine.Engine] INFO: [cached since 1057s ago] {'status_1': 'pending', 'options_1': '%node=%'}
2024-08-03 15:45:08,795 INFO sqlalchemy.engine.Engine COMMIT
2024-08-03 15:45:08,795 [sqlalchemy.engine.Engine] INFO: COMMIT
{"message": "Analysis status", "status": "init", "description": ""}
2024-08-03 15:45:09,353 INFO sqlalchemy.engine.Engine COMMIT
2024-08-03 15:45:09,353 [sqlalchemy.engine.Engine] INFO: COMMIT
2024-08-03 15:45:09,356 [lib.cuckoo.core.analysis_manager] ERROR: Task #2: failure in AnalysisManager.run
Traceback (most recent call last):
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 496, in run
    self.launch_analysis()
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 458, in launch_analysis
    success = self.perform_analysis()
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 442, in perform_analysis
    with self.machine_running(), self.result_server(), self.network_routing(), self.run_auxiliary():
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 334, in machine_running
    self.machinery_manager.machinery.release(self.machine)
  File "/opt/CAPEv2/lib/cuckoo/common/abstracts.py", line 270, in release
    return self.db.unlock_machine(machine)
  File "/opt/CAPEv2/lib/cuckoo/core/database.py", line 978, in unlock_machine
    self.session.add(machine)
  File "<string>", line 2, in add
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2648, in add
    self._save_or_update_state(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2672, in _save_or_update_state
    self._save_or_update_impl(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3289, in _save_or_update_impl
    self._update_impl(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3278, in _update_impl
    self.identity_map.add(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/identity.py", line 151, in add
    raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Can't attach instance <Machine at 0x71b1d40c2560>; another instance with key (<class 'lib.cuckoo.core.database.Machine'>, (5,), None) is already present in this session.
2024-08-03 15:45:09,799 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-08-03 15:45:09,799 [sqlalchemy.engine.Engine] INFO: BEGIN (implicit)
2024-08-03 15:45:09,799 INFO sqlalchemy.engine.Engine SELECT count(*) AS count_1 
FROM (SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved 
FROM machines 
WHERE machines.locked = true) AS anon_1
2024-08-03 15:45:09,799 [sqlalchemy.engine.Engine] INFO: SELECT count(*) AS count_1 
FROM (SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved 
FROM machines 
WHERE machines.locked = true) AS anon_1
2024-08-03 15:45:09,800 INFO sqlalchemy.engine.Engine [cached since 1058s ago] {}
2024-08-03 15:45:09,800 [sqlalchemy.engine.Engine] INFO: [cached since 1058s ago] {}
2024-08-03 15:45:09,801 INFO sqlalchemy.engine.Engine SELECT tasks.id AS tasks_id, tasks.target AS tasks_target, tasks.category AS tasks_category, tasks.cape AS tasks_cape, tasks.timeout AS tasks_timeout, tasks.priority AS tasks_priority, tasks.custom AS tasks_custom, tasks.machine AS tasks_machine, tasks.package AS tasks_package, tasks.route AS tasks_route, tasks.tags_tasks AS tasks_tags_tasks, tasks.options AS tasks_options, tasks.platform AS tasks_platform, tasks.memory AS tasks_memory, tasks.enforce_timeout AS tasks_enforce_timeout, tasks.clock AS tasks_clock, tasks.added_on AS tasks_added_on, tasks.started_on AS tasks_started_on, tasks.completed_on AS tasks_completed_on, tasks.status AS tasks_status, tasks.dropped_files AS tasks_dropped_files, tasks.running_processes AS tasks_running_processes, tasks.api_calls AS tasks_api_calls, tasks.domains AS tasks_domains, tasks.signatures_total AS tasks_signatures_total, tasks.signatures_alert AS tasks_signatures_alert, tasks.files_written AS tasks_files_written, tasks.registry_keys_modified AS tasks_registry_keys_modified, tasks.crash_issues AS tasks_crash_issues, tasks.anti_issues AS tasks_anti_issues, tasks.analysis_started_on AS tasks_analysis_started_on, tasks.analysis_finished_on AS tasks_analysis_finished_on, tasks.processing_started_on AS tasks_processing_started_on, tasks.processing_finished_on AS tasks_processing_finished_on, tasks.signatures_started_on AS tasks_signatures_started_on, tasks.signatures_finished_on AS tasks_signatures_finished_on, tasks.reporting_started_on AS tasks_reporting_started_on, tasks.reporting_finished_on AS tasks_reporting_finished_on, tasks.timedout AS tasks_timedout, tasks.sample_id AS tasks_sample_id, tasks.machine_id AS tasks_machine_id, tasks.shrike_url AS tasks_shrike_url, tasks.shrike_refer AS tasks_shrike_refer, tasks.shrike_msg AS tasks_shrike_msg, tasks.shrike_sid AS tasks_shrike_sid, tasks.parent_id AS tasks_parent_id, tasks.tlp AS tasks_tlp, tasks.user_id AS tasks_user_id, tasks.username AS tasks_username, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name, guests_1.id AS guests_1_id, guests_1.status AS guests_1_status, guests_1.name AS guests_1_name, guests_1.label AS guests_1_label, guests_1.platform AS guests_1_platform, guests_1.manager AS guests_1_manager, guests_1.started_on AS guests_1_started_on, guests_1.shutdown_on AS guests_1_shutdown_on, guests_1.task_id AS guests_1_task_id, errors_1.id AS errors_1_id, errors_1.message AS errors_1_message, errors_1.task_id AS errors_1_task_id 
FROM tasks LEFT OUTER JOIN (tasks_tags AS tasks_tags_1 JOIN tags AS tags_1 ON tags_1.id = tasks_tags_1.tag_id) ON tasks.id = tasks_tags_1.task_id LEFT OUTER JOIN guests AS guests_1 ON tasks.id = guests_1.task_id LEFT OUTER JOIN errors AS errors_1 ON tasks.id = errors_1.task_id 
WHERE tasks.status = %(status_1)s AND tasks.options NOT LIKE %(options_1)s ORDER BY tasks.priority DESC, tasks.added_on FOR UPDATE OF tasks
2024-08-03 15:45:09,801 [sqlalchemy.engine.Engine] INFO: SELECT tasks.id AS tasks_id, tasks.target AS tasks_target, tasks.category AS tasks_category, tasks.cape AS tasks_cape, tasks.timeout AS tasks_timeout, tasks.priority AS tasks_priority, tasks.custom AS tasks_custom, tasks.machine AS tasks_machine, tasks.package AS tasks_package, tasks.route AS tasks_route, tasks.tags_tasks AS tasks_tags_tasks, tasks.options AS tasks_options, tasks.platform AS tasks_platform, tasks.memory AS tasks_memory, tasks.enforce_timeout AS tasks_enforce_timeout, tasks.clock AS tasks_clock, tasks.added_on AS tasks_added_on, tasks.started_on AS tasks_started_on, tasks.completed_on AS tasks_completed_on, tasks.status AS tasks_status, tasks.dropped_files AS tasks_dropped_files, tasks.running_processes AS tasks_running_processes, tasks.api_calls AS tasks_api_calls, tasks.domains AS tasks_domains, tasks.signatures_total AS tasks_signatures_total, tasks.signatures_alert AS tasks_signatures_alert, tasks.files_written AS tasks_files_written, tasks.registry_keys_modified AS tasks_registry_keys_modified, tasks.crash_issues AS tasks_crash_issues, tasks.anti_issues AS tasks_anti_issues, tasks.analysis_started_on AS tasks_analysis_started_on, tasks.analysis_finished_on AS tasks_analysis_finished_on, tasks.processing_started_on AS tasks_processing_started_on, tasks.processing_finished_on AS tasks_processing_finished_on, tasks.signatures_started_on AS tasks_signatures_started_on, tasks.signatures_finished_on AS tasks_signatures_finished_on, tasks.reporting_started_on AS tasks_reporting_started_on, tasks.reporting_finished_on AS tasks_reporting_finished_on, tasks.timedout AS tasks_timedout, tasks.sample_id AS tasks_sample_id, tasks.machine_id AS tasks_machine_id, tasks.shrike_url AS tasks_shrike_url, tasks.shrike_refer AS tasks_shrike_refer, tasks.shrike_msg AS tasks_shrike_msg, tasks.shrike_sid AS tasks_shrike_sid, tasks.parent_id AS tasks_parent_id, tasks.tlp AS tasks_tlp, tasks.user_id AS tasks_user_id, tasks.username AS tasks_username, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name, guests_1.id AS guests_1_id, guests_1.status AS guests_1_status, guests_1.name AS guests_1_name, guests_1.label AS guests_1_label, guests_1.platform AS guests_1_platform, guests_1.manager AS guests_1_manager, guests_1.started_on AS guests_1_started_on, guests_1.shutdown_on AS guests_1_shutdown_on, guests_1.task_id AS guests_1_task_id, errors_1.id AS errors_1_id, errors_1.message AS errors_1_message, errors_1.task_id AS errors_1_task_id 
FROM tasks LEFT OUTER JOIN (tasks_tags AS tasks_tags_1 JOIN tags AS tags_1 ON tags_1.id = tasks_tags_1.tag_id) ON tasks.id = tasks_tags_1.task_id LEFT OUTER JOIN guests AS guests_1 ON tasks.id = guests_1.task_id LEFT OUTER JOIN errors AS errors_1 ON tasks.id = errors_1.task_id 
WHERE tasks.status = %(status_1)s AND tasks.options NOT LIKE %(options_1)s ORDER BY tasks.priority DESC, tasks.added_on FOR UPDATE OF tasks
2024-08-03 15:45:09,801 INFO sqlalchemy.engine.Engine [cached since 1058s ago] {'status_1': 'pending', 'options_1': '%node=%'}
2024-08-03 15:45:09,801 [sqlalchemy.engine.Engine] INFO: [cached since 1058s ago] {'status_1': 'pending', 'options_1': '%node=%'}
2024-08-03 15:45:09,804 INFO sqlalchemy.engine.Engine COMMIT
2024-08-03 15:45:09,804 [sqlalchemy.engine.Engine] INFO: COMMIT
2024-08-03 15:45:10,806 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-08-03 15:45:10,806 [sqlalchemy.engine.Engine] INFO: BEGIN (implicit)
2024-08-03 15:45:10,807 INFO sqlalchemy.engine.Engine SELECT count(*) AS count_1 
FROM (SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved 
FROM machines 
WHERE machines.locked = true) AS anon_1
2024-08-03 15:45:10,807 [sqlalchemy.engine.Engine] INFO: SELECT count(*) AS count_1 
FROM (SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved 
FROM machines 
WHERE machines.locked = true) AS anon_1
2024-08-03 15:45:10,807 INFO sqlalchemy.engine.Engine [cached since 1059s ago] {}
2024-08-03 15:45:10,807 [sqlalchemy.engine.Engine] INFO: [cached since 1059s ago] {}
2024-08-03 15:45:10,808 INFO sqlalchemy.engine.Engine SELECT tasks.id AS tasks_id, tasks.target AS tasks_target, tasks.category AS tasks_category, tasks.cape AS tasks_cape, tasks.timeout AS tasks_timeout, tasks.priority AS tasks_priority, tasks.custom AS tasks_custom, tasks.machine AS tasks_machine, tasks.package AS tasks_package, tasks.route AS tasks_route, tasks.tags_tasks AS tasks_tags_tasks, tasks.options AS tasks_options, tasks.platform AS tasks_platform, tasks.memory AS tasks_memory, tasks.enforce_timeout AS tasks_enforce_timeout, tasks.clock AS tasks_clock, tasks.added_on AS tasks_added_on, tasks.started_on AS tasks_started_on, tasks.completed_on AS tasks_completed_on, tasks.status AS tasks_status, tasks.dropped_files AS tasks_dropped_files, tasks.running_processes AS tasks_running_processes, tasks.api_calls AS tasks_api_calls, tasks.domains AS tasks_domains, tasks.signatures_total AS tasks_signatures_total, tasks.signatures_alert AS tasks_signatures_alert, tasks.files_written AS tasks_files_written, tasks.registry_keys_modified AS tasks_registry_keys_modified, tasks.crash_issues AS tasks_crash_issues, tasks.anti_issues AS tasks_anti_issues, tasks.analysis_started_on AS tasks_analysis_started_on, tasks.analysis_finished_on AS tasks_analysis_finished_on, tasks.processing_started_on AS tasks_processing_started_on, tasks.processing_finished_on AS tasks_processing_finished_on, tasks.signatures_started_on AS tasks_signatures_started_on, tasks.signatures_finished_on AS tasks_signatures_finished_on, tasks.reporting_started_on AS tasks_reporting_started_on, tasks.reporting_finished_on AS tasks_reporting_finished_on, tasks.timedout AS tasks_timedout, tasks.sample_id AS tasks_sample_id, tasks.machine_id AS tasks_machine_id, tasks.shrike_url AS tasks_shrike_url, tasks.shrike_refer AS tasks_shrike_refer, tasks.shrike_msg AS tasks_shrike_msg, tasks.shrike_sid AS tasks_shrike_sid, tasks.parent_id AS tasks_parent_id, tasks.tlp AS tasks_tlp, tasks.user_id AS tasks_user_id, tasks.username AS tasks_username, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name, guests_1.id AS guests_1_id, guests_1.status AS guests_1_status, guests_1.name AS guests_1_name, guests_1.label AS guests_1_label, guests_1.platform AS guests_1_platform, guests_1.manager AS guests_1_manager, guests_1.started_on AS guests_1_started_on, guests_1.shutdown_on AS guests_1_shutdown_on, guests_1.task_id AS guests_1_task_id, errors_1.id AS errors_1_id, errors_1.message AS errors_1_message, errors_1.task_id AS errors_1_task_id 
FROM tasks LEFT OUTER JOIN (tasks_tags AS tasks_tags_1 JOIN tags AS tags_1 ON tags_1.id = tasks_tags_1.tag_id) ON tasks.id = tasks_tags_1.task_id LEFT OUTER JOIN guests AS guests_1 ON tasks.id = guests_1.task_id LEFT OUTER JOIN errors AS errors_1 ON tasks.id = errors_1.task_id 
WHERE tasks.status = %(status_1)s AND tasks.options NOT LIKE %(options_1)s ORDER BY tasks.priority DESC, tasks.added_on FOR UPDATE OF tasks
2024-08-03 15:45:10,808 [sqlalchemy.engine.Engine] INFO: SELECT tasks.id AS tasks_id, tasks.target AS tasks_target, tasks.category AS tasks_category, tasks.cape AS tasks_cape, tasks.timeout AS tasks_timeout, tasks.priority AS tasks_priority, tasks.custom AS tasks_custom, tasks.machine AS tasks_machine, tasks.package AS tasks_package, tasks.route AS tasks_route, tasks.tags_tasks AS tasks_tags_tasks, tasks.options AS tasks_options, tasks.platform AS tasks_platform, tasks.memory AS tasks_memory, tasks.enforce_timeout AS tasks_enforce_timeout, tasks.clock AS tasks_clock, tasks.added_on AS tasks_added_on, tasks.started_on AS tasks_started_on, tasks.completed_on AS tasks_completed_on, tasks.status AS tasks_status, tasks.dropped_files AS tasks_dropped_files, tasks.running_processes AS tasks_running_processes, tasks.api_calls AS tasks_api_calls, tasks.domains AS tasks_domains, tasks.signatures_total AS tasks_signatures_total, tasks.signatures_alert AS tasks_signatures_alert, tasks.files_written AS tasks_files_written, tasks.registry_keys_modified AS tasks_registry_keys_modified, tasks.crash_issues AS tasks_crash_issues, tasks.anti_issues AS tasks_anti_issues, tasks.analysis_started_on AS tasks_analysis_started_on, tasks.analysis_finished_on AS tasks_analysis_finished_on, tasks.processing_started_on AS tasks_processing_started_on, tasks.processing_finished_on AS tasks_processing_finished_on, tasks.signatures_started_on AS tasks_signatures_started_on, tasks.signatures_finished_on AS tasks_signatures_finished_on, tasks.reporting_started_on AS tasks_reporting_started_on, tasks.reporting_finished_on AS tasks_reporting_finished_on, tasks.timedout AS tasks_timedout, tasks.sample_id AS tasks_sample_id, tasks.machine_id AS tasks_machine_id, tasks.shrike_url AS tasks_shrike_url, tasks.shrike_refer AS tasks_shrike_refer, tasks.shrike_msg AS tasks_shrike_msg, tasks.shrike_sid AS tasks_shrike_sid, tasks.parent_id AS tasks_parent_id, tasks.tlp AS tasks_tlp, tasks.user_id AS tasks_user_id, tasks.username AS tasks_username, tags_1.id AS tags_1_id, tags_1.name AS tags_1_name, guests_1.id AS guests_1_id, guests_1.status AS guests_1_status, guests_1.name AS guests_1_name, guests_1.label AS guests_1_label, guests_1.platform AS guests_1_platform, guests_1.manager AS guests_1_manager, guests_1.started_on AS guests_1_started_on, guests_1.shutdown_on AS guests_1_shutdown_on, guests_1.task_id AS guests_1_task_id, errors_1.id AS errors_1_id, errors_1.message AS errors_1_message, errors_1.task_id AS errors_1_task_id 
FROM tasks LEFT OUTER JOIN (tasks_tags AS tasks_tags_1 JOIN tags AS tags_1 ON tags_1.id = tasks_tags_1.tag_id) ON tasks.id = tasks_tags_1.task_id LEFT OUTER JOIN guests AS guests_1 ON tasks.id = guests_1.task_id LEFT OUTER JOIN errors AS errors_1 ON tasks.id = errors_1.task_id 
WHERE tasks.status = %(status_1)s AND tasks.options NOT LIKE %(options_1)s ORDER BY tasks.priority DESC, tasks.added_on FOR UPDATE OF tasks
2024-08-03 15:45:10,808 INFO sqlalchemy.engine.Engine [cached since 1059s ago] {'status_1': 'pending', 'options_1': '%node=%'}
2024-08-03 15:45:10,808 [sqlalchemy.engine.Engine] INFO: [cached since 1059s ago] {'status_1': 'pending', 'options_1': '%node=%'}
2024-08-03 15:45:10,810 INFO sqlalchemy.engine.Engine COMMIT
2024-08-03 15:45:10,810 [sqlalchemy.engine.Engine] INFO: COMMIT
2024-08-03 15:45:11,812 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-08-03 15:45:11,812 [sqlalchemy.engine.Engine] INFO: BEGIN (implicit)
2024-08-03 15:45:11,813 INFO sqlalchemy.engine.Engine SELECT count(*) AS count_1 
FROM (SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved 
FROM machines 
WHERE machines.locked = true) AS anon_1
2024-08-03 15:45:11,813 [sqlalchemy.engine.Engine] INFO: SELECT count(*) AS count_1 
FROM (SELECT machines.id AS machines_id, machines.name AS machines_name, machines.label AS machines_label, machines.arch AS machines_arch, machines.ip AS machines_ip, machines.platform AS machines_platform, machines.interface AS machines_interface, machines.snapshot AS machines_snapshot, machines.locked AS machines_locked, machines.locked_changed_on AS machines_locked_changed_on, machines.status AS machines_status, machines.status_changed_on AS machines_status_changed_on, machines.resultserver_ip AS machines_resultserver_ip, machines.resultserver_port AS machines_resultserver_port, machines.reserved AS machines_reserved 
FROM machines 
WHERE machines.locked = true) AS anon_1
marsomx commented 3 months ago

@doomedraven i updated os and cape and errors due to sqlalchemy seem to be fixed. unfortunately i got another error, Basically the analysis started as expected but after few seconds, task turned in failed and physical machine was rebooted. This is the log of the analysis task on web gui:

2024-08-22 13:31:13,357 [root] INFO: Date set to: 20240822T13:31:14, timeout set to: 200
2024-08-22 13:31:14,000 [root] DEBUG: Starting analyzer from: C:\tmp8sz0jlcw
2024-08-22 13:31:14,000 [root] DEBUG: Storing results at: C:\EJVzYsIz
2024-08-22 13:31:14,000 [root] DEBUG: Pipe server name: \\.\PIPE\QNUdrqGaF
2024-08-22 13:31:14,000 [root] DEBUG: Python path: C:\Users\sam\AppData\Local\Programs\Python\Python310-32
2024-08-22 13:31:14,000 [root] INFO: analysis running as an admin
2024-08-22 13:31:14,000 [root] INFO: analysis package specified: "exe"
2024-08-22 13:31:14,000 [root] DEBUG: importing analysis package module: "modules.packages.exe"...
2024-08-22 13:31:14,010 [root] DEBUG: imported analysis package "exe"
2024-08-22 13:31:14,010 [root] DEBUG: initializing analysis package "exe"...
2024-08-22 13:31:14,010 [lib.common.common] INFO: wrapping
2024-08-22 13:31:14,010 [lib.core.compound] INFO: C:\Users\sam\AppData\Local\Temp already exists, skipping creation
2024-08-22 13:31:14,010 [root] DEBUG: New location of moved file: C:\Users\sam\AppData\Local\Temp\2cdf95d8ff803328ea77.exe
2024-08-22 13:31:14,010 [root] INFO: Analyzer: Package modules.packages.exe does not specify a DLL option
2024-08-22 13:31:14,010 [root] INFO: Analyzer: Package modules.packages.exe does not specify a DLL_64 option
2024-08-22 13:31:14,010 [root] INFO: Analyzer: Package modules.packages.exe does not specify a loader option
2024-08-22 13:31:14,010 [root] INFO: Analyzer: Package modules.packages.exe does not specify a loader_64 option
2024-08-22 13:31:14,010 [root] DEBUG: Importing auxiliary module "modules.auxiliary.DNS_ETW"...
2024-08-22 13:31:14,026 [modules.auxiliary.DNS_ETW] DEBUG: Could not load auxiliary module DNS_ETW due to 'No module named 'etw''
2024-08-22 13:31:14,026 [root] ERROR: Traceback (most recent call last):
  File "C:\tmp8sz0jlcw\modules\auxiliary\DNS_ETW.py", line 17, in <module>
    from etw import ETW, ProviderInfo
ModuleNotFoundError: No module named 'etw'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\tmp8sz0jlcw\analyzer.py", line 1524, in <module>
    success = analyzer.run()
  File "C:\tmp8sz0jlcw\analyzer.py", line 507, in run
    __import__(name, globals(), locals(), ["dummy"])
  File "C:\tmp8sz0jlcw\modules\auxiliary\DNS_ETW.py", line 22, in <module>
    raise CuckooPackageError("In order to use DNS_ETW functionality, it " "is required to have pywintrace setup in python.")
lib.common.exceptions.CuckooPackageError: In order to use DNS_ETW functionality, it is required to have pywintrace setup in python.
Traceback (most recent call last):
  File "C:\tmp8sz0jlcw\modules\auxiliary\DNS_ETW.py", line 17, in <module>
    from etw import ETW, ProviderInfo
ModuleNotFoundError: No module named 'etw'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\tmp8sz0jlcw\analyzer.py", line 1524, in <module>
    success = analyzer.run()
  File "C:\tmp8sz0jlcw\analyzer.py", line 507, in run
    __import__(name, globals(), locals(), ["dummy"])
  File "C:\tmp8sz0jlcw\modules\auxiliary\DNS_ETW.py", line 22, in <module>
    raise CuckooPackageError("In order to use DNS_ETW functionality, it " "is required to have pywintrace setup in python.")
lib.common.exceptions.CuckooPackageError: In order to use DNS_ETW functionality, it is required to have pywintrace setup in python.
2024-08-22 13:31:14,026 [root] WARNING: Folder at path "C:\EJVzYsIz\debugger" does not exist, skipping
2024-08-22 13:31:14,026 [root] WARNING: Folder at path "C:\EJVzYsIz\tlsdump" does not exist, skipping
2024-08-22 13:31:14,026 [root] INFO: Analysis completed

from the DNS_ETW module:

try:
    from etw import ETW, ProviderInfo
    from etw import evntrace as et
    from etw.GUID import GUID
except Exception as e:
    log.debug(f"Could not load auxiliary module DNS_ETW due to '{e}'")
    raise CuckooPackageError("In order to use DNS_ETW functionality, it " "is required to have pywintrace setup in python.")

I noticed that DNS_ETW was added only 3 days ago.. pywintrace is required on physical host?. can you help me?

doomedraven commented 3 months ago

hey, sorry, for no responses, jumping between vacations and work. i have pushed fix, do git pull, you don't need to restart nothing at all

marsomx commented 3 months ago

@doomedraven thanks for reply ;-) .. as you wrote before (update) pywintrace is required on physical host?

marsomx commented 3 months ago

@doomedraven the fix did not solve the problem ... perhaps the lack of pywintrace raises the exception

doomedraven commented 3 months ago

pywintrace is not required, is windows side dependency in case you want to get ETW events, well if it didn't fit it, then remove that file from you cape, i don't have time to dig into that right now

doomedraven commented 3 months ago

also as you say it didn't fix, show the error

marsomx commented 3 months ago

yep.. think I will apply this workaround, also because it is not possible to disable the module from the auxiliary config file

doomedraven commented 3 months ago

saying that something doesn't work without error trace is not very useful

doomedraven commented 3 months ago

closing this as original issue doesn't exist anymore. and would appreciate the error msg after git pull as you say dns etw is not fixed

marsomx commented 2 months ago

the problem with sqlalchemy resurfaced after last update (machine and capev2). Capev2 was in 'clean' state. this is the log:

2024-09-08 11:28:20,564 [modules.auxiliary.QemuScreenshots] ERROR: No module named 'libvirt'
2024-09-08 11:28:22,115 [lib.cuckoo.core.machinery_manager] INFO: Using MachineryManager[physical] with max_machines_count=10
2024-09-08 11:28:22,115 [lib.cuckoo.core.scheduler] INFO: Creating scheduler with max_analysis_count=unlimited
2024-09-08 11:28:25,559 [lib.cuckoo.core.machinery_manager] INFO: Loaded 1 machine
2024-09-08 11:28:25,590 [lib.cuckoo.core.machinery_manager] INFO: max_vmstartup_count for BoundedSemaphore = 5
2024-09-08 11:28:25,594 [lib.cuckoo.core.scheduler] INFO: Waiting for analysis tasks
2024-09-08 11:53:04,553 [lib.cuckoo.core.machinery_manager] INFO: Task #1: found useable machine physical01 (arch=x64, platform=windows)
2024-09-08 11:53:04,553 [lib.cuckoo.core.scheduler] INFO: Task #1: Processing task
2024-09-08 11:53:04,667 [lib.cuckoo.core.analysis_manager] INFO: Task #1: Starting analysis of FILE '/tmp/cuckoo-tmp/upload_3ez_u6wo/cs.dll'
2024-09-08 11:53:04,695 [lib.cuckoo.core.analysis_manager] INFO: Task #1: Enabled route 'internet'.
2024-09-08 11:53:04,700 [modules.auxiliary.QemuScreenshots] INFO: QEMU screenshots module loaded
2024-09-08 11:53:04,711 [modules.auxiliary.sniffer] INFO: Started sniffer with PID 4345 (interface=enp2s0, host=192.168.1.101, dump path=/opt/CAPEv2/storage/analyses/1/dump.pcap)
2024-09-08 11:53:05,215 [lib.cuckoo.core.guest] INFO: Task #1: Starting analysis on guest (id=physical01, ip=192.168.1.101)
2024-09-08 11:53:05,231 [lib.cuckoo.core.guest] INFO: Task #1: Guest is running CAPE Agent 0.17 (id=physical01, ip=192.168.1.101)
2024-09-08 11:53:07,391 [lib.cuckoo.core.guest] INFO: Task #1: Uploading script files to guest (id=physical01, ip=192.168.1.101)
2024-09-08 11:57:27,655 [lib.cuckoo.core.guest] INFO: Task #1: End of analysis reached! (id=physical01, ip=192.168.1.101)
2024-09-08 11:57:27,768 [lib.cuckoo.core.analysis_manager] INFO: Task #1: Disabled route 'internet'
2024-09-08 12:13:22,128 [lib.cuckoo.core.analysis_manager] ERROR: Task #1: failure in AnalysisManager.run
Traceback (most recent call last):
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 497, in run
    self.launch_analysis()
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 459, in launch_analysis
    success = self.perform_analysis()
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 443, in perform_analysis
    with self.machine_running(), self.result_server(), self.network_routing(), self.run_auxiliary():
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 335, in machine_running
    self.machinery_manager.machinery.release(self.machine)
  File "/opt/CAPEv2/lib/cuckoo/common/abstracts.py", line 270, in release
    return self.db.unlock_machine(machine)
  File "/opt/CAPEv2/lib/cuckoo/core/database.py", line 978, in unlock_machine
    self.session.add(machine)
  File "<string>", line 2, in add
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2648, in add
    self._save_or_update_state(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2672, in _save_or_update_state
    self._save_or_update_impl(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3289, in _save_or_update_impl
    self._update_impl(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3278, in _update_impl
    self.identity_map.add(state)
  File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/identity.py", line 151, in add
    raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Can't attach instance <Machine at 0x73b53a191300>; another instance with key (<class 'lib.cuckoo.core.database.Machine'>, (1,), None) is already present in this session.

can anyone help me solve it definitively? or is it better to reinstall cape? @doomedraven please could you reopen the issue?

marsomx commented 2 months ago

I have investigated further and I have noticed a weird behaviour. if I restore the machine to the cleaned state and launch a normal exe with default options, the analysis works, it is completed correctly without errors. instead in the previous analysis i ran a dll with dll analyzer and some options (dllloader and function) and it failed for some reason (may be for some wrong option value), getting the sqlalchemy error message. could this have been the cause?

@doomedraven please consider closing the case again. Thanks!!

doomedraven commented 2 months ago

Sql errors are not related to cape options

El mar, 10 sept 2024, 7:12, marsomx @.***> escribió:

I have investigated further and I have noticed a weird behaviour. if I restore the machine to the cleaned state and launch a normal exe with default options, the analysis works, it is completed correctly without errors. instead in the previous analysis i ran a dll with dll analyzer and some options (dllloader and function) and it failed for some reason (may be for some wrong option value), getting the sqlalchemy error message. could this have been the cause?

@doomedraven https://github.com/doomedraven please consider closing the case again. Thanks!!

— Reply to this email directly, view it on GitHub https://github.com/kevoreilly/CAPEv2/issues/2258#issuecomment-2339751383, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOFH36UU2JUR5QA62UAV63ZV2EVDAVCNFSM6AAAAABL4F3XMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZZG42TCMZYGM . You are receiving this because you were mentioned.Message ID: @.***>

marsomx commented 2 months ago

Not directly, of course... I meant that if the analysis is not completed correctly it could cause a sqlalchemy session problem.

Sql errors are not related to cape options El mar, 10 sept 2024, 7:12, marsomx @.> escribió: … I have investigated further and I have noticed a weird behaviour. if I restore the machine to the cleaned state and launch a normal exe with default options, the analysis works, it is completed correctly without errors. instead in the previous analysis i ran a dll with dll analyzer and some options (dllloader and function) and it failed for some reason (may be for some wrong option value), getting the sqlalchemy error message. could this have been the cause? @doomedraven https://github.com/doomedraven please consider closing the case again. Thanks!! — Reply to this email directly, view it on GitHub <#2258 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOFH36UU2JUR5QA62UAV63ZV2EVDAVCNFSM6AAAAABL4F3XMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZZG42TCMZYGM . You are receiving this because you were mentioned.Message ID: @.>