search

kevoreilly / CAPEv2

Malware Configuration And Payload Extraction
https://capesandbox.com/analysis/
Other
1.89k stars 404 forks source link

Endless processing / Task #failed: Analysis X #2103

Closed wuuuw closed 3 months ago

wuuuw commented 4 months ago

Prerequisites

Expected Behavior

The web interface will complete the scanning and I can check the scan report

Current Behavior

In the web interface in the Recent window the status is processing and I cannot view the scanning report. But in the Dashboard window it is listed as completed. The terminal displays INFO: Task #5: analysis procedure completed

Steps to Reproduce

  1. poetry run python3 cuckoo.py
  2. load file in web-interface (Submit)
  3. just "processing" in Recent, but "completed" in Dashboard
  4. poetry run python utils/process.py -r 5 -d
  5. "reported" in Recent and Dashboard

Also tried sudo systemctl restart cape-processor and this poetry run python cleaners.py --clean sudo systemctl restart cape

Context

Question Answer
Git commit commit 2248ab0458f6239b4696ec2d80196cefabcfcb2c (HEAD -> master)
OS version Ubuntu 22.04.4 LTS, Windows 10 21H2

Failure Logs

Terminal from poetry run python3 cuckoo.py

OPTIONAL! Missed dependency: pip3 install https://github.com/CAPESandbox/peepdf/archive/20eda78d7d77fc5b3b652ffc2d8a5b0af796e3dd.zip#egg=peepdf==0.4.2 OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/DissectMalware/batch_deobfuscator OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/CAPESandbox/httpreplay /usr/bin/tcpdump 2024-05-06 17:51:45,486 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=10, and max_vmstartup_count=5 2024-05-06 17:51:45,494 [lib.cuckoo.core.scheduler] INFO: Loaded 1 machine/s 2024-05-06 17:51:45,511 [lib.cuckoo.core.scheduler] INFO: Waiting for analysis tasks 2024-05-06 17:52:54,972 [lib.cuckoo.core.scheduler] INFO: Task #5: Starting analysis of FILE '/tmp/cuckoo-tmp/upload_atx723hk/b5e6dde637ff9dbc4dc8.exe' 2024-05-06 17:52:55,016 [lib.cuckoo.core.scheduler] INFO: Task #5: acquired machine win10 (label=win10, arch=x64, platform=windows) 2024-05-06 17:53:20,352 [lib.cuckoo.core.scheduler] INFO: Enabled route 'internet'. /usr/bin/tcpdump 2024-05-06 17:53:20,414 [modules.auxiliary.sniffer] INFO: Started sniffer with PID 15295 (interface=virbr0, host=192.168.122.240, dump path=/opt/CAPEv2/storage/analyses/5/dump.pcap) 2024-05-06 17:53:20,435 [lib.cuckoo.core.guest] INFO: Task #5: Starting analysis on guest (id=win10, ip=192.168.122.240) 2024-05-06 17:53:20,461 [lib.cuckoo.core.guest] INFO: Task #5: Guest is running CAPE Agent 0.17 (id=win10, ip=192.168.122.240) 2024-05-06 17:53:26,773 [lib.cuckoo.core.guest] INFO: Task #5: Uploading script files to guest (id=win10, ip=192.168.122.240) 2024-05-06 17:53:32,047 [lib.cuckoo.core.guest] INFO: Task #failed: Analysis 5 (id=win10, ip=192.168.122.240) 2024-05-06 17:53:33,289 [lib.cuckoo.core.scheduler] INFO: Disabled route 'internet' 2024-05-06 17:53:33,343 [lib.cuckoo.core.scheduler] INFO: Task #5: analysis procedure completed ^C Session terminated, killing shell... ...killed.

Terminal from poetry run python utils/process.py -r 5 -d

2024-05-06 18:37:18,932 [root] DEBUG: Importing modules... OPTIONAL! Missed dependency: pip3 install https://github.com/CAPESandbox/peepdf/archive/20eda78d7d77fc5b3b652ffc2d8a5b0af796e3dd.zip#egg=peepdf==0.4.2 OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/DissectMalware/batch_deobfuscator OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/CAPESandbox/httpreplay 2024-05-06 18:37:19,894 [capa.rules] DEBUG: reading rules from directory /opt/CAPEv2/data/capa-rules 2024-05-06 18:37:19,933 [capa.rules.cache] DEBUG: loading rule set from cache: /home/cape/.cache/capa/capa-fa8c13d6.cache 2024-05-06 18:37:19,993 [capa.loader] DEBUG: reading signatures from directory /opt/CAPEv2/data/flare-signatures 2024-05-06 18:37:19,994 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/1_flare_msvc_rtf_32_64.sig 2024-05-06 18:37:19,994 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/2_flare_msvc_atlmfc_32_64.sig 2024-05-06 18:37:19,994 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/3_flare_common_libs.sig 2024-05-06 18:37:19,996 [root] DEBUG: Imported "auxiliary" modules: 2024-05-06 18:37:19,996 [root] DEBUG: -- Sniffer 2024-05-06 18:37:19,996 [root] DEBUG: Imported "processing" modules: 2024-05-06 18:37:19,996 [root] DEBUG: |-- CAPE 2024-05-06 18:37:19,996 [root] DEBUG: |-- AnalysisInfo . 2024-05-06 18:37:20,039 [root] DEBUG: Imported "machinery" modules: 2024-05-06 18:37:20,039 [root] DEBUG:-- KVM 2024-05-06 18:37:20,039 [Task 5] [root] DEBUG: Processing task 2024-05-06 18:37:20,197 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "CAPE" on analysis at "/opt/CAPEv2/storage/analyses/5" 2024-05-06 18:37:20,218 [Task 5] [lib.cuckoo.common.objects] DEBUG: Initializing Yara... 2024-05-06 18:37:20,301 [Task 5] [lib.cuckoo.common.objects] DEBUG: |-- binaries AutoIT.yar . 2024-05-06 18:37:20,591 [Task 5] [lib.cuckoo.common.objects] DEBUG: |-- CAPE xRAT.yar 2024-05-06 18:37:20,591 [Task 5] [lib.cuckoo.common.objects] DEBUG: |-- CAPE zgRAT.yar 2024-05-06 18:37:21,998 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "AnalysisInfo" on analysis at "/opt/CAPEv2/storage/analyses/5" 2024-05-06 18:37:22,046 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Processing module autoruns not found in configuration file 2024-05-06 18:37:22,047 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "BehaviorAnalysis" on analysis at "/opt/CAPEv2/storage/analyses/5" 2024-05-06 18:37:22,047 [Task 5] [modules.processing.behavior] DEBUG: Analysis results folder does not contain any file or injection was disabled 2024-05-06 18:37:22,047 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "Debug" on analysis at "/opt/CAPEv2/storage/analyses/5" 2024-05-06 18:37:22,051 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "NetworkAnalysis" on analysis at "/opt/CAPEv2/storage/analyses/5" 2024-05-06 18:37:22,055 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "Suricata" on analysis at "/opt/CAPEv2/storage/analyses/5" 2024-05-06 18:37:22,640 [Task 5] [modules.processing.suricata] DEBUG: pcapfile list: {'message': {'count': 0, 'files': []}, 'return': 'OK'} current pcap: {'message': '/opt/CAPEv2/storage/analyses/5/dump.pcap', 'return': 'OK'} 2024-05-06 18:37:27,644 [Task 5] [modules.processing.suricata] DEBUG: pcapfile list: {'message': {'count': 0, 'files': []}, 'return': 'OK'} current pcap: {'message': 'None', 'return': 'OK'} 2024-05-06 18:37:27,644 [Task 5] [modules.processing.suricata] DEBUG: Pcap not in list and not current pcap lets assume it's processed 2024-05-06 18:37:27,688 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "UrlAnalysis" on analysis at "/opt/CAPEv2/storage/analyses/5" 2024-05-06 18:37:27,689 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "script_log_processing" on analysis at "/opt/CAPEv2/storage/analyses/5" 2024-05-06 18:37:27,690 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing processing module "ProcessMemory" on analysis at "/opt/CAPEv2/storage/analyses/5" 2024-05-06 18:37:27,749 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Applying signature overlays for signatures: creates_exe 2024-05-06 18:37:27,752 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running 252 evented signatures 2024-05-06 18:37:27,752 [Task 5] [lib.cuckoo.core.plugins] DEBUG: |-- packer_themida . 2024-05-06 18:37:27,953 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running signature "dharma_mutexes" 2024-05-06 18:37:27,953 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running signature "ransomware_extensions" /opt/CAPEv2/utils/../lib/cuckoo/common/abstracts.py:1058: FutureWarning: Possible nested set at position 5 exp = re.compile(pattern, re.IGNORECASE) 2024-05-06 18:37:27,962 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running signature "ransomware_files" 2024-05-06 18:37:27,975 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running signature "fonix_mutexes" 2024-05-06 18:37:27,975 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Running signature "gandcrab_mutexes" . 2024-05-06 18:37:28,046 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing reporting module "CAPASummary" 2024-05-06 18:37:28,058 [Task 5] [lib.cuckoo.common.integrations.capa] INFO: FLARE CAPA -> No process data available 2024-05-06 18:37:28,060 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing reporting module "PCAP2CERT" 2024-05-06 18:37:28,062 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing reporting module "JsonDump" 2024-05-06 18:37:28,063 [Task 5] [lib.cuckoo.core.plugins] DEBUG: Executing reporting module "MongoDB" 2024-05-06 18:37:28,078 [Task 5] [modules.reporting.mongodb] DEBUG: Deleted previous MongoDB data for Task 5 2024-05-06 18:37:28,235 [Task 5] [root] DEBUG: Finished processing task

wuuuw commented 4 months ago

cuckoo.conf ip = 192.168.122.1 freespace = 45000

kvm.conf machines = win10 interface = virbr0 label = win10 ip = 192.168.122.240 arch = x64

routing.conf route = internet internet = virbr0

doomedraven commented 4 months ago

hey, yes we suspect the bug is here https://github.com/kevoreilly/CAPEv2/pull/2036/files we probably will revert that

wuuuw commented 4 months ago

@doomedraven I need any working version of the sandbox. (I don't get any information in behavioral and network analysis, none at all)

where could I get it? image image

doomedraven commented 4 months ago

you can revert that commit that i mention, i can-t guarantee that one cause the problem but is our suspicion. well is open source so you can help us dig into issue, is not our paid job so we have to first handle our job and then work on this

kevoreilly commented 4 months ago

I have been able to recreate this issue on an instance of mine, and found that reverting the changes in #2306 fixes the issue.

I haven't had time to look into why it is occurring. @qux-bbb perhaps you can help diagnose this issue?

qux-bbb commented 4 months ago

I'll diagnose this issue in a few days.

qux-bbb commented 4 months ago

@wuuuw It's not a bug, you run the wrong process command.
poetry run python utils/process.py -r 5 -d is not a right command for this situation. 

  -d, --debug           Display debug messages
  -r, --report          Re-generate report

You should run process.py just like in cape-processor. service: 

cd /opt/CAPEv2/utils/
poetry run python process.py -p7 auto -pt 900
kevoreilly commented 4 months ago

Er hang on. I use -d -r all the time in my daily work. It was working before, it needs to work.

qux-bbb commented 4 months ago

Er hang on. I use -d -r all the time in my daily work. It was working before, it needs to work.

I want to know all the status about cape services before you do anything.
You can run this command: `systemctl status cape.service`

qux-bbb commented 4 months ago

@wuuuw @kevoreilly If the cape-processor service is not running at the begining, the behavior is like that.
You guys can give me more info, then I can verify it.

wuuuw commented 4 months ago

@qux-bbb ● cape-web.service - CAPE WSGI app Loaded: loaded (/lib/systemd/system/cape-web.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2024-05-16 17:06:54 UTC; 8min ago Docs: https://github.com/kevoreilly/CAPEv2 Main PID: 1401 (python) Tasks: 3 (limit: 6938) Memory: 480.7M CPU: 1min 6.531s CGroup: /system.slice/cape-web.service ├─1401 /home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/bin/python manage.py runserver_plus 0.0.0.0:8000 --traceback --keep-meta-shutdown └─2550 /home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/bin/python manage.py runserver_plus 0.0.0.0:8000 --traceback --keep-meta-shutdown

мая 16 17:07:04 nan-pc python3[2550]: Missed dependency flare-floss: poetry run pip install -U flare-floss мая 16 17:07:04 nan-pc python3[2550]: System check identified no issues (3 silenced). мая 16 17:07:04 nan-pc python3[2550]: You have 32 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): account, admin, auth, authtoken, cont> мая 16 17:07:04 nan-pc python3[2550]: Run 'python manage.py migrate' to apply them. мая 16 17:07:04 nan-pc python3[2550]: Django version 4.2.11, using settings 'web.settings' мая 16 17:07:04 nan-pc python3[2550]: Development server is running at http://0.0.0.0:8000/ мая 16 17:07:04 nan-pc python3[2550]: Using the Werkzeug debugger (http://werkzeug.pocoo.org/) мая 16 17:07:04 nan-pc python3[2550]: Quit the server with CONTROL-C. мая 16 17:07:04 nan-pc python3[2550]: Debugger is active! мая 16 17:07:04 nan-pc python3[2550]: Debugger PIN: 968-109-742

● cape-processor.service - CAPE report processor Loaded: loaded (/lib/systemd/system/cape-processor.service; enabled; vendor preset: enabled) Active: activating (auto-restart) since Thu 2024-05-16 17:12:04 UTC; 2min 50s ago Docs: https://github.com/kevoreilly/CAPEv2 Process: 3484 ExecStart=/usr/bin/python3 -m poetry run python process.py -p7 auto -pt 900 (code=exited, status=0/SUCCESS) Main PID: 3484 (code=exited, status=0/SUCCESS) CPU: 3.653s

● cape.service - CAPE Loaded: loaded (/lib/systemd/system/cape.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2024-05-16 17:06:54 UTC; 8min ago Docs: https://github.com/kevoreilly/CAPEv2 Main PID: 1402 (python) Tasks: 58 (limit: 6938) Memory: 266.0M CPU: 6min 3.042s CGroup: /system.slice/cape.service └─1402 /home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/bin/python cuckoo.py

мая 16 17:06:57 nan-pc python3[1402]: _.-'.iii.' мая 16 17:06:57 nan-pc python3[1402]: """"""""" мая 16 17:06:57 nan-pc python3[1402]: Cuckoo Sandbox 2.4-CAPE мая 16 17:06:57 nan-pc python3[1402]: www.cuckoosandbox.org мая 16 17:06:57 nan-pc python3[1402]: Copyright (c) 2010-2015 мая 16 17:06:57 nan-pc python3[1402]: CAPE: Config and Payload Extraction мая 16 17:06:57 nan-pc python3[1402]: github.com/kevoreilly/CAPEv2 мая 16 17:07:00 nan-pc python3[1402]: 2024-05-16 17:07:00,788 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=10, and max_vmstart> мая 16 17:07:00 nan-pc python3[1402]: 2024-05-16 17:07:00,796 [lib.cuckoo.core.scheduler] INFO: Loaded 1 machine/s мая 16 17:07:00 nan-pc python3[1402]: 2024-05-16 17:07:00,812 [lib.cuckoo.core.scheduler] INFO: Waiting for analysis tasks

● cape-rooter.service - CAPE rooter Loaded: loaded (/lib/systemd/system/cape-rooter.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2024-05-16 17:06:54 UTC; 8min ago Docs: https://github.com/kevoreilly/CAPEv2 Process: 922 ExecStartPre=/usr/bin/python3 -m poetry config cache-dir /opt/CAPEv2/.cache/pypoetry (code=exited, status=0/SUCCESS) Main PID: 1398 (python) Tasks: 1 (limit: 6938) Memory: 28.1M CPU: 1.912s CGroup: /system.slice/cape-rooter.service └─1398 /opt/CAPEv2/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/bin/python rooter.py -g cape

мая 16 17:06:52 nan-pc systemd[1]: Starting CAPE rooter... мая 16 17:06:54 nan-pc systemd[1]: Started CAPE rooter.

qux-bbb commented 3 months ago

@wuuuw Can your cape-processor.service be runing status?

wuuuw commented 3 months ago

~$ systemctl status cape-processor.service ● cape-processor.service - CAPE report processor Loaded: loaded (/lib/systemd/system/cape-processor.service; enabled; vendor preset: enabled) Active: activating (auto-restart) since Fri 2024-05-17 10:10:03 UTC; 2min 33s ago Docs: https://github.com/kevoreilly/CAPEv2 Process: 1378 ExecStart=/usr/bin/python3 -m poetry run python process.py -p7 auto -pt 900 (code=exited, status=0/SUCCESS) Main PID: 1378 (code=exited, status=0/SUCCESS) CPU: 5.530s

мая 17 10:10:03 nan-pc systemd[1]: cape-processor.service: Consumed 5.530s CPU time.

~$ sudo systemctl restart cape-processor.service

~$ systemctl status cape-processor.service ● cape-processor.service - CAPE report processor Loaded: loaded (/lib/systemd/system/cape-processor.service; enabled; vendor preset: enabled) Active: activating (auto-restart) since Fri 2024-05-17 10:13:12 UTC; 11s ago Docs: https://github.com/kevoreilly/CAPEv2 Process: 4295 ExecStart=/usr/bin/python3 -m poetry run python process.py -p7 auto -pt 900 (code=exited, status=0/SUCCESS) Main PID: 4295 (code=exited, status=0/SUCCESS) CPU: 3.281s

qux-bbb commented 3 months ago

@wuuuw Can you run these commands and give me the output? 

sudo systemctl stop cape-processor.service
cd /opt/CAPEv2/utils/
sudo su cape
/usr/bin/python3 -m poetry run python process.py -p7 auto -pt 900 -d
qux-bbb commented 3 months ago

@AghaHannan You need to create a new issue.

wuuuw commented 3 months ago

@qux-bbb 2024-05-17 22:16:50,644 [root] DEBUG: Importing modules... OPTIONAL! Missed dependency: pip3 install https://github.com/CAPESandbox/peepdf/archive/20eda78d7d77fc5b3b652ffc2d8a5b0af796e3dd.zip#egg=peepdf==0.4.2 OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/DissectMalware/batch_deobfuscator OPTIONAL! Missed dependency: poetry run pip install -U git+https://github.com/CAPESandbox/httpreplay 2024-05-17 22:16:51,639 [capa.rules] DEBUG: reading rules from directory /opt/CAPEv2/data/capa-rules 2024-05-17 22:16:51,685 [capa.rules.cache] DEBUG: loading rule set from cache: /home/cape/.cache/capa/capa-fa8c13d6.cache 2024-05-17 22:16:51,754 [capa.loader] DEBUG: reading signatures from directory /opt/CAPEv2/data/flare-signatures 2024-05-17 22:16:51,754 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/1_flare_msvc_rtf_32_64.sig 2024-05-17 22:16:51,754 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/2_flare_msvc_atlmfc_32_64.sig 2024-05-17 22:16:51,754 [capa.loader] DEBUG: found signature file: /opt/CAPEv2/data/flare-signatures/3_flare_common_libs.sig 2024-05-17 22:16:51,757 [root] DEBUG: Imported "auxiliary" modules: 2024-05-17 22:16:51,757 [root] DEBUG: -- Sniffer 2024-05-17 22:16:51,757 [root] DEBUG: Imported "processing" modules: 2024-05-17 22:16:51,757 [root] DEBUG: |-- CAPE 2024-05-17 22:16:51,757 [root] DEBUG: |-- AnalysisInfo 2024-05-17 22:16:51,757 [root] DEBUG: |-- Autoruns 2024-05-17 22:16:51,757 [root] DEBUG: |-- BehaviorAnalysis 2024-05-17 22:16:51,758 [root] DEBUG: |-- Debug 2024-05-17 22:16:51,758 [root] DEBUG: |-- NetworkAnalysis 2024-05-17 22:16:51,758 [root] DEBUG: |-- ProcessMemory 2024-05-17 22:16:51,758 [root] DEBUG: |-- script_log_processing 2024-05-17 22:16:51,758 [root] DEBUG: |-- Suricata 2024-05-17 22:16:51,758 [root] DEBUG:-- UrlAnalysis 2024-05-17 22:16:51,758 [root] DEBUG: Imported "signatures" modules: 2024-05-17 22:16:51,758 [root] DEBUG: |-- ClamAV 2024-05-17 22:16:51,758 [root] DEBUG: |-- KnownVirustotal

2024-05-17 22:16:51,809 [root] DEBUG: |-- LinuxReadsFiles 2024-05-17 22:16:51,809 [root] DEBUG: -- LinuxWritesFiles 2024-05-17 22:16:51,809 [root] DEBUG: Imported "reporting" modules: 2024-05-17 22:16:51,809 [root] DEBUG: |-- BinGraph 2024-05-17 22:16:51,809 [root] DEBUG: |-- CAPASummary 2024-05-17 22:16:51,809 [root] DEBUG: |-- JsonDump 2024-05-17 22:16:51,809 [root] DEBUG: |-- MongoDB 2024-05-17 22:16:51,809 [root] DEBUG:-- PCAP2CERT 2024-05-17 22:16:51,809 [root] DEBUG: Imported "feeds" modules: 2024-05-17 22:16:51,809 [root] DEBUG: -- AbuseCH_SSL 2024-05-17 22:16:51,809 [root] DEBUG: Imported "machinery" modules: 2024-05-17 22:16:51,809 [root] DEBUG:-- KVM 2024-05-17 22:16:51,810 [root] INFO: Processing analysis data Traceback (most recent call last): File "/opt/CAPEv2/utils/process.py", line 298, in autoprocess with pebble.ProcessPool(max_workers=parallel, max_tasks=maxtasksperchild, initializer=init_worker) as pool: File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/process.py", line 61, in init self._pool_manager = PoolManager(self._context, mp_context) File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/process.py", line 200, in init self.worker_manager = WorkerManager(context.workers, File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/process.py", line 342, in init self.pool_channel, self.workers_channel = channels(mp_context) File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/channel.py", line 34, in channels WorkerChannel(read0, write1, (read1, write0), mp_context)) File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/channel.py", line 86, in init self.mutex = ChannelMutex(mp_context) File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/pebble/pool/channel.py", line 132, in init self.reader_mutex = mp_context.RLock() File "/usr/lib/python3.10/multiprocessing/context.py", line 73, in RLock return RLock(ctx=self.get_context()) File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 187, in init SemLock.init(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx) File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 57, in init sl = self._semlock = _multiprocessing.SemLock( OSError: [Errno 12] Cannot allocate memory

qux-bbb commented 3 months ago

@wuuuw Sorry, I pulled the latest code, but I can't reproduce that error.
Do all samples have that problem, or the special sample?
If the special sample has the problem, can you provide the sample?

qux-bbb commented 3 months ago

@wuuuw Can you run the command and give me the output? 

free -h
doomedraven commented 3 months ago

Oh looks like someone doesn't have enought ram to run processing

wuuuw commented 3 months ago

@qux-bbb free -h total used free shared buff/cache available Mem: 5,7Gi 2,6Gi 982Mi 103Mi 2,2Gi 2,9Gi Swap: 2,0Gi 0B 2,0Gi

qux-bbb commented 3 months ago

@wuuuw Can You try to install CAPE on a computer with more RAM, for example 16G?

doomedraven commented 3 months ago

i have added better message in those cases with ram https://github.com/kevoreilly/CAPEv2/commit/b1df2192323ed5828e4d68880afe358eda81774e

qux-bbb commented 3 months ago

@doomedraven Thanks!
@kevoreilly Do you also have the error "Cannot allocate memory"?

doomedraven commented 3 months ago

ok i finally found some time to jump to this and test latest master on my side, i cant reproduce it.

qux-bbb commented 3 months ago

To reproduce that "Cannot allocate memory", you can do like this:

First, stop cape-processor.service: 

sudo systemctl stop cape-processor.service

Second, use many RAM, my computer RAM is 16G, so I will use 10G(10240M): 

mkdir /tmp/memory
sudo mount -t tmpfs -o size=10240M tmpfs /tmp/memory
dd if=/dev/zero of=/tmp/memory/block

Third, try to start process.py: 

cd /opt/CAPEv2/utils/
sudo su cape
/usr/bin/python3 -m poetry run python process.py -p7 auto -pt 900 -d

Then you can get the error message.

Finally, restore your RAM: 

rm /tmp/memory/block
sudo umount /tmp/memory
rm /tmp/memory -r

Maybe the issue can be closed.

doomedraven commented 3 months ago

yes, i can't reproduce, and the user side low hardware, we cant do nothing there