search

kevoreilly / CAPEv2

Malware Configuration And Payload Extraction
https://capesandbox.com/analysis/
Other
1.92k stars 411 forks source link

select machine via tags on webpage (and API) #818

Closed bolle123 closed 2 years ago

bolle123 commented 2 years ago

Prerequisites

Please answer the following questions for yourself before submitting an issue.

Expected Behavior

If I submit a "sample" via API or website, a suitable vm should be selected based on the tags. As soon as no suitable machine is available Cape should wait as long a machine is done and ready.

The web page does show the available tags, i choosed win10x64 in this case: Screenshot 2022-04-11 at 12 02 27

Current Behavior

The first to be used are the two correct machines. However, the third submission uses a wrong machine (win7x86). The task api endpoint shows following task information:

{
            "id": 44,
            "target": "/tmp/cuckoo-sflock/tmpj6hgmphs/c665d34bb18a555dcf1b.exe",
            "category": "file",
            "cape": "",
            "timeout": 200,
            "priority": 2,
            "custom": "",
            "machine": "win7x86_2",
            "package": "exe",
            "route": "tor",
            "tags_tasks": "",
            "options": "",
            "platform": "windows",
            "memory": false,
            "enforce_timeout": false,
            "clock": "2022-04-11 09:57:05",
            "added_on": "2022-04-11 09:57:06",
            "started_on": "2022-04-11 09:57:06",
            "completed_on": null,
            "status": "running",
            "dropped_files": null,
            "running_processes": null,
            "api_calls": null,
            "domains": null,
            "signatures_total": null,
            "signatures_alert": null,
            "files_written": null,
            "registry_keys_modified": null,
            "crash_issues": null,
            "anti_issues": null,
            "analysis_started_on": null,
            "analysis_finished_on": null,
            "processing_started_on": null,
            "processing_finished_on": null,
            "signatures_started_on": null,
            "signatures_finished_on": null,
            "reporting_started_on": null,
            "reporting_finished_on": null,
            "timedout": false,
            "sample_id": 14,
            "machine_id": 3994,
            "shrike_url": null,
            "shrike_refer": null,
            "shrike_msg": null,
            "shrike_sid": null,
            "parent_id": null,
            "tlp": null,
            "user_id": 0,
            "username": "false",
            "tags": [
                "win10x64",
                "x86"
            ],
            "guest": {
                "id": 43,
                "status": "running",
                "name": "win7x86_2",
                "label": "win7x86_2",
                "manager": "KVM",
                "started_on": "2022-04-11 09:57:06",
                "shutdown_on": null,
                "task_id": 44
            },
            "errors": [],
            "sample": {
                "id": 14,
                "file_size": 271333,
                "file_type": "PE32 executable (GUI) Intel 80386, for MS Windows",
                "md5": "54bc2a6e917ed4993560447ddc2413e7",
                "crc32": "CE6A077D",
                "sha1": "65bfe66d677ea4ff22dfb83028763790a1ef7ebd",
                "sha256": "c665d34bb18a555dcf1b248dab633db2ff88d54e93e3f8de8f785b60ea07ab27",
                "sha512": "8e19fc5d7844c6381971ec1f53e2ad2e15b6079fc8151b1ec49a8f6c9f0fd0c9937bae647cd3d323a4cc2b2f42c83b7c0b2e80a6850e9cc77d37e866b0151c07",
                "ssdeep": "6144:HNeZmEjcUE0DdB49QljX0CHTc9TWMKrFnFXbay8M6FcE:HNlE15qQlZzcondGy8dp",
                "parent": 13,
                "source_url": null
            }
        }

Context

My goal is to have multiple VM types (Win7x86|Win7x64|Win10x86|Win10x64) with multiple machines. I want to send samples to these via the API and define which VM type should be used. A simple Python script might look like these:

import requests

REST_URL = "http://cape-sandbox-dev.local:8000/apiv2/tasks/create/file/"
SAMPLE_FILE = "../downloads/c665d34bb18a555dcf1b248dab633db2ff88d54e93e3f8de8f785b60ea07ab27.zip"

with open(SAMPLE_FILE, "rb") as sample:
    multipart_file = {
        'file': sample,
        'tags': ('', 'win10x64')
    }
    response = requests.post(REST_URL, files=multipart_file)

response_json = response.json()
print(f"task_id: {response_json['data']['task_ids']}")

However, I somehow do not manage to pass tags via api.

VMware Esxi on physical machine => Ubuntu 20.04 LTS => CAPEv2. I have installed Cape2 on 01 February 2022. Installation was made via ./kvm-qemu.sh and ./cape2.sh scripts.

API endpoint /apiv2/machines/list/:

{
    "data": [
        {
            "id": 3992,
            "name": "win10x64_2",
            "label": "win10x64_2",
            "arch": "x64",
            "ip": "192.168.1.199",
            "platform": "windows",
            "interface": "virbr0",
            "snapshot": null,
            "locked": false,
            "locked_changed_on": null,
            "status": "poweroff",
            "status_changed_on": "2022-04-11 09:25:18",
            "resultserver_ip": "192.168.1.1",
            "resultserver_port": "2042",
            "tags": [
                "win10x64"
            ]
        },
        {
            "id": 3991,
            "name": "win10x64_1",
            "label": "win10x64",
            "arch": "x64",
            "ip": "192.168.1.151",
            "platform": "windows",
            "interface": "virbr0",
            "snapshot": "20220307_win10x64_snapshot",
            "locked": false,
            "locked_changed_on": null,
            "status": "poweroff",
            "status_changed_on": "2022-04-11 09:25:18",
            "resultserver_ip": "192.168.1.1",
            "resultserver_port": "2042",
            "tags": [
                "win10x64"
            ]
        },
        {
            "id": 3994,
            "name": "win7x86_2",
            "label": "win7x86_2",
            "arch": "x86",
            "ip": "192.168.1.186",
            "platform": "windows",
            "interface": "virbr0",
            "snapshot": null,
            "locked": false,
            "locked_changed_on": null,
            "status": "poweroff",
            "status_changed_on": "2022-04-11 09:25:18",
            "resultserver_ip": "192.168.1.1",
            "resultserver_port": "2042",
            "tags": [
                "win7x86"
            ]
        },
        {
            "id": 3993,
            "name": "win7x86_1",
            "label": "win7x86_1",
            "arch": "x86",
            "ip": "192.168.1.242",
            "platform": "windows",
            "interface": "virbr0",
            "snapshot": null,
            "locked": false,
            "locked_changed_on": null,
            "status": "poweroff",
            "status_changed_on": "2022-04-11 09:25:18",
            "resultserver_ip": "192.168.1.1",
            "resultserver_port": "2042",
            "tags": [
                "win7x86"
            ]
        }
    ],
    "error": false
}

Thanks!

doomedraven commented 2 years ago

Hello, first of all why do you send 2 tags and as empty? it won't work. 'tags': ('', 'win10x64') <- why not just 'tags': 'win10x64'?

https://github.com/kevoreilly/CAPEv2/blob/master/lib/cuckoo/common/web_utils.py#L1063

bolle123 commented 2 years ago

Hello, that was my first try, but that simply did not work. It took me quite a bit of time to find out what was wrong.

With the parameter 'machine' the response code was even 500, see below example. It searched for python+requests+multipart_form and got some results like:

my code:

multipart_file = {
        'file': sample,
        'machine': ('', 'win7x86_2')
    }

does work and created this task:

{
            "id": 79,
            "target": "/tmp/cuckoo-sflock/tmpfitoeswv/c665d34bb18a555dcf1b.exe",
            "category": "file",
            "cape": "",
            "timeout": 0,
            "priority": 1,
            "custom": "",
            "machine": "win7x86_2",
            "package": "exe",
            "route": "false",
            "tags_tasks": null,
            "options": "",
            "platform": "windows",

            [...]
}

but with the following code

multipart_file = {
        'file': sample,
        'machine': 'win7x86_2'
    }

i got a HTTP 500 response: cape_response.txt feel free to rename it to *.html

doomedraven commented 2 years ago

yes that is normal that you getting error you are totally wrong here

multipart_file goes under files=multipart_file

with open(SAMPLE_FILE, "rb") as sample:
    multipart_file = {'file': sample}
    response = requests.post(REST_URL, files=multipart_file, data={"tags": "win10x64"})

any options goes inside of the data

bolle123 commented 2 years ago

well, perhaps this was not the most elegant way, but the result remains the same. However that was not my issue. Regardless of which code (or even the browser) i use, the problem remains.

Why this task is possible, please compare tags and choosen machine, is this because of the x86 tag? Where does this tag come from?

{
            "id": 83,
            "target": "/tmp/cuckoo-sflock/tmppnwz28ce/c665d34bb18a555dcf1b.exe",
            "category": "file",
            "cape": "",
            "timeout": 0,
            "priority": 1,
            "custom": "",
            "machine": "win7x86_2",
            "package": "exe",
            "route": "false",
            "tags_tasks": null,
            "options": "",
            "platform": "windows",
            "memory": false,
            "enforce_timeout": false,
            "clock": "2022-04-11 20:07:14",
            "added_on": "2022-04-11 20:07:14",
            "started_on": "2022-04-11 20:07:15",
            "completed_on": null,
            "status": "running",
            "dropped_files": null,
            "running_processes": null,
            "api_calls": null,
            "domains": null,
            "signatures_total": null,
            "signatures_alert": null,
            "files_written": null,
            "registry_keys_modified": null,
            "crash_issues": null,
            "anti_issues": null,
            "analysis_started_on": null,
            "analysis_finished_on": null,
            "processing_started_on": null,
            "processing_finished_on": null,
            "signatures_started_on": null,
            "signatures_finished_on": null,
            "reporting_started_on": null,
            "reporting_finished_on": null,
            "timedout": false,
            "sample_id": 14,
            "machine_id": 4010,
            "shrike_url": null,
            "shrike_refer": null,
            "shrike_msg": null,
            "shrike_sid": null,
            "parent_id": null,
            "tlp": null,
            "user_id": 0,
            "username": "false",
            "tags": [
                "x86",
                "win10x64"
            ],
            "guest": {
                "id": 82,
                "status": "running",
                "name": "win7x86_2",
                "label": "win7x86_2",
                "manager": "KVM",
                "started_on": "2022-04-11 20:07:15",
                "shutdown_on": null,
                "task_id": 83
            },
            "errors": [],
            "sample": {
                "id": 14,
                "file_size": 271333,
                "file_type": "PE32 executable (GUI) Intel 80386, for MS Windows",
                "md5": "54bc2a6e917ed4993560447ddc2413e7",
                "crc32": "CE6A077D",
                "sha1": "65bfe66d677ea4ff22dfb83028763790a1ef7ebd",
                "sha256": "c665d34bb18a555dcf1b248dab633db2ff88d54e93e3f8de8f785b60ea07ab27",
                "sha512": "8e19fc5d7844c6381971ec1f53e2ad2e15b6079fc8151b1ec49a8f6c9f0fd0c9937bae647cd3d323a4cc2b2f42c83b7c0b2e80a6850e9cc77d37e866b0151c07",
                "ssdeep": "6144:HNeZmEjcUE0DdB49QljX0CHTc9TWMKrFnFXbay8M6FcE:HNlE15qQlZzcondGy8dp",
                "parent": 13,
                "source_url": null
            }
        }
doomedraven commented 2 years ago

x86 is there https://github.com/kevoreilly/CAPEv2/blob/master/lib/cuckoo/core/database.py#L1303, i will check your issue once i will have some spare time, as right now im tired and not see the reason of the problem

derfel1989 commented 2 years ago

@bolle123 I saw that you are using Win10. Do you mind sharing your W10 Guest configuration? I am facing issues getting screenshots during the analysis.

On my W10 guest, I have python 3.6 and the last version of the pillow.

Thank you in advance.

bolle123 commented 2 years ago

@derfel1989 I don't know exactly what you want...but I just noticed that analyses with Win10 don't have screenshots for me either.

 I'll look at the logs next week My grep from logs (report json from web frontend):

2022-03-07 11:47:46,850 [root] DEBUG: Importing auxiliary module \"modules.auxiliary.screenshots\"...
2022-03-07 11:47:46,869 [lib.api.screenshot] DEBUG: Importing 'PIL.ImageChops'
2022-03-07 11:47:49,365 [lib.api.screenshot] DEBUG: Importing 'PIL.ImageGrab'
2022-03-07 11:47:49,578 [lib.api.screenshot] DEBUG: Importing 'PIL.ImageDraw'
2022-03-07 11:47:52,617 [root] DEBUG: Initializing auxiliary module \"Screenshots\"...
2022-03-07 11:47:52,617 [root] DEBUG: Started auxiliary module Screenshots
2022-03-07 11:47:53,694 [modules.auxiliary.screenshots] ERROR: Cannot take screenshot: screen grab failed

  The last line occurs several times, probably whenever a screenshot is to be taken.

Python 3.10 is running and pillowis installed via pip3.

derfel1989 commented 2 years ago

@bolle123

Thank you for reply my question.

I don't have any errors in the logs. But I don't have any screenshots too.

Sample log:

2022-05-02 21:39:40,592 [root] DEBUG: Importing auxiliary module "modules.auxiliary.screenshots"... 2022-05-02 21:39:40,626 [lib.api.screenshot] DEBUG: Importing 'PIL.ImageChops' 2022-05-02 21:39:40,873 [lib.api.screenshot] DEBUG: Importing 'PIL.ImageGrab' 2022-05-02 21:39:40,899 [lib.api.screenshot] DEBUG: Importing 'PIL.ImageDraw'

What I see here is that I have a couple of issues linked to permission as shown below:

2022-05-02 21:39:43,162 [root] DEBUG: Error 5 (0x5) - InjectDll: Failed to open process: Access is denied.

2022-05-02 21:39:53,233 [root] DEBUG: Error 5 (0x5) - AmsiDumper: Is CAPE agent running elevated? Initialisation failed: Access is denied.

Have you seen behaviours like that with win10?

kevoreilly commented 2 years ago

The errors you quote there are from the monitor and so not related to screenshots. The monitor problem is literally what it says: AMSI dumping requires that the malware process be elevated - this is also recommended anyway as otherwise it's running with reduced privileges. To solve this is as simple as making sure the agent is launched elevated or 'run as administrator'.

TheMythologist commented 2 years ago

Can confirm: screenshots on windows 10 work fine if set up correctly

bolle123 commented 2 years ago

Can confirm: screenshots on windows 10 work fine if set up correctly

can you explain in more detail what you mean by that? What Python version do you use? Did you "just" perform the steps from "choco.bat" in an administrator shell?

TheMythologist commented 2 years ago

Can confirm: screenshots on windows 10 work fine if set up correctly

can you explain in more detail what you mean by that? What Python version do you use? Did you "just" perform the steps from "choco.bat" in an administrator shell?

I'm using Python 3.6 32-bit version, and make sure agent.py is running as administrator (I set up a scheduling task to run on start-up as admin and rebooted the vm)

As for installation steps, I didn't use choco.bat, I installed everything manually.

derfel1989 commented 2 years ago

The errors you quote there are from the monitor and so not related to screenshots. The monitor problem is literally what it says: AMSI dumping requires that the malware process be elevated - this is also recommended anyway as otherwise it's running with reduced privileges. To solve this is as simple as making sure the agent is launched elevated or 'run as administrator'.

I created a user with admin privileges. I also tried calling the agent.py over cmd running as administrator. But, nothing brought the screenshots. The third attempt was to create a service with NSSM, but this did even not connect with the cape server, so I left this.

In your point of view, what is the best approach @kevoreilly .

@TheMythologist Thank you for your reply. I will try it and get back with the results. Do you have any additional recommendations?

Thanks all.

derfel1989 commented 2 years ago

I just realize that I can see the screenshots when I export a report as an HTML file.

So, the issue might not be with the pillow module or the guest (w10). It's probably something on the cape side which is not allowing to load of the thumbnails on the analysis page.

@bolle123 could you test it?

doomedraven commented 2 years ago

i have just created win10 20h2 with latest pillow, works just fine

Captura de Pantalla 2022-05-22 a las 8 44 28
doomedraven commented 2 years ago

solved. details here. thanks to @cccs-kevin https://github.com/kevoreilly/CAPEv2/commit/0bfc39aa766c8ac5a7d9c927e348a10548431879