kevoreilly / CAPEv2

Malware Configuration And Payload Extraction
https://capesandbox.com/analysis/
Other
1.91k stars 411 forks source link

No packets from/to guests? #2304

Closed xme closed 3 weeks ago

xme commented 3 weeks ago

Prerequisites

Please answer the following questions for yourself before submitting an issue.

Expected Behavior

CAPE using kvm, "standard" setup.

Current Behavior

When I boot guests manually, they have network connectivity (through virbr0), I can access the agent (via port 8000), all good! When I submit a file to CAPE, the VM is started but... no network at all! Seems that no packets are sent/received by the guest and all analysis fail.

I'm using "Internet" access and my ens160 interface as "dirty line".

2024-09-02 14:50:35,282 [lib.cuckoo.core.analysis_manager] ERROR: Task #310: Machine win10x64: the guest initialization hit the critical timeout, analysis aborted
Traceback (most recent call last):
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 445, in perform_analysis
    self.run_analysis_on_guest()
  File "/opt/CAPEv2/lib/cuckoo/core/analysis_manager.py", line 411, in run_analysis_on_guest
    guest_manager.start_analysis(options)
  File "/opt/CAPEv2/lib/cuckoo/core/guest.py", line 253, in start_analysis
    self.wait_available()
  File "/opt/CAPEv2/lib/cuckoo/core/guest.py", line 162, in wait_available
    raise CuckooGuestCriticalTimeout(
lib.cuckoo.common.exceptions.CuckooGuestCriticalTimeout: Machine win10x64: the guest initialization hit the critical timeout, analysis aborted

Steps to Reproduce

tcpdump in virbr0 reveals that the guest can't even to an ARP lookup to get the default gw MAC! Here are the rules made by Rooter:

# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
LIBVIRT_INP  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED /* CAPE-rooter */

Chain FORWARD (policy DROP)
target     prot opt source               destination
CAPE_ACCEPTED_SEGMENTS  all  --  0.0.0.0/0            0.0.0.0/0            /* CAPE-rooter */
CAPE_REJECTED_SEGMENTS  all  --  0.0.0.0/0            0.0.0.0/0            /* CAPE-rooter */
LIBVIRT_FWX  all  --  0.0.0.0/0            0.0.0.0/0
LIBVIRT_FWI  all  --  0.0.0.0/0            0.0.0.0/0
LIBVIRT_FWO  all  --  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
LIBVIRT_OUT  all  --  0.0.0.0/0            0.0.0.0/0

Chain CAPE_ACCEPTED_SEGMENTS (1 references)
target     prot opt source               destination
ACCEPT     all  --  192.168.122.1        192.168.122.106      /* CAPE-rooter */
ACCEPT     tcp  --  192.168.122.106      192.168.122.1        multiport dports 2042 /* CAPE-rooter */
ACCEPT     all  --  192.168.122.106      0.0.0.0/0            /* CAPE-rooter */
ACCEPT     all  --  0.0.0.0/0            192.168.122.106      /* CAPE-rooter */
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED /* CAPE-rooter */

Chain CAPE_REJECTED_SEGMENTS (1 references)
target     prot opt source               destination

Chain LIBVIRT_FWI (1 references)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
target     prot opt source               destination
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain LIBVIRT_INP (1 references)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:67

Chain LIBVIRT_OUT (1 references)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:68
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:68

Context

Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions. Operating system version, bitness, installed software versions, test sample details/hash/binary (if applicable).

Question Answer
Git commit fb34305ec3db3fd2b2db6dac10980a3539e95e8b
OS version Ubuntu 22.04.4 LTS
doomedraven commented 3 weeks ago

that might be related to recent changes, as i see you using NAT instead of hostonly, you might need to enable NAT and/or local_traffic in routing.conf

xme commented 3 weeks ago

These two parameters where missing in my routing.conf indeed. I'm not sure to understand the purpose of "nat". I tried both and it still the same. My setup is pretty simple: 192.168.122.0/24 is used by KVM through virbr0. 192.168.122.0/24 is NAT'd to reach other hosts via ens160 (my default gw)

Funny, when I start an analysis and keep a ping running to the VM, I see that, when the VM snapshot has been restored, 2 or 3 ICMP packets are accepted then... no traffic anymore! I looks like these packets are received before rooter.py makes the changes!

Here is my routing.conf:

[routing]

# Enable pcap generation for non live connections?
# If you have huge number of VMs, pcap generation can be a bottleneck
enable_pcap = no

# Default network routing mode; "none", "internet", or "vpn_name".
# In none mode we don't do any special routing - the VM doesn't have any
# network access (this has been the default actually for quite a while).
# In internet mode by default all the VMs will be routed through the network
# interface configured below (the "dirty line").
# And in VPN mode by default the VMs will be routed through the VPN identified
# by the given name of the VPN.
# Note that just like enabling VPN configuration setting this option to
# anything other than "none" requires one to run utils/rooter.py as root next
# to the CAPE instance (as it's required for setting up the routing).
route = internet

# Network interface that allows a VM to connect to the entire internet, the
# "dirty line" so to say. Note that, just like with the VPNs, this will allow
# malicious traffic through your network. So think twice before enabling it.
# (For example, to route all VMs through eth0 by default: "internet = eth0").
internet = ens160

# When set to no masquerade rule has been not generated.
nat = no

# When property nat set to yes. That property not used.
# When property nat set to no and no_local_routing to yes, a vrf configuration
# will be generated and local traffic will go through by internet interface (dirty_line).
no_local_routing = yes

# Routing table name/id for "dirty line" interface. If "dirty line" is
# also default gateway in the system you can leave "main" value. Otherwise add
# new routing table by adding "<id> <name>" line to /etc/iproute2/rt_tables
# (e.g., "200 eth0"). ID and name must be unique across the system (refer to
# /etc/iproute2/rt_tables for existing names and IDs).
rt_table = main

# When using "dirty line", you can reject forwarding to a certain network segment.
# For example, a request targeting 192.168.12.1/24,172.16.22.1/24 will not be
# forwarded, but will be rejected:
# "reject_segments = 192.168.12.1/24,172.16.22.1/24"
reject_segments = none

# When ussing "dirty line", you can reject guest access a certain port.
# For example, a request targeting host's port 8000 and 8080 will be rejected:
# "reject_hostports = 8000,8080"
reject_hostports = none

# To route traffic through multiple network interfaces CAPE uses
# Policy Routing with separate routing table for each output interface
# (VPN or "dirty line"). If this option is enabled CAPE on start will try
# to automatically initialise routing tables by copying routing entries from
# main routing table to the new routing tables. Depending on your network/vpn
# configuration this might not be sufficient. In such case you would need to
# initialise routing tables manually. Note that enabling this option won't
# affect main routing table.
auto_rt = no

# The drop route basically drops any outgoing network (except for CAPE
# traffic) whereas the regular none route still allows a VM to access its own
# subnet (e.g., 192.168.122.1/24). It is disabled by default as it does require
# the optional rooter to run (unlike the none route, where literally nothing
# happens). One can either explicitly enable the drop route or if the rooter
# is enabled anyway, it is automatically enabled.
drop = no

# Should check if the inteface is up
verify_interface = yes

# Should check if rt_table exists before initializing
verify_rt_table = yes

[inetsim]
# Inetsim quick deploy, chose your vm manager if is not kvm
# wget https://googledrive.com/host/0B6fULLT_NpxMQ1Rrb1drdW42SkE/remnux-6.0-ova-public.ova
# tar xvf remnux-6.0-ova-public.ova
# qemu-img convert -O qcow2 REMnuxV6-disk1.vmdk remnux.qcow2

enabled = no
server = 192.168.1.2
dnsport = 53
interface = virbr1
# Redirect TCP ports (should we also support UDP?). If specified, this should
# represent whitespace-separated src:dst pairs. E.g., "80:8080 443:8080" will
# redirect all 80/443 traffic to 8080 on the specified InetSim host.
# Source port range redirection is also supported. E.g., "996-2041:80" will
# redirect all traffic directed at ports between 996 and 2041 inclusive to port 80
# on the specified InetSim host.
ports =

[tor]
enabled = no
dnsport = 5353
proxyport = 9040
interface = virbr1

[vpn]
# By default we disable VPN support as it requires running utils/rooter.py as
# root next to cuckoo.py (which should run as regular user).
enabled = no

# select one of the configured vpns randomly
random_vpn = no

# Comma-separated list of the available VPNs.
vpns = vpn0

[vpn0]
# Name of this VPN. The name is represented by the filepath to the
# configuration file, e.g., cuckoo would represent /etc/openvpn/cuckoo.conf
# Note that you can't assign the names "none" and "internet" as those would
# conflict with the routing section in cuckoo.conf.
name = vpn0

# The description of this VPN which will be displayed in the web interface.
# Can be used to for example describe the country where this VPN ends up.
description = openvpn_tunnel

# The tun device hardcoded for this VPN. Each VPN *must* be configured to use
# a hardcoded/persistent tun device by explicitly adding the line "dev tunX"
# to its configuration (e.g., /etc/openvpn/vpn1.conf) where X in tunX is a
# unique number between 0 and your lucky number of choice.
interface = tun0

# Routing table name/id for this VPN. If table name is used it *must* be
# added to /etc/iproute2/rt_tables as "<id> <name>" line (e.g., "201 tun0").
# ID and name must be unique across the system (refer /etc/iproute2/rt_tables
# for existing names and IDs).
rt_table = tun0

[socks5]
# By default we disable socks5 support as it requires running utils/rooter.py as
# root next to cuckoo.py (which should run as regular user).
enabled = no

# select one of the configured socks5 proxies randomly
random_socks5 = no

# Comma-separated list of the available proxies.
proxies = socks_ch

[socks_ch]
name = ch_socks
description = ch_socks
proxyport = 5008
dnsport = 10053
doomedraven commented 3 weeks ago

i would suggest try to set nat=yes. My all servers are in hostonly mode, so with default values for those 2 entries everything works just fine. But in nat mode i guess you need to enable nat. restart router and cape services are required

xme commented 3 weeks ago

Switched NAT to "yes" and same behaviour... You said that your servers are in host only mode. How to you connect to the Internet then? (Ex: when the malware needs to download the next stage) When I use the VMs manually, they have Internet access. Is there a way to disable rooter.sh at all?

[Updated] I tested with "route = none" and it works now! I think that I misunderstood the purpose of this parameter. If you specify "none", rooter.sh does nothing and (in my case) it uses the system config -> Routing/NAT is performed by KVM.

doomedraven commented 3 weeks ago

We have cape-rooter service which enables desired network route for the vm, but denies vm to discover other vms in the same network. You can enable internet in routing.conf and select Internet, see documentation

El mar, 3 sept 2024, 14:36, Xavier Mertens @.***> escribió:

Switched NAT to "yes" and same behaviour... You said that your servers are in host only mode. How to you connect to the Internet then? (Ex: when the malware needs to download the next stage)

— Reply to this email directly, view it on GitHub https://github.com/kevoreilly/CAPEv2/issues/2304#issuecomment-2326413222, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOFH37DHJOUPU4EIJEFH3DZUWUMRAVCNFSM6AAAAABNQP43NWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRWGQYTGMRSGI . You are receiving this because you commented.Message ID: @.***>

doomedraven commented 3 weeks ago

so we can close this? in case that you see that any description can be improved, any suggestion is more than welcome

xme commented 3 weeks ago

Closed! Tx!