draios / sysdig

Linux system exploration and troubleshooting tool with first class support for containers
http://www.sysdig.com/
Other
7.72k stars 726 forks source link

Sysdig only tracks 'switch' events or none for an executable #357

Closed boyokmard closed 1 year ago

boyokmard commented 9 years ago

One of my servers was infected by a malware. I had sysdig running on the server for whole time. I dig out further and find out sysdig did not capture any events related to executable. I managed to extract the malware and run it manually and realised still sysdig does not capture any event but only 'switch'. You can see below output of strace, lsof, ps and sysdig.

ps -ef --forest

root      1397     1  0 17:02 ?        00:00:00 upstart-udev-bridge --daemon
root      1400     1  0 17:02 ?        00:00:00 upstart-file-bridge --daemon
root      1403     1  0 17:02 ?        00:00:00 upstart-socket-bridge --daemon
root     13308     1  0 19:10 ?        00:00:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
[snip]
root     13694     1 96 19:31 ?        00:33:03 id                 
root     20271     1  0 20:06 ?        00:00:00 bash                          
root     20274     1  0 20:06 ?        00:00:00 id                          
root     20277     1  0 20:06 ?        00:00:00 su                          
root     20279     1  0 20:06 ?        00:00:00 id                          
root     20280     1  0 20:06 ?        00:00:00 grep "A"                          
root     20288     1  0 20:06 ?        00:00:00 uptime                          
root     20291     1  0 20:06 ?        00:00:00 pwd                          
root     20293     1  0 20:06 ?        00:00:00 sh                          
root     20294     1  0 20:06 ?        00:00:00 sleep 1 

Malware hide itself under id pid = 13694

I had sysdig running with no filter before running the malware, and it didn't record any event related to pid 13694. Later, I ran strace and then sysdig pointing both to the pid. See the outputs below.

strace -p 13694 http://pastebin.aquilenet.fr/?ad0bc95929856832#ZP4Du/Zc/jIeXV7eWOTB2zdYMWO1IRYdF2GvbHFTH64=

sysdig "proc.pid=13694" http://pastebin.aquilenet.fr/?4e525c6958eae9d3#9eeCQbc/BBujbT2RMiwz3evJyDNOcjYrw5QcnC5s1mI= (only switch calls)

lsof -p 13694

COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
ymthytscl 13694 root  cwd    DIR  252,0     4096  22536 /tmp/bin
ymthytscl 13694 root  rtd    DIR  252,0     4096      2 /
ymthytscl 13694 root  txt    REG  252,0   617704 289985 /usr/bin/ymthytsclk
ymthytscl 13694 root    0u   CHR    1,3      0t0   5294 /dev/null
ymthytscl 13694 root    1u   CHR    1,3      0t0   5294 /dev/null
ymthytscl 13694 root    2u   CHR    1,3      0t0   5294 /dev/null

Any other thing I can do to figure out the root cause of this issue? Is this behaviour something to do with sysdig's kernel driver? or by default it ignores some events?

It seems like this malware propagate also through some library (.so) file and use some process hiding techniques (if you want to re-produce the output, contact me to get a copy it).

I am using sysdig 0.1.99 under Ubuntu 14.04.1 64bit.

ldegio commented 9 years ago

Can you check if the binary is a 32bit one? http://superuser.com/questions/791506/how-to-determine-if-a-linux-binary-file-is-32-bit-or-64-bit

32bit executables use a different path for system calls on 64bit systems, and sysdig doesn't support capturing it yet. As a consequence, events for 32bit processes are not shown. There's an issue open here: https://github.com/draios/sysdig/issues/14.

Loris

On 5/9/2015 3:40 AM, boyokmard wrote:

One of my servers was infected by a malware. I had sysdig running on the server for whole time. I dig out further and find out sysdig did not capture any event related to executable. I managed to extract the malware and run it manually and realised still sysdig does not capture any event but only 'switch'. You can see below output of strace, lsof, ps and sysdig.

ps -ef --forest

root 1397 1 0 17:02 ? 00:00:00 upstart-udev-bridge --daemon root 1400 1 0 17:02 ? 00:00:00 upstart-file-bridge --daemon root 1403 1 0 17:02 ? 00:00:00 upstart-socket-bridge --daemon root 13308 1 0 19:10 ? 00:00:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 [snip] root 13694 1 96 19:31 ? 00:33:03 id root 20271 1 0 20:06 ? 00:00:00 bash root 20274 1 0 20:06 ? 00:00:00 id root 20277 1 0 20:06 ? 00:00:00 su root 20279 1 0 20:06 ? 00:00:00 id root 20280 1 0 20:06 ? 00:00:00 grep "A" root 20288 1 0 20:06 ? 00:00:00 uptime root 20291 1 0 20:06 ? 00:00:00 pwd root 20293 1 0 20:06 ? 00:00:00 sh root 20294 1 0 20:06 ? 00:00:00 sleep 1

Malware hide itself under |id| pid = 13694

I had sysdig running with no filter before running the malware, and it didn't record any event related to pid 13694. I ran strace and then sysdig pointing both to the pid.

strace -p 13694 http://pastebin.aquilenet.fr/?ad0bc95929856832#ZP4Du/Zc/jIeXV7eWOTB2zdYMWO1IRYdF2GvbHFTH64=

sysdig "proc.pid=13694" http://pastebin.aquilenet.fr/?4e525c6958eae9d3#9eeCQbc/BBujbT2RMiwz3evJyDNOcjYrw5QcnC5s1mI= (only switch calls)

lsof -p 13694

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ymthytscl 13694 root cwd DIR 252,0 4096 22536 /tmp/bin ymthytscl 13694 root rtd DIR 252,0 4096 2 / ymthytscl 13694 root txt REG 252,0 617704 289985 /usr/bin/ymthytsclk ymthytscl 13694 root 0u CHR 1,3 0t0 5294 /dev/null ymthytscl 13694 root 1u CHR 1,3 0t0 5294 /dev/null ymthytscl 13694 root 2u CHR 1,3 0t0 5294 /dev/null

Is this behaviour something to do with sysdig's kernel driver? or by default it ignores some events? It seems like this malware propagate also through some library (.so) file and use some process hiding techniques (if you want to re-produce the output, contact me to get a copy it).

I am using sysdig 0.1.99 under Ubuntu 14.04.1 64bit.

— Reply to this email directly or view it on GitHub https://github.com/draios/sysdig/issues/357.

boyokmard commented 9 years ago

Thanks Loris.

/usr/bin/ymthytsclk: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, not stripped it is indeed a 32bit executable.

That puts me in a limit as I can see this issue would happen again. Also there are going to be other legitimate 32bit binaries in use that I may not be be able to track. One option I can think off is to disable 32bit emulation (is it possible without recompiling the kernel?).

Do you have any suggestions or a workaround until it gets supported by sysdig?

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.