[Bug] openvas and nmap processes traps or segfaults frequently

[tmuncks]

Describe the bug When scanning, often the process is "interrupted" before completing, and messages such as these shows up in dmesg on the host.

openvas randomly segfaults, and nmap segfaults or traps. The results are unreliable, as hosts or services are missed, even if the scanning does not end up as interrupted.

[ 2693.808760] traps: nmap[35860] trap invalid opcode ip:55817d153440 sp:7fffb5e326e0 error:0 in nmap[55817d105000+8e000]
[ 3359.422992] traps: nmap[49098] trap invalid opcode ip:564ed9eca440 sp:7ffe1c1e2010 error:0 in nmap[564ed9e7c000+8e000]
[ 3362.479530] traps: nmap[49100] trap invalid opcode ip:55abef014440 sp:7fffbe46dce0 error:0 in nmap[55abeefc6000+8e000]
[ 3440.949354] traps: nmap[49061] trap invalid opcode ip:55fd5d253440 sp:7fff20e88640 error:0 in nmap[55fd5d205000+8e000]
[ 3453.341204] nmap[49004]: segfault at 7efe666dd020 ip 000055dfacbe9448 sp 00007fffde1dc920 error 4 in nmap[55dfacb9b000+8e000]
[ 3453.341216] Code: 24 08 0f b7 cb 48 39 e8 73 09 48 8d 14 08 48 39 d5 72 0f 48 39 e8 76 0c 48 8d 54 0d 00 48 39 d0 73 02 0f 0b 48 89 c7 48 89 ee <f3> a4 66 41 c7 44 24 06 00 00 4c 89 e0 5a 5b 5d 41 5c 41 5d 41 5e
[ 4191.012447] traps: nmap[70547] trap invalid opcode ip:55949bb89440 sp:7ffc176d6d00 error:0 in nmap[55949bb3b000+8e000]
[ 5130.462535] traps: nmap[91191] trap invalid opcode ip:558715bcc440 sp:7fff806baaf0 error:0 in nmap[558715b7e000+8e000]
[ 7261.504889] traps: nmap[127905] trap invalid opcode ip:5626d9ef7440 sp:7ffc469912b0 error:0 in nmap[5626d9ea9000+8e000]
[ 9403.617868] openvas[2488]: segfault at 7f498fae1d80 ip 00007f4991a76313 sp 00007fff87ed2610 error 4 in libgvm_base.so.21.4.2[7f4991a74000+8000]
[ 9403.617879] Code: 5d 41 5e 41 5f c3 55 48 89 fd bf 10 00 00 00 53 48 89 f3 50 e8 1e df ff ff 48 89 28 48 89 58 08 5a 5b 5d c3 48 85 ff 74 2b 55 <83> 7f 10 00 48 89 fd 75 08 48 8b 3f e8 94 e8 ff ff 48 8b 7d 18 48
[20059.115781] traps: openvas[161562] general protection fault ip:7f627b0a1896 sp:7fffe0c00948 error:0 in ld-musl-x86_64.so.1[7f627b093000+48000]
[20338.874947] traps: nmap[167472] trap invalid opcode ip:55816621e440 sp:7ffef834f8e0 error:0 in nmap[5581661d0000+8e000]
[39743.448666] traps: openvas[48511] general protection fault ip:7fb82608d896 sp:7fffb961cff8 error:0 in ld-musl-x86_64.so.1[7fb82607f000+48000]
[42038.536454] openvas[72556]: segfault at 0 ip 00007f2fcd70a081 sp 00007ffdc4a7f5a8 error 4 in ld-musl-x86_64.so.1[7f2fcd6d0000+48000]
[42038.536467] Code: 20 c7 44 24 0c 30 00 00 00 48 89 44 24 18 e8 ce fe ff ff 48 81 c4 d8 00 00 00 c3 89 f8 99 31 d0 29 d0 c3 31 f6 e9 7d 09 00 00 <0f> be 17 89 d0 83 ea 09 83 fa 04 77 05 48 ff c7 eb ee 3c 20 74 f7
[54412.214396] traps: openvas[144887] general protection fault ip:7f3844d4f896 sp:7ffccc6df338 error:0 in ld-musl-x86_64.so.1[7f3844d41000+48000]
[81753.907458] traps: nmap[330293] trap invalid opcode ip:5609dabcc440 sp:7fff86095920 error:0 in nmap[5609dab7e000+8e000]
[81852.061828] nmap[330299]: segfault at 7f1fb29a0020 ip 00005589dbcb2448 sp 00007ffe740c76a0 error 4 in nmap[5589dbc64000+8e000]
[81852.061866] Code: 24 08 0f b7 cb 48 39 e8 73 09 48 8d 14 08 48 39 d5 72 0f 48 39 e8 76 0c 48 8d 54 0d 00 48 39 d0 73 02 0f 0b 48 89 c7 48 89 ee <f3> a4 66 41 c7 44 24 06 00 00 4c 89 e0 5a 5b 5d 41 5c 41 5d 41 5e
[81915.730041] traps: nmap[330307] trap invalid opcode ip:559711ad3440 sp:7ffc79e170b0 error:0 in nmap[559711a85000+8e000]
[82046.006118] traps: nmap[339261] trap invalid opcode ip:55a2f8890440 sp:7fffdbbab200 error:0 in nmap[55a2f8842000+8e000]
[82205.134292] traps: nmap[347358] trap invalid opcode ip:557c04521440 sp:7ffc7aa75510 error:0 in nmap[557c044d3000+8e000]
[82251.198955] traps: nmap[351206] trap invalid opcode ip:562658bea440 sp:7fff706de6e0 error:0 in nmap[562658b9c000+8e000]
[82404.256601] traps: nmap[363056] trap invalid opcode ip:55ac34db6440 sp:7ffe30cf4f70 error:0 in nmap[55ac34d68000+8e000]
[82960.779933] traps: nmap[376132] trap invalid opcode ip:56457442a440 sp:7ffcee51e470 error:0 in nmap[5645743dc000+8e000]
[83482.088115] traps: nmap[396343] trap invalid opcode ip:559faf5fc440 sp:7fff817eb360 error:0 in nmap[559faf5ae000+8e000]
[83485.654411] traps: nmap[398085] trap invalid opcode ip:5631cbbba440 sp:7ffdb22b3ad0 error:0 in nmap[5631cbb6c000+8e000]
[83701.749437] nmap[404134]: segfault at 7fde00003020 ip 0000559481cdb448 sp 00007ffe68d6c540 error 4 in nmap[559481c8d000+8e000]
[83701.749447] Code: 24 08 0f b7 cb 48 39 e8 73 09 48 8d 14 08 48 39 d5 72 0f 48 39 e8 76 0c 48 8d 54 0d 00 48 39 d0 73 02 0f 0b 48 89 c7 48 89 ee <f3> a4 66 41 c7 44 24 06 00 00 4c 89 e0 5a 5b 5d 41 5c 41 5d 41 5e
[84947.667435] traps: nmap[432432] trap invalid opcode ip:5566ae0a1440 sp:7ffd2a52f780 error:0 in nmap[5566ae053000+8e000]
[85905.583853] nmap[448314]: segfault at 7fed19cca020 ip 000055a60fbde448 sp 00007ffd5e8935e0 error 4 in nmap[55a60fb90000+8e000]
[85905.583864] Code: 24 08 0f b7 cb 48 39 e8 73 09 48 8d 14 08 48 39 d5 72 0f 48 39 e8 76 0c 48 8d 54 0d 00 48 39 d0 73 02 0f 0b 48 89 c7 48 89 ee <f3> a4 66 41 c7 44 24 06 00 00 4c 89 e0 5a 5b 5d 41 5c 41 5d 41 5e
[86291.951492] traps: nmap[458906] trap invalid opcode ip:55b732c21440 sp:7fffdad874c0 error:0 in nmap[55b732bd3000+8e000]
[88792.139689] traps: openvas[376047] general protection fault ip:7f6f0cdb1896 sp:7ffce68edc48 error:0 in ld-musl-x86_64.so.1[7f6f0cda3000+48000]
[91899.867229] traps: nmap[489571] trap invalid opcode ip:5641d022e440 sp:7fff95dfe730 error:0 in nmap[5641d01e0000+8e000]
[91918.141517] traps: nmap[489592] trap invalid opcode ip:560cb5499440 sp:7ffe286d8cd0 error:0 in nmap[560cb544b000+8e000]
[91948.451528] traps: nmap[489598] trap invalid opcode ip:56160af38440 sp:7fffb7ed41a0 error:0 in nmap[56160aeea000+8e000]
[91967.426301] traps: nmap[489580] trap invalid opcode ip:55cfa6100440 sp:7fffd6ec3110 error:0 in nmap[55cfa60b2000+8e000]
[92294.752484] traps: nmap[491526] trap invalid opcode ip:5626df88f440 sp:7fffced184e0 error:0 in nmap[5626df841000+8e000]
[92589.964141] traps: nmap[493348] trap invalid opcode ip:56508ad73440 sp:7ffd26f2c600 error:0 in nmap[56508ad25000+8e000]
[93833.598764] traps: openvas[489491] general protection fault ip:7f3b38d8c896 sp:7ffd272f0c58 error:0 in ld-musl-x86_64.so.1[7f3b38d7e000+48000]

To Reproduce Steps to reproduce the behavior:

1. Just running a simple scanning.

Target has 32 IP addresses Alive Test: Consider Alive Scanning Config: Full and fast (unmodified)

Expected behavior Finalized scanning with no code errors, and all hosts covered in the result.

Host Device:

• OS: Ubuntu 20.04 LTS

Image in use:

            "Labels": {
                "org.opencontainers.image.created": "2021-08-08T20:27:54.303Z",
                "org.opencontainers.image.description": "Greenbone Vulnerability Management Docker Image with OpenVAS",
                "org.opencontainers.image.licenses": "MIT",
                "org.opencontainers.image.revision": "45166f1b9bc1c5314303b99ee1a94bb7552bc153",
                "org.opencontainers.image.source": "https://github.com/Secure-Compliance-Solutions-LLC/GVM-Docker",
                "org.opencontainers.image.title": "GVM-Docker",
                "org.opencontainers.image.url": "https://github.com/Secure-Compliance-Solutions-LLC/GVM-Docker",
                "org.opencontainers.image.version": "21.4.3-v1"
            }

[Dexus]

Can you provide a coredump?

Maybe you need also to add --cap-add=SYS_PTRACE to docker run or eq in docker-compose file. Maybe also this is related to https://github.com/greenbone/openvas-scanner/issues/825

[tmuncks]

I'll look into the test_alive_hosts_only = no option and try to get a crash dump...

[Dexus]

I checked all hosts.

Fun Fact: It not segfault on all hosts even if they run already for days with this version. But on some I see the same error.

[tmuncks]

That's odd... I'm running a couple of scans right now, to hopefully collect coredumps of the openvas segfault, nmap segfault and nmap trap.

[Dexus]

I will check in the meantime if this also occurs with nmap 7.92.

[tmuncks]

cores.tar.gz

Okay, so here are a couple of nmap coredumps, just to confirm it's the same problem in all of them. And a single openvas coredump, which was the only one encountered on this particular run.

[tmuncks]

core.openvas.46751.gz

Just adding another openvas dump, to see if these are different problems

[Dexus]

thank you, hope that helps.

[Dexus]

@tmuncks Can you please check the system side cpu microcode?

[tmuncks]

Sure. What would you like me to check? This is a virtual machine running under VMware ESXi.

[Dexus]

Version? I'm not 100% sure, but maybe there is a problem. Because they some systems where i see the traps have different versions.

[austinsonger]

How much resources do you have assigned to your virtual machine.

[Dexus]

The machines I have access to are mostly up to 192GiB, locally and on some client servers I have 16-32 GB and some are running almost at 80~95% RAM usage.

Do you think that there are peaks and that's why the segfaults and traps happen? At least I don't see any OOM messages.

[tmuncks]

My VM is much smaller at 6GB memory, but I'm only ever running a single scanning at a time (4 hosts, 4 NVTs simultaneously). Memory consumption does not look problematic here, however; No swapping, no OOM.

[tmuncks]

Version? I'm not 100% sure, but maybe there is a problem. Because they some systems where i see the traps have different versions.

VMware ESXi, 7.0.2, 17867351

[tmuncks]

Gut feeling is some compatibility issue between Ubuntu kernel/system libraries and the Alpine base image.

I have never seen anything like this before myself, but these guys are seeing something similar: https://github.com/soedinglab/hh-suite/issues/104 - Their solution has been a switch to a Debian base.

This doesn't explain the problem though, but FWIW, I tried a different GVM docker image based on Ubuntu, and I don't see any segfaults there.

[austinsonger]

Have switch to Debian. Closing this.