search

greenbone / gvmd

Greenbone Vulnerability Manager - The database backend for the Greenbone Community Edition
GNU Affero General Public License v3.0
286 stars 153 forks source link

`free(): invalid pointer` / `corrupted size vs. prev_size` when starting up gvmd #2190

Open cfi-gb opened 6 months ago

cfi-gb commented 6 months ago

Note: As the Greenbone Operating System is not affected (as it doesn't use the TCP communication within gvmd and gsad) this is raised here in case if the community wants to have a look at this / is stumbling over the same.

Expected behavior

  1. No continued restart / crash of the gvmd process
  2. No relevant log files in the logs

Actual behavior

  1. After doing a fresh build and/or feed sync the gvmd process seems to crash in the gvmd: Initializing phase
  2. The logs posted below can be seen

Steps to reproduce

Note: Not fully clear how to reproduce, but it is related to the usage of --listen (to make the gvmd process to listen on a TCP port instead of an Unix socket) together with --dh-params

  1. Start all GVM components including the gsad and gvmd using TCP sockets
    • For gvmd use the --listen and --dh-params (pointing to a file with a DH parameter length of 2048 bits) with a systemd file like e.g. gvmd.service.txt
    • For gsad use the --mlisten= and --mport= to point to the gvmd IP and port as well as the --dh-params to point to the same file mentioned previously
  2. Sync the feeds again
  3. From time to time (not always 100% reproducible) get the gvmd process crashed

To solve this problem the following helps from time to time (not always, maybe it's also only the restart of the gvmd process which helps):

  1. Stop gvmd
  2. gvmd --rebuild
  3. gvmd --rebuild-scap
  4. gvmd --rebuild-gvmd-data=all
  5. Start gvmd

GVM versions

gsa: Shouldn't be relevant

gvm: 23.6.3~dev1 (current main branch)

openvas-scanner: Shouldn't be relevant

gvm-libs: 22.10.0 (current main branch)

Environment

Operating system: Debian 12/bookworm

Installation method / source: source installation

Logfiles

Around the time this issue occurs the following entries can be seen in /var/log/syslog:

2024-04-30T18:57:56.858912+02:00 <redacted> gvmd[383564]: free(): invalid pointer
2024-04-30T19:00:16.359358+02:00 <redacted> gvmd[383698]: corrupted size vs. prev_size

and since #2184 the following additional ones in the gvmd.log:

md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: gvmd: Initializing(+0x6c848) [0x60bebe406848]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x75c2d347f050]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /lib/x86_64-linux-gnu/libc.so.6(+0x8ae2c) [0x75c2d34cde2c]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12) [0x75c2d347efb2]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x75c2d3469472]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /lib/x86_64-linux-gnu/libc.so.6(+0x7f430) [0x75c2d34c2430]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /lib/x86_64-linux-gnu/libc.so.6(+0x947aa) [0x75c2d34d77aa]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /lib/x86_64-linux-gnu/libc.so.6(+0x9506e) [0x75c2d34d806e]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /lib/x86_64-linux-gnu/libc.so.6(+0x966db) [0x75c2d34d96db]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /lib/x86_64-linux-gnu/libc.so.6(__libc_free+0x6f) [0x75c2d34dbe8f]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /opt/gvm/lib/libgvm_util.so.22(unload_gnutls_file+0x22) [0x75c2d36ce63c]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /opt/gvm/lib/libgvm_util.so.22(set_gnutls_dhparams+0x7b) [0x75c2d36d0192]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: gvmd: Initializing(gvmd+0x247a) [0x60bebe40a8ea]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: gvmd: Initializing(main+0x28) [0x60bebe405671]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x75c2d346a24a]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x75c2d346a305]
md   main:MESSAGE:2024-04-30 19h00.16 utc:598: BACKTRACE: gvmd: Initializing(_start+0x21) [0x60bebe405581]
md manage:MESSAGE:2024-04-30 19h00.16 utc:598: Received Aborted signal
cfi-gb commented 4 months ago

As it got "worse" over the last couple of weeks (even if the gvmd started up correctly the start of a scan caused 10-15 opened / hanging gvmd processes like Serving Client and scans never finishing completely) i had disabled the usage of TCP communication and use the Unix socket communication.

After doing so the issue is gone so this has definitely something to do with the (currently basically unsupported) TCP communication between the components. I have adjusted / extended the issue description around this a little.