elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.15k stars 4.91k forks source link

[Auditbeat] Unable to start; unable to guess one or more required parameters: guess_ip_local_out failed #18755

Closed Aqualie closed 2 years ago

Aqualie commented 4 years ago

Linux 5.6.14-arch1-1 Auditbeat 7.7.0

May 26 18:33:36 REPLACED systemd[1]: Started Audit the activities of users and processes on your system..
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.632-0400        INFO        instance/beat.go:621        Home path: [/opt/elastic/auditbeat] Config path: [/opt/elastic/auditbeat/conf] Data path: [/opt/elastic/auditbeat/data] Logs path: [/opt/elastic/auditbeat/logs]
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.632-0400        INFO        instance/beat.go:629        Beat ID: REPLACED
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.663-0400        INFO        [seccomp]        seccomp/seccomp.go:124        Syscall filter successfully installed
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.663-0400        INFO        [beat]        instance/beat.go:957        Beat info        {"system_info": {"beat": {"path": {"config": "/opt/elastic/auditbeat/conf", "data": "/opt/elastic/auditbeat/data", "home": "/opt/elastic/auditbeat", "logs": "/opt/elastic/auditbeat/logs"}, "type": "auditbeat", "uuid": "REPLACED"}}}
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.663-0400        INFO        [beat]        instance/beat.go:966        Build info        {"system_info": {"build": {"commit": "5e69e25b920e3d93bec76a09a31da3ab35a55607", "libbeat": "7.7.0", "time": "2020-05-17T13:09:33.000Z", "version": "7.7.0"}}}
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.663-0400        INFO        [beat]        instance/beat.go:969        Go runtime info        {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.13.9"}}}
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.664-0400        INFO        [beat]        instance/beat.go:973        Host info        {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-05-26T14:13:53-04:00","containerized":false,"name":"REPLACED","ip":["127.0.0.1/8","169.254.169.22/16","REPLACED","REPLACED","REPLACED","172.17.0.1/16","172.18.0.1/16"],"kernel_version":"5.6.14-arch1-1","mac":["REPLACED","REPLACED","REPLACED","REPLACED","REPLACED","REPLACED","REPLACED","REPLACED","REPLACED"],"os":{"family":"","platform":"arch","name":"Arch Linux","version":"","major":0,"minor":0,"patch":0,"build":"rolling"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"REPLACED"}}}
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.664-0400        INFO        [beat]        instance/beat.go:1002        Process info        {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/", "exe": "/opt/elastic/auditbeat/auditbeat", "name": "auditbeat", "pid": 182610, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-05-26T18:33:36.410-0400"}}}
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.664-0400        INFO        instance/beat.go:297        Setup Beat: auditbeat; Version: 7.7.0
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.665-0400        INFO        [publisher]        pipeline/module.go:110        Beat name: REPLACED
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.667-0400        INFO        [auditd]        auditd/audit_linux.go:106        auditd module is running as euid=0 on kernel=5.6.14-arch1-1
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.667-0400        INFO        [auditd]        auditd/audit_linux.go:133        socket_type=unicast will be used.
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.668-0400        WARN        [cfgwarn]        host/host.go:167        BETA: The system/host dataset is beta
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.670-0400        WARN        [cfgwarn]        login/login.go:95        BETA: The system/login dataset is beta
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.672-0400        WARN        [cfgwarn]        user/user.go:205        BETA: The system/user dataset is beta
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.674-0400        WARN        [cfgwarn]        process/process.go:131        BETA: The system/process dataset is beta
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.675-0400        WARN        [cfgwarn]        socket/socket_linux.go:87        BETA: The system/socket dataset is beta.
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.694-0400        INFO        [socket]        socket/socket_linux.go:227        Setting up system/socket for kernel 5.6.14-arch1-1
May 26 18:33:36 REPLACED auditbeat[182610]: 2020-05-26T18:33:36.956-0400        INFO        [socket]        guess/guess.go:258        Running 17 guesses ...
May 26 18:33:36 REPLACED auditbeat[182311]: 2020-05-26T18:33:36.300-0400        INFO        instance/beat.go:411        auditbeat stopped.
May 26 18:33:36 REPLACED auditbeat[182311]: 2020-05-26T18:33:36.300-0400        ERROR        instance/beat.go:932        Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_ip_local_out failed: timeout while waiting for event
May 26 18:33:36 REPLACED auditbeat[182311]: Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_ip_local_out failed: timeout while waiting for event
May 26 18:33:36 REPLACED systemd[1]: auditbeat.service: Main process exited, code=exited, status=1/FAILURE
May 26 18:33:36 REPLACED systemd[1]: auditbeat.service: Failed with result 'exit-code'.
May 26 18:33:36 REPLACED systemd[1]: auditbeat.service: Scheduled restart job, restart counter is at 1.
May 26 18:33:36 REPLACED systemd[1]: Stopped Audit the activities of users and processes on your system..
elasticmachine commented 4 years ago

Pinging @elastic/siem (Team:SIEM)

BBQigniter commented 4 years ago

to finally be able to test auditbeat on my Fedora 32 machine I had to disable "socket"-stuff completely by setting

- module: system
  datasets:
    - host    # General host information, e.g. uptime, IPs
    - login   # User logins, logouts, and system boots.
    - process # Started and stopped processes
#    - socket  # Opened and closed sockets
    - user    # User information
kimsyversen commented 4 years ago

I can confirm similar behaviour on Ubuntu 20.04 Server LTS.

Aqualie commented 4 years ago

Still broken in 7.8

nickchappell commented 4 years ago

I'm running into this as well on CentOS 8.1, 5.6 kernel.

MakoWish commented 3 years ago

Same on Kali Linux 2020.3 (Debian) using Auditbeat 7.9.2.

legoguy1000 commented 3 years ago

same on centos 7.8 with auditbeat 7.8.1 and 7.9.2

xennn commented 3 years ago

Same on Kali 2020.3 with Auditbeat 7.9.3

adriansr commented 3 years ago

Thanks all for your reports. Can you please provide the kernel version (uname -a)?

Also, does it always fail for the same "guess", or do you get different guess names in the error:

unable to guess one or more required parameters: guess_ip_local_out failed

Does adding socket.guess_timeout: 1m make a difference?

 - module: system
   datasets:
     - host    # General host information, e.g. uptime, IPs
     - login   # User logins, logouts, and system boots.
     - process # Started and stopped processes
     - socket  # Opened and closed sockets
     - user    # User information

   # How often datasets send state updates with the
   # current state of the system (e.g. all currently
   # running processes, all open sockets).
   state.period: 12h

   # Enabled by default. Auditbeat will read password fields in
   # /etc/passwd and /etc/shadow and store a hash locally to
   # detect any changes.
   user.detect_password_changes: true

   # File patterns of the login record files.
   login.wtmp_file_pattern: /var/log/wtmp*
   login.btmp_file_pattern: /var/log/btmp*
+  socket.guess_timeout: 1m
Aqualie commented 3 years ago

Thanks all for your reports. Can you please provide the kernel version (uname -a)?

Also, does it always fail for the same "guess", or do you get different guess names in the error:

unable to guess one or more required parameters: guess_ip_local_out failed

Does adding socket.guess_timeout: 1m make a difference?

 - module: system
   datasets:
     - host    # General host information, e.g. uptime, IPs
     - login   # User logins, logouts, and system boots.
     - process # Started and stopped processes
     - socket  # Opened and closed sockets
     - user    # User information

   # How often datasets send state updates with the
   # current state of the system (e.g. all currently
   # running processes, all open sockets).
   state.period: 12h

   # Enabled by default. Auditbeat will read password fields in
   # /etc/passwd and /etc/shadow and store a hash locally to
   # detect any changes.
   user.detect_password_changes: true

   # File patterns of the login record files.
   login.wtmp_file_pattern: /var/log/wtmp*
   login.btmp_file_pattern: /var/log/btmp*
+  socket.guess_timeout: 1m

5.8.3-arch-1-1

Added in socket.guess_timeout: 1m

Nov 06 06:57:55 xxxxxx auditbeat[3173119]: 2020-11-06T06:57:55.991-0500        INFO        [publisher]        pipeline/module.go:113        Beat name: xxxxxx
Nov 06 06:57:55 xxxxxx auditbeat[3173119]: 2020-11-06T06:57:55.996-0500        INFO        [auditd]        auditd/audit_linux.go:106        auditd module is running as euid=0 on kernel=5.8.3-arch1-1
Nov 06 06:57:55 xxxxxx auditbeat[3173119]: 2020-11-06T06:57:55.997-0500        INFO        [auditd]        auditd/audit_linux.go:133        socket_type=unicast will be used.
Nov 06 06:57:56 xxxxxx auditbeat[3173119]: 2020-11-06T06:57:56.000-0500        WARN        [cfgwarn]        host/host.go:184        BETA: The system/host dataset is beta
Nov 06 06:57:56 xxxxxx auditbeat[3173119]: 2020-11-06T06:57:56.001-0500        WARN        [cfgwarn]        login/login.go:95        BETA: The system/login dataset is beta
Nov 06 06:57:56 xxxxxx auditbeat[3173119]: 2020-11-06T06:57:56.002-0500        WARN        [cfgwarn]        user/user.go:232        BETA: The system/user dataset is beta
Nov 06 06:57:56 xxxxxx auditbeat[3173119]: 2020-11-06T06:57:56.005-0500        WARN        [cfgwarn]        process/process.go:146        BETA: The system/process dataset is beta
Nov 06 06:57:56 xxxxxx auditbeat[3173119]: 2020-11-06T06:57:56.006-0500        WARN        [cfgwarn]        socket/socket_linux.go:87        BETA: The system/socket dataset is beta.
Nov 06 06:57:56 xxxxxx auditbeat[3173119]: 2020-11-06T06:57:56.027-0500        INFO        [socket]        socket/socket_linux.go:227        Setting up system/socket for kernel 5.8.3-arch1-1
Nov 06 06:57:56 xxxxxx auditbeat[3173119]: 2020-11-06T06:57:56.399-0500        INFO        [socket]        guess/guess.go:258        Running 17 guesses ...
Nov 06 06:59:00 xxxxxx auditbeat[3173119]: 2020-11-06T06:59:00.026-0500        INFO        instance/beat.go:436        auditbeat stopped.
Nov 06 06:59:00 xxxxxx auditbeat[3173119]: 2020-11-06T06:59:00.026-0500        ERROR        instance/beat.go:958        Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_ip_local_out failed: timeout while waiting for event
Nov 06 06:59:00 xxxxxx auditbeat[3173119]: Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_ip_local_out failed: timeout while waiting for event
Nov 06 06:59:00 xxxxxx systemd[1]: auditbeat.service: Main process exited, code=exited, status=1/FAILURE
Nov 06 06:59:00 xxxxxx systemd[1]: auditbeat.service: Failed with result 'exit-code'.
Nov 06 06:59:00 xxxxxx systemd[1]: auditbeat.service: Scheduled restart job, restart counter is at 1.
xennn commented 3 years ago

Uninstall old Packeage with dpkg -P auditbeat. Then reinstall works for me! Then the service start. But the config has now problems:

root@hostname:/etc/auditbeat# auditbeat test config Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_ip_local_out failed: timeout while waiting for event

With Kernel Linux 5.8.0-kali2-amd64 #1 SMP Debian 5.8.10-1kali1 (2020-09-22) x86_64 GNU/Linux

adriansr commented 3 years ago

There's a few things that can be going wrong:

rcmelendez commented 3 years ago

Same error on Fedora 33, kernel 5.8.16-300.fc33.x86_64 and auditbeat 7.9.3. I also added socket.guess_timeout: 1m but the same.

Aqualie commented 3 years ago

There's a few things that can be going wrong:

  • Check that /proc/sys/kernel/ftrace_enabled is 1.
  • Check that /sys/kernel/debug/kprobes/enabled is 1.
  • Make sure that the system/socket dataset is started from an auditbeat.modules entry in auditbeat.yml (default), and not from a separate file in modules.d via auditbeat.config.modules. That will fail due to #20851

cat /sys/kernel/debug/kprobes/enabled 1 sudo cat /sys/kernel/debug/kprobes/enabled 1

system/socket is being run from auditbeat.yml

111andre111 commented 3 years ago

@adriansr I have tested this one on Archlinux with Kernel Linux archlinux 5.9.10-arch1-1 #1 SMP PREEMPT Sun, 22 Nov 2020 14:16:59 +0000 x86_64 GNU/Linux by executing only the socket like this:

touch empty.yml
./auditbeat -c empty.yml -E auditbeat.modules.0.module=system -E auditbeat.modules.0.datasets.0=socket -E output.console.pretty=true -e -d '*' > 2>&1 >auditbeat-debug.log

And under Auditbeat 7.7.0 it crashed immediately: auditbeat-debug.log error message was:

ERROR   instance/beat.go:932    Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_sk_buff_proto failed: failed to add kprobe 'r:guess_recv_datagram {{.RECV_UDP_DATAGRAM}} +0({{.RET}}):u64 +8({{.RET}}):u64 +16({{.RET}}):u64 +24({{.RET}}):u64 +32({{.RET}}):u64 +40({{.RET}}):u64 +48({{.RET}}):u64 +56({{.RET}}):u64 +64({{.RET}}):u64 +72({{.RET}}):u64 +80({{.RET}}):u64 +88({{.RET}}):u64 +96({{.RET}}):u64 +104({{.RET}}):u64 +112({{.RET}}):u64 +120({{.RET}}):u64 +128({{.RET}}):u64 +136({{.RET}}):u64 +144({{.RET}}):u64 +152({{.RET}}):u64 +160({{.RET}}):u64 +168({{.RET}}):u64 +176({{.RET}}):u64 +184({{.RET}}):u64 +192({{.RET}}):u64 +200({{.RET}}):u64 +208({{.RET}}):u64 +216({{.RET}}):u64 +224({{.RET}}):u64 +232({{.RET}}):u64 +240({{.RET}}):u64 +248({{.RET}}):u64 +256({{.RET}}):u64 +264({{.RET}}):u64 +272({{.RET}}):u64 +280({{.RET}}):u64 +288({{.RET}}):u64 +296({{.RET}}):u64 +304({{.RET}}):u64 +312({{.RET}}):u64 +320({{.RET}}):u64 +328({{.RET}}):u64 +336({{.RET}}):u64 +344({{.RET}}):u64 +352({{.RET}}):u64 +360({{.RET}}):u64 +368({{.RET}}):u64 +376({{.RET}}):u64 +384({{.RET}}):u64 +392({{.RET}}):u64 +400({{.RET}}):u64 +408({{.RET}}):u64 +416({{.RET}}):u64 +424({{.RET}}):u64 +432({{.RET}}):u64 +440({{.RET}}):u64 +448({{.RET}}):u64 +456({{.RET}}):u64 +464({{.RET}}):u64 +472({{.RET}}):u64 +480({{.RET}}):u64 +488({{.RET}}):u64 +496({{.RET}}):u64 +504({{.RET}}):u64 +512({{.RET}}):u64 +520({{.RET}}):u64 +528({{.RET}}):u64 +536({{.RET}}):u64 +544({{.RET}}):u64 +552({{.RET}}):u64 +560({{.RET}}):u64 +568({{.RET}}):u64 +576({{.RET}}):u64 +584({{.RET}}):u64 +592({{.RET}}):u64 +600({{.RET}}):u64 +608({{.RET}}):u64 +616({{.RET}}):u64 +624({{.RET}}):u64 +632({{.RET}}):u64 +640({{.RET}}):u64 +648({{.RET}}):u64 +656({{.RET}}):u64 +664({{.RET}}):u64 +672({{.RET}}):u64 +680({{.RET}}):u64 +688({{.RET}}):u64 +696({{.RET}}):u64 +704({{.RET}}):u64 +712({{.RET}}):u64 +720({{.RET}}):u64 +728({{.RET}}):u64 +736({{.RET}}):u64 +744({{.RET}}):u64 +752({{.RET}}):u64 +760({{.RET}}):u64 +768({{.RET}}):u64 +776({{.RET}}):u64 +784({{.RET}}):u64 +792({{.RET}}):u64 +800({{.RET}}):u64 +808({{.RET}}):u64 +816({{.RET}}):u64 +824({{.RET}}):u64 +832({{.RET}}):u64 +840({{.RET}}):u64 +848({{.RET}}):u64 +856({{.RET}}):u64 +864({{.RET}}):u64 +872({{.RET}}):u64 +880({{.RET}}):u64 +888({{.RET}}):u64 +896({{.RET}}):u64 +904({{.RET}}):u64 +912({{.RET}}):u64 +920({{.RET}}):u64 +928({{.RET}}):u64 +936({{.RET}}):u64 +944({{.RET}}):u64 +952({{.RET}}):u64 +960({{.RET}}):u64 +968({{.RET}}):u64 +976({{.RET}}):u64 +984({{.RET}}):u64 +992({{.RET}}):u64 +1000({{.RET}}):u64 +1008({{.RET}}):u64 +1016({{.RET}}):u64': failed installing probe 'r:auditbeat/guess_recv_datagram __skb_recv_udp +0(%ax):u64 +8(%ax):u64 +16(%ax):u64 +24(%ax):u64 +32(%ax):u64 +40(%ax):u64 +48(%ax):u64 +56(%ax):u64 +64(%ax):u64 +72(%ax):u64 +80(%ax):u64 +88(%ax):u64 +96(%ax):u64 +104(%ax):u64 +112(%ax):u64 +120(%ax):u64 +128(%ax):u64 +136(%ax):u64 +144(%ax):u64 +152(%ax):u64 +160(%ax):u64 +168(%ax):u64 +176(%ax):u64 +184(%ax):u64 +192(%ax):u64 +200(%ax):u64 +208(%ax):u64 +216(%ax):u64 +224(%ax):u64 +232(%ax):u64 +240(%ax):u64 +248(%ax):u64 +256(%ax):u64 +264(%ax):u64 +272(%ax):u64 +280(%ax):u64 +288(%ax):u64 +296(%ax):u64 +304(%ax):u64 +312(%ax):u64 +320(%ax):u64 +328(%ax):u64 +336(%ax):u64 +344(%ax):u64 +352(%ax):u64 +360(%ax):u64 +368(%ax):u64 +376(%ax):u64 +384(%ax):u64 +392(%ax):u64 +400(%ax):u64 +408(%ax):u64 +416(%ax):u64 +424(%ax):u64 +432(%ax):u64 +440(%ax):u64 +448(%ax):u64 +456(%ax):u64 +464(%ax):u64 +472(%ax):u64 +480(%ax):u64 +488(%ax):u64 +496(%ax):u64 +504(%ax):u64 +512(%ax):u64 +520(%ax):u64 +528(%ax):u64 +536(%ax):u64 +544(%ax):u64 +552(%ax):u64 +560(%ax):u64 +568(%ax):u64 +576(%ax):u64 +584(%ax):u64 +592(%ax):u64 +600(%ax):u64 +608(%ax):u64 +616(%ax):u64 +624(%ax):u64 +632(%ax):u64 +640(%ax):u64 +648(%ax):u64 +656(%ax):u64 +664(%ax):u64 +672(%ax):u64 +680(%ax):u64 +688(%ax):u64 +696(%ax):u64 +704(%ax):u64 +712(%ax):u64 +720(%ax):u64 +728(%ax):u64 +736(%ax):u64 +744(%ax):u64 +752(%ax):u64 +760(%ax):u64 +768(%ax):u64 +776(%ax):u64 +784(%ax):u64 +792(%ax):u64 +800(%ax):u64 +808(%ax):u64 +816(%ax):u64 +824(%ax):u64 +832(%ax):u64 +840(%ax):u64 +848(%ax):u64 +856(%ax):u64 +864(%ax):u64 +872(%ax):u64 +880(%ax):u64 +888(%ax):u64 +896(%ax):u64 +904(%ax):u64 +912(%ax):u64 +920(%ax):u64 +928(%ax):u64 +936(%ax):u64 +944(%ax):u64 +952(%ax):u64 +960(%ax):u64 +968(%ax):u64 +976(%ax):u64 +984(%ax):u64 +992(%ax):u64 +1000(%ax):u64 +1008(%ax):u64 +1016(%ax):u64': write /sys/kernel/tracing/kprobe_events: file exists
Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_sk_buff_proto failed: failed to add kprobe 'r:guess_recv_datagram {{.RECV_UDP_DATAGRAM}} +0({{.RET}}):u64 +8({{.RET}}):u64 +16({{.RET}}):u64 +24({{.RET}}):u64 +32({{.RET}}):u64 +40({{.RET}}):u64 +48({{.RET}}):u64 +56({{.RET}}):u64 +64({{.RET}}):u64 +72({{.RET}}):u64 +80({{.RET}}):u64 +88({{.RET}}):u64 +96({{.RET}}):u64 +104({{.RET}}):u64 +112({{.RET}}):u64 +120({{.RET}}):u64 +128({{.RET}}):u64 +136({{.RET}}):u64 +144({{.RET}}):u64 +152({{.RET}}):u64 +160({{.RET}}):u64 +168({{.RET}}):u64 +176({{.RET}}):u64 +184({{.RET}}):u64 +192({{.RET}}):u64 +200({{.RET}}):u64 +208({{.RET}}):u64 +216({{.RET}}):u64 +224({{.RET}}):u64 +232({{.RET}}):u64 +240({{.RET}}):u64 +248({{.RET}}):u64 +256({{.RET}}):u64 +264({{.RET}}):u64 +272({{.RET}}):u64 +280({{.RET}}):u64 +288({{.RET}}):u64 +296({{.RET}}):u64 +304({{.RET}}):u64 +312({{.RET}}):u64 +320({{.RET}}):u64 +328({{.RET}}):u64 +336({{.RET}}):u64 +344({{.RET}}):u64 +352({{.RET}}):u64 +360({{.RET}}):u64 +368({{.RET}}):u64 +376({{.RET}}):u64 +384({{.RET}}):u64 +392({{.RET}}):u64 +400({{.RET}}):u64 +408({{.RET}}):u64 +416({{.RET}}):u64 +424({{.RET}}):u64 +432({{.RET}}):u64 +440({{.RET}}):u64 +448({{.RET}}):u64 +456({{.RET}}):u64 +464({{.RET}}):u64 +472({{.RET}}):u64 +480({{.RET}}):u64 +488({{.RET}}):u64 +496({{.RET}}):u64 +504({{.RET}}):u64 +512({{.RET}}):u64 +520({{.RET}}):u64 +528({{.RET}}):u64 +536({{.RET}}):u64 +544({{.RET}}):u64 +552({{.RET}}):u64 +560({{.RET}}):u64 +568({{.RET}}):u64 +576({{.RET}}):u64 +584({{.RET}}):u64 +592({{.RET}}):u64 +600({{.RET}}):u64 +608({{.RET}}):u64 +616({{.RET}}):u64 +624({{.RET}}):u64 +632({{.RET}}):u64 +640({{.RET}}):u64 +648({{.RET}}):u64 +656({{.RET}}):u64 +664({{.RET}}):u64 +672({{.RET}}):u64 +680({{.RET}}):u64 +688({{.RET}}):u64 +696({{.RET}}):u64 +704({{.RET}}):u64 +712({{.RET}}):u64 +720({{.RET}}):u64 +728({{.RET}}):u64 +736({{.RET}}):u64 +744({{.RET}}):u64 +752({{.RET}}):u64 +760({{.RET}}):u64 +768({{.RET}}):u64 +776({{.RET}}):u64 +784({{.RET}}):u64 +792({{.RET}}):u64 +800({{.RET}}):u64 +808({{.RET}}):u64 +816({{.RET}}):u64 +824({{.RET}}):u64 +832({{.RET}}):u64 +840({{.RET}}):u64 +848({{.RET}}):u64 +856({{.RET}}):u64 +864({{.RET}}):u64 +872({{.RET}}):u64 +880({{.RET}}):u64 +888({{.RET}}):u64 +896({{.RET}}):u64 +904({{.RET}}):u64 +912({{.RET}}):u64 +920({{.RET}}):u64 +928({{.RET}}):u64 +936({{.RET}}):u64 +944({{.RET}}):u64 +952({{.RET}}):u64 +960({{.RET}}):u64 +968({{.RET}}):u64 +976({{.RET}}):u64 +984({{.RET}}):u64 +992({{.RET}}):u64 +1000({{.RET}}):u64 +1008({{.RET}}):u64 +1016({{.RET}}):u64': failed installing probe 'r:auditbeat/guess_recv_datagram __skb_recv_udp +0(%ax):u64 +8(%ax):u64 +16(%ax):u64 +24(%ax):u64 +32(%ax):u64 +40(%ax):u64 +48(%ax):u64 +56(%ax):u64 +64(%ax):u64 +72(%ax):u64 +80(%ax):u64 +88(%ax):u64 +96(%ax):u64 +104(%ax):u64 +112(%ax):u64 +120(%ax):u64 +128(%ax):u64 +136(%ax):u64 +144(%ax):u64 +152(%ax):u64 +160(%ax):u64 +168(%ax):u64 +176(%ax):u64 +184(%ax):u64 +192(%ax):u64 +200(%ax):u64 +208(%ax):u64 +216(%ax):u64 +224(%ax):u64 +232(%ax):u64 +240(%ax):u64 +248(%ax):u64 +256(%ax):u64 +264(%ax):u64 +272(%ax):u64 +280(%ax):u64 +288(%ax):u64 +296(%ax):u64 +304(%ax):u64 +312(%ax):u64 +320(%ax):u64 +328(%ax):u64 +336(%ax):u64 +344(%ax):u64 +352(%ax):u64 +360(%ax):u64 +368(%ax):u64 +376(%ax):u64 +384(%ax):u64 +392(%ax):u64 +400(%ax):u64 +408(%ax):u64 +416(%ax):u64 +424(%ax):u64 +432(%ax):u64 +440(%ax):u64 +448(%ax):u64 +456(%ax):u64 +464(%ax):u64 +472(%ax):u64 +480(%ax):u64 +488(%ax):u64 +496(%ax):u64 +504(%ax):u64 +512(%ax):u64 +520(%ax):u64 +528(%ax):u64 +536(%ax):u64 +544(%ax):u64 +552(%ax):u64 +560(%ax):u64 +568(%ax):u64 +576(%ax):u64 +584(%ax):u64 +592(%ax):u64 +600(%ax):u64 +608(%ax):u64 +616(%ax):u64 +624(%ax):u64 +632(%ax):u64 +640(%ax):u64 +648(%ax):u64 +656(%ax):u64 +664(%ax):u64 +672(%ax):u64 +680(%ax):u64 +688(%ax):u64 +696(%ax):u64 +704(%ax):u64 +712(%ax):u64 +720(%ax):u64 +728(%ax):u64 +736(%ax):u64 +744(%ax):u64 +752(%ax):u64 +760(%ax):u64 +768(%ax):u64 +776(%ax):u64 +784(%ax):u64 +792(%ax):u64 +800(%ax):u64 +808(%ax):u64 +816(%ax):u64 +824(%ax):u64 +832(%ax):u64 +840(%ax):u64 +848(%ax):u64 +856(%ax):u64 +864(%ax):u64 +872(%ax):u64 +880(%ax):u64 +888(%ax):u64 +896(%ax):u64 +904(%ax):u64 +912(%ax):u64 +920(%ax):u64 +928(%ax):u64 +936(%ax):u64 +944(%ax):u64 +952(%ax):u64 +960(%ax):u64 +968(%ax):u64 +976(%ax):u64 +984(%ax):u64 +992(%ax):u64 +1000(%ax):u64 +1008(%ax):u64 +1016(%ax):u64': write /sys/kernel/tracing/kprobe_events: file exists 

However under 7.10.0 it didn't crash: auditbeat-debug-7.10.0.log

And that sounds like this issue https://github.com/elastic/beats/issues/20851 or this one: https://github.com/elastic/beats/issues/16046

111andre111 commented 3 years ago

@legoguy1000 @Aqualie @xennn Can you please test auditbeat 7.10.0?

Aqualie commented 3 years ago

Finished upgrading both kernel and auditbeat still having the same error:

» uname -a Linux xxxxxxx 5.9.10-arch1-1 #1 SMP PREEMPT Sun, 22 Nov 2020 14:16:59 +0000 x86_64 GNU/Linux

Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.415-0500        INFO        instance/beat.go:645        Home path: [/opt/elastic/auditbeat] Config path: [/opt/elastic/auditbeat/conf] Data path: [/opt/elastic/auditbeat/data] Logs path: [/opt/elastic/auditbeat/logs]
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.415-0500        INFO        instance/beat.go:653        Beat ID: xxxxxxxxxxxxxxxx
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.448-0500        INFO        [seccomp]        seccomp/seccomp.go:124        Syscall filter successfully installed
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.448-0500        INFO        [beat]        instance/beat.go:981        Beat info        {"system_info": {"beat": {"path": {"config": "/opt/elastic/auditbeat/conf", "data": "/opt/elastic/auditbeat/data", "home": "/opt/elastic/auditbeat", "logs": "/opt/elastic/auditbeat/logs"}, "type": "auditbeat", "uuid": "xxxxxxxxxxx"}}}
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.448-0500        INFO        [beat]        instance/beat.go:990        Build info        {"system_info": {"build": {"commit": "1428d58cf2ed945441fb2ed03961cafa9e4ad3eb", "libbeat": "7.10.0", "time": "2020-11-09T19:51:05.000Z", "version": "7.10.0"}}}
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.448-0500        INFO        [beat]        instance/beat.go:993        Go runtime info        {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.14.7"}}}
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.450-0500        INFO        [beat]        instance/beat.go:997        Host info        {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-11-27T10:18:20-05:00","containerized":false,"name":"xxxxx","ip":["xxxxxxxxxxxx"],"kernel_version":"5.9.10-arch1-1","mac":["xxxxxxxx"],"os":{"family":"","platform":"arch","name":"Arch Linux","version":"","major":0,"minor":0,"patch":0,"build":"rolling"},"timezone":"EST","timezone_offset_sec":-18000,"id":"xxxxxx"}}}
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.450-0500        INFO        [beat]        instance/beat.go:1026        Process info        {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/", "exe": "/opt/elastic/auditbeat/auditbeat", "name": "auditbeat", "pid": xxxxx, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-11-27T11:22:58.790-0500"}}}
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.451-0500        INFO        instance/beat.go:299        Setup Beat: auditbeat; Version: 7.10.0
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.451-0500        INFO        [publisher]        pipeline/module.go:113        Beat name: xxxxx
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.454-0500        INFO        [auditd]        auditd/audit_linux.go:106        auditd module is running as euid=0 on kernel=5.9.10-arch1-1
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.454-0500        INFO        [auditd]        auditd/audit_linux.go:133        socket_type=unicast will be used.
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.455-0500        WARN        [cfgwarn]        host/host.go:184        BETA: The system/host dataset is beta
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.458-0500        WARN        [cfgwarn]        login/login.go:95        BETA: The system/login dataset is beta
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.460-0500        WARN        [cfgwarn]        user/user.go:232        BETA: The system/user dataset is beta
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.464-0500        WARN        [cfgwarn]        process/process.go:146        BETA: The system/process dataset is beta
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.470-0500        WARN        [cfgwarn]        socket/socket_linux.go:91        BETA: The system/socket dataset is beta.
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.496-0500        INFO        [socket]        socket/socket_linux.go:231        Setting up system/socket for kernel 5.9.10-arch1-1
Nov 27 11:22:59 xxxxx auditbeat[xxx]: 2020-11-27T11:22:59.784-0500        INFO        [socket]        guess/guess.go:258        Running 17 guesses ...
Nov 27 11:23:16 xxxxx auditbeat[xxx]: 2020-11-27T11:23:16.698-0500        INFO        instance/beat.go:424        auditbeat stopped.
Nov 27 11:23:16 xxxxx auditbeat[xxx]: 2020-11-27T11:23:16.699-0500        ERROR        instance/beat.go:956        Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_ip_local_out failed: timeout while waiting for event
Nov 27 11:23:16 xxxxx auditbeat[xxx]: Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_ip_local_out failed: timeout while waiting for event
Nov 27 11:23:16 xxxxx systemd[1]: auditbeat.service: Main process exited, code=exited, status=1/FAILURE
Nov 27 11:23:16 xxxxx systemd[1]: auditbeat.service: Failed with result 'exit-code'.
Nov 27 11:23:16 xxxxx systemd[1]: auditbeat.service: Scheduled restart job, restart counter is at 201.

sysctl if it makes a difference:

net.ipv4.ip_local_port_range = 2048 65535
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
vm.max_map_count = 262144
net.core.rmem_max=33554432
net.core.rmem_default=262144
net.core.somaxconn=65535
net.core.netdev_max_backlog=65536
net.core.optmem_max=25165824
net.ipv4.udp_rmem_min=131072
net.ipv4.udp_mem=2097152 4194304 8388608
Aqualie commented 3 years ago

Also to add this machine has multiple vlan's with a bridged interface (same network configuration I've had since using auditbeat since version 6)

111andre111 commented 3 years ago

@Aqualie Would you mind to execute the 2 commands for the oneliner and post the debug log. Please obfuscate first what you want to hide from the log like the machine name:

touch empty.yml
./auditbeat -c empty.yml -E auditbeat.modules.0.module=system -E auditbeat.modules.0.datasets.0=socket -E output.console.pretty=true -e -d '*' > 2>&1 >auditbeat-debug.log

And last but not least it might be helpful to understand how this network config looks like: ip -d addr show

Aqualie commented 3 years ago

Had to modify the output abit as I'm using zsh and directory ownership is different: sudo -E ./auditbeat -c empty.yml -E auditbeat.modules.0.module=system -E auditbeat.modules.0.datasets.0=socket -E output.console.pretty=true -e -d '*' &>auditbeat-debug.log

2020-11-27T17:20:07.743-0500    INFO    instance/beat.go:645    Home path: [/opt/elastic/auditbeat] Config path: [/opt/elastic/auditbeat] Data path: [/opt/elastic/auditbeat/data] Logs path: [/opt/elastic/auditbeat/logs]
2020-11-27T17:20:07.743-0500    DEBUG   [beat]  instance/beat.go:697    Beat metadata path: /opt/elastic/auditbeat/data/meta.json
2020-11-27T17:20:07.743-0500    INFO    instance/beat.go:653    Beat ID: xxxxxxxxxxxxxxx
2020-11-27T17:20:07.743-0500    DEBUG   [seccomp]   seccomp/seccomp.go:117  Loading syscall filter  {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev","umask","mremap","perf_event_open","eventfd2","mount","umount2"],"action":"allow"}]}}}
2020-11-27T17:20:07.744-0500    INFO    [seccomp]   seccomp/seccomp.go:124  Syscall filter successfully installed
2020-11-27T17:20:07.744-0500    INFO    [beat]  instance/beat.go:981    Beat info   {"system_info": {"beat": {"path": {"config": "/opt/elastic/auditbeat", "data": "/opt/elastic/auditbeat/data", "home": "/opt/elastic/auditbeat", "logs": "/opt/elastic/auditbeat/logs"}, "type": "auditbeat", "uuid": "xxxxxxxxxxxxxxx"}}}
2020-11-27T17:20:07.744-0500    INFO    [beat]  instance/beat.go:990    Build info  {"system_info": {"build": {"commit": "1428d58cf2ed945441fb2ed03961cafa9e4ad3eb", "libbeat": "7.10.0", "time": "2020-11-09T19:51:05.000Z", "version": "7.10.0"}}}
2020-11-27T17:20:07.744-0500    INFO    [beat]  instance/beat.go:993    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.14.7"}}}
2020-11-27T17:20:07.745-0500    INFO    [beat]  instance/beat.go:997    Host info   {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-11-27T12:00:02-05:00","containerized":false,"name":"xxxxxxxxxxxxxxx","ip":["xxxxxxxxxxxxxxx"],"kernel_version":"5.9.10-arch1-1","mac":["xxxxxxxxxxxxxxx"],"os":{"family":"","platform":"arch","name":"Arch Linux","version":"","major":0,"minor":0,"patch":0,"build":"rolling"},"timezone":"EST","timezone_offset_sec":-18000,"id":"xxxxxxxxxxxxx}}}
2020-11-27T17:20:07.745-0500    INFO    [beat]  instance/beat.go:1026   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/opt/elastic/auditbeat", "exe": "/opt/elastic/auditbeat/auditbeat", "name": "auditbeat", "pid": xxxxx, "ppid": xxxx, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-11-27T17:20:07.630-0500"}}}
2020-11-27T17:20:07.745-0500    INFO    instance/beat.go:299    Setup Beat: auditbeat; Version: 7.10.0
2020-11-27T17:20:07.745-0500    DEBUG   [beat]  instance/beat.go:325    Initializing output plugins
2020-11-27T17:20:07.745-0500    DEBUG   [publisher] pipeline/consumer.go:148    start pipeline event consumer
2020-11-27T17:20:07.745-0500    INFO    [publisher] pipeline/module.go:113  Beat name: xxxxxx
2020-11-27T17:20:07.745-0500    DEBUG   [modules]   beater/metricbeat.go:151    Available modules and metricsets: Register [ModuleFactory:[system], MetricSetFactory:[auditd/auditd, file_integrity/file, system/host, system/login, system/package, system/process, system/socket, system/user]]
2020-11-27T17:20:07.748-0500    WARN    [cfgwarn]   socket/socket_linux.go:91   BETA: The system/socket dataset is beta.
2020-11-27T17:20:07.775-0500    INFO    [socket]    socket/socket_linux.go:231  Setting up system/socket for kernel 5.9.10-arch1-1
2020-11-27T17:20:07.776-0500    DEBUG   [socket]    socket/socket_linux.go:279  IPv6 supported: false
2020-11-27T17:20:07.776-0500    DEBUG   [socket]    socket/socket_linux.go:286  IPv6 enabled: false
2020-11-27T17:20:08.196-0500    DEBUG   [socket]    socket/socket_linux.go:347  Selected kernel function __x64_sys_execve for SYS_EXECVE
2020-11-27T17:20:08.196-0500    DEBUG   [socket]    socket/socket_linux.go:347  Selected kernel function __x64_sys_gettimeofday for SYS_GETTIMEOFDAY
2020-11-27T17:20:08.196-0500    DEBUG   [socket]    socket/socket_linux.go:347  Selected kernel function __x64_sys_newuname for SYS_UNAME
2020-11-27T17:20:08.196-0500    DEBUG   [socket]    socket/socket_linux.go:347  Selected kernel function ip_local_out for IP_LOCAL_OUT
2020-11-27T17:20:08.196-0500    DEBUG   [socket]    socket/socket_linux.go:347  Selected kernel function __skb_recv_udp for RECV_UDP_DATAGRAM
2020-11-27T17:20:08.202-0500    INFO    [socket]    guess/guess.go:258  Running 17 guesses ...
2020-11-27T17:20:08.366-0500    DEBUG   [socket]    guess/guess.go:287  Guess guess_udp_sendmsg completed: {"UDP_SENDMSG_LEN":"%dx","UDP_SENDMSG_MSG":"%si","UDP_SENDMSG_SOCK":"%di"}
2020-11-27T17:20:08.513-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock run #1: {"INET_SOCK_LADDR":[4,84,768,920],"INET_SOCK_LPORT":[776,930],"INET_SOCK_RADDR":[0,68,924],"INET_SOCK_RPORT":[12,928]}
2020-11-27T17:20:08.646-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock run #2: {"INET_SOCK_LADDR":[4,84,768,920],"INET_SOCK_LPORT":[776,930],"INET_SOCK_RADDR":[0,68,924],"INET_SOCK_RPORT":[12,928]}
2020-11-27T17:20:08.749-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock run #3: {"INET_SOCK_LADDR":[4,84,768,920],"INET_SOCK_LPORT":[776,930],"INET_SOCK_RADDR":[0,68,924],"INET_SOCK_RPORT":[12,928]}
2020-11-27T17:20:08.879-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock run #4: {"INET_SOCK_LADDR":[4,84,768,920],"INET_SOCK_LPORT":[776,930],"INET_SOCK_RADDR":[0,68,924],"INET_SOCK_RPORT":[12,928]}
2020-11-27T17:20:08.880-0500    DEBUG   [socket]    guess/guess.go:287  Guess guess_inet_sock completed: {"INET_SOCK_LADDR":4,"INET_SOCK_LADDR_LIST":[4,84,768,920],"INET_SOCK_LPORT":776,"INET_SOCK_LPORT_LIST":[776,930],"INET_SOCK_RADDR":0,"INET_SOCK_RADDR_LIST":[0,68,924],"INET_SOCK_RPORT":12,"INET_SOCK_RPORT_LIST":[12,928]}
2020-11-27T17:20:09.079-0500    DEBUG   [socket]    guess/guess.go:287  Guess guess_struct_socket_sk completed: {"SOCKET_SOCK":24}
2020-11-27T17:20:09.166-0500    DEBUG   [socket]    guess/guess.go:287  Guess guess_syscall_args completed: {"SYS_P1":"+0x70(%di)","SYS_P2":"+0x68(%di)","SYS_P3":"+0x60(%di)","SYS_P4":"+0x38(%di)","SYS_P5":"+0x48(%di)","SYS_P6":"+0x40(%di)"}
2020-11-27T17:20:09.166-0500    DEBUG   [socket]    guess/guess.go:270  Guess guess_inet_sock_ipv6 skipped.
2020-11-27T17:20:09.281-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock_af run #1: {"INET_SOCK_AF":[16]}
2020-11-27T17:20:09.409-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock_af run #2: {"INET_SOCK_AF":[16]}
2020-11-27T17:20:09.526-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock_af run #3: {"INET_SOCK_AF":[16]}
2020-11-27T17:20:09.626-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock_af run #4: {"INET_SOCK_AF":[16]}
2020-11-27T17:20:09.722-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock_af run #5: {"INET_SOCK_AF":[16]}
2020-11-27T17:20:09.780-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock_af run #6: {"INET_SOCK_AF":[16]}
2020-11-27T17:20:09.872-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock_af run #7: {"INET_SOCK_AF":[16]}
2020-11-27T17:20:09.976-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock_af run #8: {"INET_SOCK_AF":[16]}
2020-11-27T17:20:10.072-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock_af run #9: {"INET_SOCK_AF":[16]}
2020-11-27T17:20:10.176-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_inet_sock_af run #10: {"INET_SOCK_AF":[16]}
2020-11-27T17:20:10.176-0500    DEBUG   [socket]    guess/guess.go:287  Guess guess_inet_sock_af completed: {"INET_SOCK_AF":16}
2020-11-27T17:20:10.282-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_sk_buff_proto run #1: {"SK_BUFF_PROTO":[156,176]}
2020-11-27T17:20:10.378-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_sk_buff_proto run #2: {"SK_BUFF_PROTO":[176,432,944]}
2020-11-27T17:20:10.526-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_sk_buff_proto run #3: {"SK_BUFF_PROTO":[176,432,688,944]}
2020-11-27T17:20:10.636-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_sk_buff_proto run #4: {"SK_BUFF_PROTO":[176]}
2020-11-27T17:20:10.746-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_sk_buff_proto run #5: {"SK_BUFF_PROTO":[176]}
2020-11-27T17:20:10.833-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_sk_buff_proto run #6: {"SK_BUFF_PROTO":[176,540,546,796,802]}
2020-11-27T17:20:10.926-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_sk_buff_proto run #7: {"SK_BUFF_PROTO":[176,432,688,944]}
2020-11-27T17:20:11.056-0500    DEBUG   [socket]    guess/guess.go:112   --- result of guess_sk_buff_proto run #8: {"SK_BUFF_PROTO":[176,432,688,944]}
2020-11-27T17:20:11.056-0500    DEBUG   [socket]    guess/guess.go:287  Guess guess_sk_buff_proto completed: {"SK_BUFF_PROTO":176}
2020-11-27T17:20:11.056-0500    DEBUG   [socket]    guess/guess.go:270  Guess guess_inet6_csk_xmit skipped.
2020-11-27T17:20:11.056-0500    DEBUG   [socket]    guess/guess.go:270  Guess guess_deref skipped.
2020-11-27T17:20:11.156-0500    DEBUG   [socket]    guess/guess.go:287  Guess tcp_sendmsg_guess completed: {"TCP_SENDMSG_LEN":"%dx"}
2020-11-27T17:20:11.243-0500    DEBUG   [socket]    guess/guess.go:287  Guess guess_tcp_sendmsg_sock completed: {"TCP_SENDMSG_SOCK":"%di"}
2020-11-27T17:20:11.326-0500    DEBUG   [socket]    guess/guess.go:287  Guess guess_struct_creds completed: {"STRUCT_CRED_EGID":24,"STRUCT_CRED_EUID":20,"STRUCT_CRED_GID":8,"STRUCT_CRED_UID":4}
2020-11-27T17:20:11.326-0500    DEBUG   [socket]    guess/guess.go:121   --- guess_sk_buff_data_ptr run #0
2020-11-27T17:20:11.505-0500    DEBUG   [socket]    guess/guess.go:121   --- guess_sk_buff_data_ptr run #1
2020-11-27T17:20:11.662-0500    DEBUG   [socket]    guess/guess.go:287  Guess guess_sk_buff_data_ptr completed: {"SK_BUFF_HAS_POINTERS":false,"SK_BUFF_HEAD":192,"SK_BUFF_MAC":182,"SK_BUFF_NETWORK":180,"SK_BUFF_TRANSPORT":178}
2020-11-27T17:20:11.752-0500    DEBUG   [socket]    guess/guess.go:287  Guess guess_sockaddr_in completed: {"SOCKADDR_IN_ADDR":4,"SOCKADDR_IN_AF":0,"SOCKADDR_IN_PORT":2}
2020-11-27T17:20:11.752-0500    DEBUG   [socket]    guess/guess.go:270  Guess guess_sockaddr_in6 skipped.
2020-11-27T17:20:26.929-0500    INFO    instance/beat.go:424    auditbeat stopped.
2020-11-27T17:20:26.941-0500    ERROR   instance/beat.go:956    Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_ip_local_out failed: timeout while waiting for event
Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_ip_local_out failed: timeout while waiting for event
111andre111 commented 3 years ago

@Aqualie What kernel package are you using under ArchLinux? pacman -Q | grep linux And I have seen you disabled IPv6. How did you do do that? And would you mind to show the network config? ip -d addr show

Aqualie commented 3 years ago

pacman -Q | grep linux

archlinux-keyring 20201028-1
linux 5.9.10.arch1-1
linux-api-headers 5.8-1
linux-firmware 20201113.2ea8667-1
linux-headers 5.9.10.arch1-1
linux-lts-headers 5.4.79-1
linux-mainline-docs 5.4rc1-1
linuxdoc-tools 0.9.82-1
util-linux 2.36.1-3
util-linux-libs 2.36.1-3

cat /etc/sysctl.d/99-sysctl.conf | grep ipv6

net.ipv6.conf.all.disable_ipv6 = 1

ip -d addr show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether xxxxxxxxxxxx brd xxxxxxxxxxxx promiscuity 1 minmtu 60 maxmtu 9000
    bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.a6:xxxxxxxxxxxx designated_root 8000.a6:xxxxxxxxxxxx hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 5 numrxqueues 5 gso_max_size 65536 gso_max_segs 65535
3: wlp1s20u1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2312 qdisc mq master br0 state UP group default qlen 1000
    link/ether xxxxxxxxxxxx brd xxxxxxxxxxxx promiscuity 1 minmtu 60 maxmtu 2312
    bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8002 port_no 0x2 designated_port 32770 designated_cost 0 designated_bridge 8000.a6:xxxxxxxxxxxx designated_root 8000.a6:xxxxxxxxxxxx hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 4 numrxqueues 4 gso_max_size 65536 gso_max_segs 65535
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xxxxxxxxxxxx brd xxxxxxxxxxxx promiscuity 0 minmtu 68 maxmtu 65535
    bridge forward_delay 1499 hello_time 199 max_age 1999 ageing_time 29999 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.a6:xxxxxxxxxxxx designated_root 8000.a6:xxxxxxxxxxxx root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer   66.50 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:xxxxxxxxxxxxmcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 99 mcast_membership_interval 25999 mcast_querier_interval 25499 mcast_query_interval 12499 mcast_query_response_interval 999 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    inet xxxxxxxxxxxx brd xxxxxxxxxxxx scope global br0
       valid_lft forever preferred_lft forever
5: enp1s0f0.98@enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xxxxxxxxxxxx brd xxxxxxxxxxxx promiscuity 0 minmtu 0 maxmtu 65535
    vlan protocol 802.1Q id 98 <REORDER_HDR> numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    inet xxxxxxxxxxxx brd xxxxxxxxxxxx scope global enp1s0f0.98
       valid_lft forever preferred_lft forever
6: enp1s0f0.80@enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xxxxxxxxxxxx brd xxxxxxxxxxxx promiscuity 0 minmtu 0 maxmtu 65535
    vlan protocol 802.1Q id 80 <REORDER_HDR> numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    inet xxxxxxxxxxxx brd xxxxxxxxxxxx scope global enp1s0f0.80
       valid_lft forever preferred_lft forever
7: br-6b6f6e752b74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether xxxxxxxxxxxx brd xxxxxxxxxxxx promiscuity 0 minmtu 68 maxmtu 65535
    bridge forward_delay 1499 hello_time 199 max_age 1999 ageing_time 29999 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.2:xxxxxxxxxxxx designated_root 8000.2:xxxxxxxxxxxx root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer  194.82 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:xxxxxxxxxxxx mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 99 mcast_membership_interval 25999 mcast_querier_interval 25499 mcast_query_interval 12499 mcast_query_response_interval 999 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    inet xxxxxxxxxxxx brd xxxxxxxxxxxx scope global br-6b6f6e752b74
       valid_lft forever preferred_lft forever
8: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether xxxxxxxxxxxx brd xxxxxxxxxxxx promiscuity 0 minmtu 68 maxmtu 65535
    bridge forward_delay 1499 hello_time 199 max_age 1999 ageing_time 29999 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.2:xxxxxxxxxxxx designated_root 8000.2:xxxxxxxxxxxx root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer  186.50 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:xxxxxxxxxxxx mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 99 mcast_membership_interval 25999 mcast_querier_interval 25499 mcast_query_interval 12499 mcast_query_response_interval 999 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
10: veth54d6232@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether xxxxxxxxxxxx brd xxxxxxxxxxxx link-netnsid 0 promiscuity 1 minmtu 68 maxmtu 65535
    veth
    bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.2:xxxxxxxxxxxx designated_root 8000.2:xxxxxxxxxxxx hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
12: veth441da46@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master xxxxxxxxxxxx state UP group default
    link/ether xxxxxxxxxxxx brd xxxxxxxxxxxx link-netnsid 1 promiscuity 1 minmtu 68 maxmtu 65535
    veth
    bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.2:xxxxxxxxxxxx designated_root 8000.2:xxxxxxxxxxxx hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
111andre111 commented 3 years ago

I was able to reproduce now in both IPv6 enabled and disabled, just in the case of IPv6 disabled the error kicks in faster than with IPv6 enabled. Interesting.

Aqualie commented 3 years ago

I had tested enabling IPv6 before and did notice any difference my guess is multiple interfaces that's the culript ?

adriansr commented 3 years ago

Hi @Aqualie,

I think we have a fix. Can you try with one of the packages in here? https://drive.google.com/drive/u/0/folders/1REyd_8Ov_tjuqsO4YsvfWHUotHPzkgiK

Newer fix below

Aqualie commented 3 years ago

Thanks @adriansr this works perfectly and I can see all the socket events in ES!

111andre111 commented 3 years ago

Nice to hear, that it's working for you now @Aqualie . What about you @legoguy1000 @xennn

adriansr commented 3 years ago

Here's an updated snapshot build with an additional fix:

https://drive.google.com/drive/u/0/folders/1HJVrTF2iG45vYqSdBDcSynZjiiGSm31w

It includes:

legoguy1000 commented 3 years ago

Nice to hear, that it's working for you now @Aqualie . What about you @legoguy1000 @xennn

I haven't done anything with it in a while. The project I was working on was finished and we just left it out.

xennn commented 3 years ago

I give you next week a update. I'm out of office for this week

xennn commented 3 years ago

Its working for me.

Aqualie commented 3 years ago

I've started to upgrade my servers as this issue appeared to be fixed however after upgrading both the kernel and auditbeat to 7.10.1 which has the fix I'm still receiving the error. This host has IPV6 enabled with IPV6 addresses assigned and is running the linux-hardened kernel instead. Below is the debug file as requested before:

[root@REPLACED auditbeat]# pacman -Q | grep -i linux
archlinux-keyring 20201028-1
linux 5.9.13.arch1-1
linux-api-headers 5.8-1
linux-firmware 20201120.bc9cd0b-1
linux-hardened 5.9.12.a-1
util-linux 2.36.1-4
util-linux-libs 2.36.1-4
[root@REPLACED auditbeat]# ./auditbeat -c empty.yml -E auditbeat.modules.0.module=system -E auditbeat.modules.0.datasets.0=socket -E output.console.pretty=true -e -d '*' &>auditbeat-debug.log
[root@REPLACED auditbeat]# cat auditbeat-debug.log
2020-12-09T18:47:40.982-0500    INFO    instance/beat.go:645    Home path: [/opt/elastic/auditbeat] Config path: [/opt/elastic/auditbeat] Data path: [/opt/elastic/auditbeat/data] Logs path: [/opt/elastic/auditbeat/logs]
2020-12-09T18:47:40.982-0500    DEBUG   [beat]  instance/beat.go:697    Beat metadata path: /opt/elastic/auditbeat/data/meta.json
2020-12-09T18:47:40.982-0500    INFO    instance/beat.go:653    Beat ID: REPLACED
2020-12-09T18:47:40.982-0500    DEBUG   [seccomp]       seccomp/seccomp.go:117  Loading syscall filter  {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev","umask","mremap","perf_event_open","eventfd2","mount","umount2"],"action":"allow"}]}}}
2020-12-09T18:47:40.983-0500    INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2020-12-09T18:47:40.983-0500    INFO    [beat]  instance/beat.go:981    Beat info       {"system_info": {"beat": {"path": {"config": "/opt/elastic/auditbeat", "data": "/opt/elastic/auditbeat/data", "home": "/opt/elastic/auditbeat", "logs": "/opt/elastic/auditbeat/logs"}, "type": "auditbeat", "uuid": "REPLACED"}}}
2020-12-09T18:47:40.983-0500    INFO    [beat]  instance/beat.go:990    Build info      {"system_info": {"build": {"commit": "1da173a9e716715a7a54bb3ff4db05b5c24fc8ce", "libbeat": "7.10.1", "time": "2020-12-04T23:20:49.000Z", "version": "7.10.1"}}}
2020-12-09T18:47:40.983-0500    INFO    [beat]  instance/beat.go:993    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.14.12"}}}
2020-12-09T18:47:40.985-0500    INFO    [beat]  instance/beat.go:997    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-12-09T18:26:29-05:00","containerized":false,"name":"REPLACED","ip":["REPLACED"],"kernel_version":"5.9.12.a-1-hardened","mac":["REPLACED"],"os":{"family":"","platform":"arch","name":"Arch Linux","version":"","major":0,"minor":0,"patch":0,"build":"rolling"},"timezone":"EST","timezone_offset_sec":-18000,"id":"REPLACED"}}}
2020-12-09T18:47:40.985-0500    INFO    [beat]  instance/beat.go:1026   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/opt/elastic/auditbeat", "exe": "/opt/elastic/auditbeat/auditbeat", "name": "auditbeat", "pid": 2175, "ppid": 2159, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-12-09T18:47:40.840-0500"}}}
2020-12-09T18:47:40.985-0500    INFO    instance/beat.go:299    Setup Beat: auditbeat; Version: 7.10.1
2020-12-09T18:47:40.985-0500    DEBUG   [beat]  instance/beat.go:325    Initializing output plugins
2020-12-09T18:47:40.985-0500    DEBUG   [publisher]     pipeline/consumer.go:148        start pipeline event consumer
2020-12-09T18:47:40.985-0500    INFO    [publisher]     pipeline/module.go:113  Beat name: REPLACED
2020-12-09T18:47:40.986-0500    DEBUG   [modules]       beater/metricbeat.go:151        Available modules and metricsets: Register [ModuleFactory:[system], MetricSetFactory:[auditd/auditd, file_integrity/file, system/host, system/login, system/package, system/process, system/socket, system/user]]
2020-12-09T18:47:40.986-0500    WARN    [cfgwarn]       socket/socket_linux.go:124      BETA: The system/socket dataset is beta.
2020-12-09T18:47:41.020-0500    INFO    [socket]        socket/socket_linux.go:259      Setting up system/socket for kernel 5.9.12.a-1-hardened
2020-12-09T18:47:41.023-0500    DEBUG   [socket]        socket/socket_linux.go:306      IPv6 supported: true
2020-12-09T18:47:41.023-0500    DEBUG   [socket]        socket/socket_linux.go:313      IPv6 enabled: true
2020-12-09T18:47:41.207-0500    DEBUG   [socket]        socket/socket_linux.go:374      Selected kernel function ip_local_out for IP_LOCAL_OUT
2020-12-09T18:47:41.207-0500    DEBUG   [socket]        socket/socket_linux.go:374      Selected kernel function __skb_recv_udp for RECV_UDP_DATAGRAM
2020-12-09T18:47:41.207-0500    DEBUG   [socket]        socket/socket_linux.go:374      Selected kernel function __x64_sys_execve for SYS_EXECVE
2020-12-09T18:47:41.207-0500    DEBUG   [socket]        socket/socket_linux.go:374      Selected kernel function __x64_sys_gettimeofday for SYS_GETTIMEOFDAY
2020-12-09T18:47:41.207-0500    DEBUG   [socket]        socket/socket_linux.go:374      Selected kernel function __x64_sys_newuname for SYS_UNAME
2020-12-09T18:47:41.211-0500    INFO    [socket]        guess/guess.go:258      Running 17 guesses ...
2020-12-09T18:47:41.419-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_struct_socket_sk completed: {"SOCKET_SOCK":24}
2020-12-09T18:47:41.419-0500    DEBUG   [socket]        guess/guess.go:270      Guess guess_deref skipped.
2020-12-09T18:47:41.546-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock run #1: {"INET_SOCK_LADDR":[4,84,768,920],"INET_SOCK_LPORT":[776,930],"INET_SOCK_RADDR":[0,68,924],"INET_SOCK_RPORT":[12,928]}
2020-12-09T18:47:41.672-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock run #2: {"INET_SOCK_LADDR":[4,84,768,920],"INET_SOCK_LPORT":[776,930],"INET_SOCK_RADDR":[0,68,924],"INET_SOCK_RPORT":[12,928]}
2020-12-09T18:47:41.806-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock run #3: {"INET_SOCK_LADDR":[4,84,768,920],"INET_SOCK_LPORT":[776,930],"INET_SOCK_RADDR":[0,68,924],"INET_SOCK_RPORT":[12,928]}
2020-12-09T18:47:41.913-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock run #4: {"INET_SOCK_LADDR":[4,84,768,920],"INET_SOCK_LPORT":[776,930],"INET_SOCK_RADDR":[0,68,924],"INET_SOCK_RPORT":[12,928]}
2020-12-09T18:47:41.913-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_inet_sock completed: {"INET_SOCK_LADDR":4,"INET_SOCK_LADDR_LIST":[4,84,768,920],"INET_SOCK_LPORT":776,"INET_SOCK_LPORT_LIST":[776,930],"INET_SOCK_RADDR":0,"INET_SOCK_RADDR_LIST":[0,68,924],"INET_SOCK_RPORT":12,"INET_SOCK_RPORT_LIST":[12,928]}
2020-12-09T18:47:42.099-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_inet_sock_ipv6 completed: {"INET_SOCK_V6_LADDR_A":"+72","INET_SOCK_V6_LADDR_B":"+80","INET_SOCK_V6_LIMIT":56,"INET_SOCK_V6_RADDR_A":"+56","INET_SOCK_V6_RADDR_B":"+64","INET_SOCK_V6_TERM":":u64"}
2020-12-09T18:47:42.159-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_struct_creds completed: {"STRUCT_CRED_EGID":40,"STRUCT_CRED_EUID":36,"STRUCT_CRED_GID":24,"STRUCT_CRED_UID":20}
2020-12-09T18:47:42.243-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_sockaddr_in completed: {"SOCKADDR_IN_ADDR":4,"SOCKADDR_IN_AF":0,"SOCKADDR_IN_PORT":2}
2020-12-09T18:47:42.326-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_sockaddr_in6 completed: {"SOCKADDR_IN6_ADDRA":8,"SOCKADDR_IN6_ADDRB":16,"SOCKADDR_IN6_AF":0,"SOCKADDR_IN6_PORT":2}
2020-12-09T18:47:42.433-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock_af run #1: {"INET_SOCK_AF":[16]}
2020-12-09T18:47:42.529-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock_af run #2: {"INET_SOCK_AF":[16]}
2020-12-09T18:47:42.602-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock_af run #3: {"INET_SOCK_AF":[16]}
2020-12-09T18:47:42.699-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock_af run #4: {"INET_SOCK_AF":[16]}
2020-12-09T18:47:42.789-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock_af run #5: {"INET_SOCK_AF":[16]}
2020-12-09T18:47:42.899-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock_af run #6: {"INET_SOCK_AF":[16]}
2020-12-09T18:47:42.999-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock_af run #7: {"INET_SOCK_AF":[16]}
2020-12-09T18:47:43.086-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock_af run #8: {"INET_SOCK_AF":[16]}
2020-12-09T18:47:43.175-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock_af run #9: {"INET_SOCK_AF":[16]}
2020-12-09T18:47:43.285-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_inet_sock_af run #10: {"INET_SOCK_AF":[16]}
2020-12-09T18:47:43.286-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_inet_sock_af completed: {"INET_SOCK_AF":16}
2020-12-09T18:47:43.393-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_sk_buff_proto run #1: {"SK_BUFF_PROTO":[176]}
2020-12-09T18:47:43.526-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_sk_buff_proto run #2: {"SK_BUFF_PROTO":[176]}
2020-12-09T18:47:43.636-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_sk_buff_proto run #3: {"SK_BUFF_PROTO":[176]}
2020-12-09T18:47:43.739-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_sk_buff_proto run #4: {"SK_BUFF_PROTO":[176]}
2020-12-09T18:47:43.865-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_sk_buff_proto run #5: {"SK_BUFF_PROTO":[176]}
2020-12-09T18:47:44.019-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_sk_buff_proto run #6: {"SK_BUFF_PROTO":[176]}
2020-12-09T18:47:44.138-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_sk_buff_proto run #7: {"SK_BUFF_PROTO":[176]}
2020-12-09T18:47:44.226-0500    DEBUG   [socket]        guess/guess.go:112       --- result of guess_sk_buff_proto run #8: {"SK_BUFF_PROTO":[176]}
2020-12-09T18:47:44.226-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_sk_buff_proto completed: {"SK_BUFF_PROTO":176}
2020-12-09T18:47:44.283-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_syscall_args completed: {"SYS_P1":"+0x70(%di)","SYS_P2":"+0x68(%di)","SYS_P3":"+0x60(%di)","SYS_P4":"+0x38(%di)","SYS_P5":"+0x48(%di)","SYS_P6":"+0x40(%di)"}
2020-12-09T18:47:44.475-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_inet6_csk_xmit completed: {"INET6_CSK_XMIT_SKBUFF":"%si","INET6_CSK_XMIT_SOCK":"%di"}
2020-12-09T18:47:44.556-0500    DEBUG   [socket]        guess/guess.go:287      Guess tcp_sendmsg_guess completed: {"TCP_SENDMSG_LEN":"%dx"}
2020-12-09T18:47:44.646-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_tcp_sendmsg_sock completed: {"TCP_SENDMSG_SOCK":"%di"}
2020-12-09T18:47:44.744-0500    DEBUG   [socket]        guess/guess.go:287      Guess guess_udp_sendmsg completed: {"UDP_SENDMSG_LEN":"%dx","UDP_SENDMSG_MSG":"%si","UDP_SENDMSG_SOCK":"%di"}
2020-12-09T18:47:59.926-0500    INFO    instance/beat.go:424    auditbeat stopped.
2020-12-09T18:47:59.928-0500    ERROR   instance/beat.go:956    Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_ip_local_out failed: timeout while waiting for event
Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_ip_local_out failed: timeout while waiting for event
111andre111 commented 3 years ago

@Aqualie I was able to reproduce that with hardened kernel Would you mind please to try out this particular release? https://drive.google.com/drive/folders/1HJVrTF2iG45vYqSdBDcSynZjiiGSm31w?usp=sharing This one holds this former fix plus another one that was fixed here: https://github.com/elastic/beats/pull/22827 Because with these 2 fixes in place, I didn't hit into issues with the hardened kernel.

Aqualie commented 3 years ago

This release fixed the problem it's working now, thank you.

111andre111 commented 3 years ago

That's nice to hear, that this fixed it for you.

adriansr commented 2 years ago

Issue fixed