search

systemd / systemd-netlogd

Forwards messages from the journal to other hosts over the network using syslog format RFC 5424 and RFC 3164
GNU General Public License v2.0
69 stars 25 forks source link

Assertion 'message' failed at ../src/netlog/netlog-network.c:104, function manager_push_to_network(). Aborting. #27

Closed OderDochNicht closed 4 years ago

OderDochNicht commented 4 years ago

It worked before and somehow stopped some time ago.

I tried rebuild and reinstall, disable-enable, etc. Before I start to dig deeper into the code I'd like to ask for a starting point.

uname -a Linux name 5.5.0-2-amd64 #1 SMP Debian 5.5.17-1 (2020-04-15) x86_64 GNU/Linux

System:    Host: monster Kernel: 5.5.0-2-amd64 x86_64 bits: 64 Desktop: Xfce 4.12.4 
           Distro: Debian GNU/Linux 10 (buster) 
Machine:   Type: Desktop System: Micro-Star product: MS-7A32 v: 1.0 serial: <root required> 
           Mobo: Micro-Star model: X370 GAMING PRO CARBON AC (MS-7A32) v: 1.0 serial: <root required> 
           UEFI: American Megatrends v: 2.IQ date: 11/29/2019 
CPU:       Topology: 8-Core model: AMD Ryzen 7 1700X bits: 64 type: MT MCP L2 cache: 4096 KiB 
           Speed: 3481 MHz min/max: N/A Core speeds (MHz): 1: 3481 2: 3498 3: 3499 4: 3493 5: 3494 
           6: 3496 7: 3499 8: 3498 9: 3499 10: 3499 11: 3471 12: 3493 13: 3498 14: 3498 15: 3510 
           16: 3497 
Network:   Device-1: Intel I211 Gigabit Network driver: igb 
           IF: eno1 state: up speed: 1000 Mbps duplex: full mac: xx
           IP v4: xx/24 type: noprefixroute scope: global 
           IP v6: xx/64 type: temporary dynamic scope: global 
           IP v6: xx/64 type: dynamic mngtmpaddr noprefixroute 
           scope: global 
           IP v6: xx/64 type: noprefixroute scope: link 
           Device-2: Intel Dual Band Wireless-AC 3168NGW [Stone Peak] driver: iwlwifi 
           IF: wlp35s0 state: down mac: xx 
           WAN IP: No WAN IP data found. Connected to the web? SSL issues? 
Info:      Processes: 376 Uptime: 1d 2h 30m Memory: 31.38 GiB used: 6.92 GiB (22.1%) Shell: bash 
           inxi: 3.0.32

```systemd-netlogd.service - Journal Syslog Unicast and Multicast Daemon
   Loaded: loaded (/lib/systemd/system/systemd-netlogd.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Fri 2020-05-01 17:19:42 CEST; 3s ago
     Docs: man:systemd-netlogd.conf(5)
  Process: 29930 ExecStart=/lib/systemd/systemd-netlogd (code=killed, signal=ABRT)
 Main PID: 29930 (code=killed, signal=ABRT)
   Status: "Idle."

Mai 01 17:19:40 name systemd[29930]: systemd-netlogd.service: Executing: /lib/systemd/systemd-netlogd
Mai 01 17:19:40 name systemd[1]: systemd-netlogd.service: Got notification message from PID 29930 (WATCHDOG=1)
Mai 01 17:19:40 name systemd[1]: systemd-netlogd.service: Got notification message from PID 29930 (READY=1, STATUS=Processing input...)
Mai 01 17:19:40 name systemd[1]: systemd-netlogd.service: Got notification message from PID 29930 (STATUS=Idle.)
Mai 01 17:19:42 name systemd-netlogd[29930]: Assertion 'message' failed at ../src/netlog/netlog-network.c:104, function manager_push_to_network(). Aborting.
Mai 01 17:19:42 name systemd[1]: systemd-netlogd.service: Child 29930 belongs to systemd-netlogd.service.
Mai 01 17:19:42 name systemd[1]: systemd-netlogd.service: Main process exited, code=killed, status=6/ABRT
Mai 01 17:19:42 name systemd[1]: systemd-netlogd.service: Failed with result 'signal'.
Mai 01 17:19:42 name systemd[1]: systemd-netlogd.service: Changed running -> failed
Mai 01 17:19:42 name systemd[1]: systemd-netlogd.service: Unit entered failed state.
ssahani commented 4 years ago

Plese test with https://github.com/systemd/systemd-netlogd/pull/28

OderDochNicht commented 4 years ago

Inserted #28 in ../src/netlog/netlog-network.c at line 109: compiles, installs and starts ok.

root:~# systemctl status systemd-netlogd.service 
● systemd-netlogd.service - Journal Syslog Unicast and Multicast Daemon
   Loaded: loaded (/lib/systemd/system/systemd-netlogd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2020-05-03 08:51:45 CEST; 2s ago
     Docs: man:systemd-netlogd.conf(5)
 Main PID: 12657 (systemd-netlogd)
   Status: "Idle."
    Tasks: 1 (limit: 4915)
   Memory: 9.9M
   CGroup: /system.slice/systemd-netlogd.service
           └─12657 /lib/systemd/systemd-netlogd

Mai 03 08:51:45 monster systemd[12657]: Successfully mounted /tmp/systemd-private-ef2dfff9c79a4b0fb622ef3216b8bbde-systemd-netlogd.servi
Mai 03 08:51:45 monster systemd[12657]: Applying namespace mount on /run/systemd/unit-root/var/tmp
Mai 03 08:51:45 monster systemd[12657]: Successfully mounted /var/tmp/systemd-private-ef2dfff9c79a4b0fb622ef3216b8bbde-systemd-netlogd.s
Mai 03 08:51:45 monster systemd[12657]: Operating on architecture: x86
Mai 03 08:51:45 monster systemd[12657]: Operating on architecture: x32
Mai 03 08:51:45 monster systemd[12657]: Operating on architecture: x86-64
Mai 03 08:51:45 monster systemd[12657]: systemd-netlogd.service: Executing: /lib/systemd/systemd-netlogd
Mai 03 08:51:45 monster systemd[1]: systemd-netlogd.service: Got notification message from PID 12657 (WATCHDOG=1)
Mai 03 08:51:45 monster systemd[1]: systemd-netlogd.service: Got notification message from PID 12657 (READY=1, STATUS=Processing input..
Mai 03 08:51:45 monster systemd[1]: systemd-netlogd.service: Got notification message from PID 12657 (STATUS=Idle.)

now the log gets filled with:

Mai 03 08:00:01 name systemd[1]: systemd-netlogd.service: Got notification message from PID 6280 (WATCHDOG=1)
Mai 03 08:05:43 name systemd[1]: systemd-netlogd.service: Got notification message from PID 6280 (WATCHDOG=1)
Mai 03 08:10:57 name systemd[1]: systemd-netlogd.service: Got notification message from PID 6280 (WATCHDOG=1)
Mai 03 08:15:57 name systemd[1]: systemd-netlogd.service: Got notification message from PID 6280 (WATCHDOG=1)
Mai 03 08:20:57 name systemd[1]: systemd-netlogd.service: Got notification message from PID 6280 (WATCHDOG=1)
Mai 03 08:26:43 name systemd[1]: systemd-netlogd.service: Got notification message from PID 6280 (WATCHDOG=1)
Mai 03 08:32:43 name systemd[1]: systemd-netlogd.service: Got notification message from PID 6280 (WATCHDOG=1)
Mai 03 08:37:50 name systemd[1]: systemd-netlogd.service: Got notification message from PID 6280 (WATCHDOG=1)
Mai 03 08:43:33 name systemd[1]: systemd-netlogd.service: Got notification message from PID 6280 (WATCHDOG=1)
Mai 03 08:49:33 name systemd[1]: systemd-netlogd.service: Got notification message from PID 6280 (WATCHDOG=1)

doesn't look like there are messages sent to the remote logger.

# ss | grep 192.168.233.19
tcp   ESTAB      0      0                              192.168.233.54:930                   192.168.233.19:nfs                     
tcp   ESTAB      0      0                              192.168.233.54:48704               192.168.233.19:6690                    
tcp   ESTAB      0      0                              192.168.233.54:49576               192.168.233.19:6690

ss should show something on port 5571:

root:~# cat /etc/systemd/systemd-netlogd.conf 
[Network]
#Address=239.0.0.1:6000
Address=192.168.233.19:5571
ssahani commented 4 years ago

I just tested with same conf. Captured wireshark

Frame 43: 180 bytes on wire (1440 bits), 180 bytes captured (1440 bits) on interface any, id 0
Linux cooked capture
Internet Protocol Version 4, Src: 172.16.85.131, Dst: 192.168.233.19
User Datagram Protocol, Src Port: 39700, Dst Port: 5571
    Source Port: 39700
    Destination Port: 5571
    Length: 144
    Checksum: 0xabf1 [unverified]
    [Checksum Status: Unverified]
    [Stream index: 1]
    [Timestamps]
Data (136 bytes)
    Data: 3c32393e3120323032302d30352d30335430393a33313a33…
    [Length: 136]
)ÒBhE;ì@@S¬UÀ¨éëì<13>1 2020-05-03T09:33:04.258731+02:00 Zeus sus 35363 - - test 1111111111111111 22222222222222222222222          333333333333333333
OderDochNicht commented 4 years ago

No luck. systemd-netlog shows as running. Wireshark shows traffic from and to the remote logging-service (1,2 mio pakets) on the Synology but as soon as I enter the filter "tcp.port==5571" there remains not even one line. systemctl restart systemd-netlogd doesn't change anything. journalctl -f on the localhost shows logging activities. The protocol-center on Synology Diskstation doesn't. But it worked some time ago. ufw disable doesn't change anything. ss | grep 5571 shows nothing.

Any other ideas?

ssahani commented 4 years ago

probably you need to generate a event like from terminal

$logger test 123

ssahani commented 4 years ago

OK I am going to close this as no response