Fluent-bit version that supports new Docker_Mode_Parser

[cmedley2]

For our current cradlepoint kubernetes environments we need to enable the fluent-bit Docker_Mode, and at the same time we need the functionality of multi-line support. The ability to enable both functionalities was added in fluent-bit 1.5.1, but our initial attempts to upgrade the sumo chart with this version of fluent-bit were not successful. With 1.5.1 fluent-bit with the configuration given below did not seem to work, in that our json logs would get concatenated together along with all the other lines: parsers: enabled: true regex:

• name: multi_line regex: (?^{"log":"\d{4}-\d{1,2}-\d{1,2}.\d{2}:\d{2}:\d{2}.*)

[INPUT] Name tail Path /var/log/containers/.log Docker_Mode On Docker_Mode_Parser multi_line Tag containers. Refresh_Interval 1 Rotate_Wait 60 Mem_Buf_Limit 5MB Skip_Long_Lines On DB /tail-db/tail-containers-state-sumo.db DB.Sync Normal

I have not been able to figure out a regex for the multiline parser that fixes the json issue, so I am not sure if it is a bug or if I have it miss-configured somehow. It would also be nice if the new Docker_Mode_Parser supported multiple multiline regex, like the regular multiline feater with Docker_Mode_ParserN.

[frankreno]

@cmedley2 thanks for this. We will have a look and get back to you.

[git-pchauhan]

@frankreno - thank you, would appreciate your guidance on this.

[sumo-drosiek]

@cmedley2 Could you provide few log lines which are being concatenated?

[cmedley2]

Here they are:

"Hello world and the dog!82 Hello world and the dog!82 {\"timestamp\": \"2020-07-27T20:46:51.116313\", \"module\": \"config\", \"name\": \"apps.core.stream_callbacks.config\", \"threadname\": null, \"message\": \"Router 1703693: clearing pending patch\", \"causality\": \"\", \"opid\": \"342e9672\", \"opid_chain\": \"342e9672\", \"dd.trace_id\": 1533466686850586945, \"dd.span_id\": 14783603438013284304, \"level\": \"INFO\"} Hello world and the dog!82 {\"log_type\": \"audit\", \"message\": \"{\\"actor_id81\\": null, \\"actor_type\\": \\"NoneType\\", \\"object_id\\": 1704729, \\"object_type\\": \\"Manager\\", \\"object_before\\": [{\\"model\\": \\"core.manager\\", \\"pk\\": 1704729, \\"fields\\": {\\"deleted\\": null, \\"account\\": 53751, \\"router\\": 1703693, \\"actual_config\\": [{\\"lan\\": {\\"0\\": {\\"ip_address\\": \\"192.168.20.1\\", \\"dhcpd\\": {\\"lease6_time\\": 3600, \\"dhcp6_mode\\": \\"slaacdhcp\\", \\"options\\": []}, \\"vrrp\\": {\\"ipverify\\": {\\"test_id\\": \\"\\"}, \\"auth_type\\": \\"none\\", \\"init_state\\": \\"master\\", \\"advert_int\\": 1, \\"priority\\": 100, \\"vrid\\": 10}, \\"stp\\": {\\"priority\\": 32768}, \\"wired_8021x\\": {\\"radius\\": {\\"auth_servers\\": {\\"0\\": {\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1812}}, \\"acct_servers\\": {\\"0\\": {\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1813}}}, \\"eap\\": {\\"reauth_period\\": 3600}}, \\"passthrough_cycle_time\\": 10, \\"ip6_prefixlen\\": 64}}, \\"wan\\": {\\"rules2\\": {\\"5\\": {\\"priority\\": 0, \\"trigger_name\\": \\"Ethernet-wan\\", \\"trigger_string\\": \\"type|is|ethernet%uid|is|wan\\", \\"id\\": \\"00000005-a81d-3590-93ca-8b1fcfeb8e14\\", \\"stabilityCheck\\": {\\"enabled\\": true}, \\"def_conn_state\\": \\"alwayson\\"}, \\"6\\": {\\"priority\\": 3.2500560105000003, \\"trigger_name\\": \\"WWAN-fa:1d:0f:6c:fd:50\\", \\"trigger_string\\": \\"type|is|wwan%uid|is|fa:1d:0f:6c:fd:50\\", \\"id\\": \\"00000006-a81d-3590-93ca-8b1fcfeb8e14\\", \\"def_conn_state\\": \\"standby\\"}}}, \\"ecm\\": {\\"server_host\\": \\"stream-qa4.cradlepointecm.com\\", \\"config_version\\": 3}, \\"system\\": {\\"pci_dss\\": false, \\"ui_activated\\": true, \\"show_cloud_setup\\": false, \\"logging\\": {\\"level\\": \\"debug\\"}, \\"users\\": {\\"0\\": {\\"password\\": \\"\\"}}, \\"admin\\": {\\"product_name\\": \\"IBR900-600M\\"}}, \\"wwan\\": {\\"radio\\": {\\"0\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": null, \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"tt-desk-wifi\\", \\"wpapsk\\": \\"$1$65a38624$kpWgcPoQaspaksP4s1K9Jg==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"tt-desk-wifi:2_4G\\"}}}, \\"1\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": \\"fa:1d:0f:6c:fd:50\\", \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"CGNM-FD58-5G\\", \\"wpapsk\\": \\"$1$2590edbb$10C10l2R5JmHxiqeHknNkA==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"fa:1d:0f:6c:fd:50\\"}}}}}, \\"stats\\": {\\"client_usage\\": {\\"enabled\\": true}}}, []], \\"synched\\": false, \\"_suspended\\": false, \\"pending_patch\\": [{\\"ecm\\": {\\"config_version\\": 1}}, []], \\"expect_diff\\": true, \\"config_version\\": 1, \\"lock\\": \\"unlocked\\", \\"native_config\\": [{\\"ecm\\": {\\"server_host\\": \\"stream-qa4.cradlepointecm.com\\"}, \\"lan\\": {\\"00000000-0d93-319d-8220-4a1fb0372b51\\": {\\"dhcpd\\": {\\"dhcp6_mode\\": \\"slaacdhcp\\", \\"lease6_time\\": 3600, \\"options\\": []}, \\"ip_address\\": \\"192.168.20.1\\", \\"stp\\": {\\"priority\\": 32768}, \\"vrrp\\": {\\"advert_int\\": 1, \\"auth_type\\": \\"none\\", \\"init_state\\": \\"master\\", \\"ipverify\\": {\\"test_id\\": \\"\\"}, \\"priority\\": 100, \\"vrid\\": 10}, \\"wired_8021x\\": {\\"eap\\": {\\"reauth_period\\": 3600}, \\"radius\\": {\\"auth_servers\\": {\\"0\\": {\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1812}}, \\"acct_servers\\": {\\"0\\": {\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1813}}}}, \\"ip6_prefixlen\\": 64, \\"passthrough_cycle_time\\": 10, \\"id\\": \\"00000000-0d93-319d-8220-4a1fb0372b51\\"}}, \\"stats\\": {\\"client_usage\\": {\\"enabled\\": true}}, \\"system\\": {\\"admin\\": {\\"product_name\\": \\"IBR900-600M\\"}, \\"logging\\": {\\"level\\": \\"debug\\"}, \\"pci_dss\\": false, \\"show_cloud_setup\\": false, \\"ui_activated\\": true, \\"users\\": {\\"0\\": {\\"password\\": \\"\\"}}}, \\"wan\\": {\\"rules2\\": {\\"00000005-a81d-3590-93ca-8b1fcfeb8e14\\": {\\"priority\\": 0, \\"trigger_name\\": \\"Ethernet-wan\\", \\"trigger_string\\": \\"type|is|ethernet%uid|is|wan\\", \\"id\\": \\"00000005-a81d-3590-93ca-8b1fcfeb8e14\\", \\"stabilityCheck\\": {\\"enabled\\": true}, \\"def_conn_state\\": \\"alwayson\\"}, \\"00000006-a81d-3590-93ca-8b1fcfeb8e14\\": {\\"priority\\": 3.2500560105000003, \\"trigger_name\\": \\"WWAN-fa:1d:0f:6c:fd:50\\", \\"trigger_string\\": \\"type|is|wwan%uid|is|fa:1d:0f:6c:fd:50\\", \\"id\\": \\"00000006-a81d-3590-93ca-8b1fcfeb8e14\\", \\"def_conn_state\\": \\"standby\\"}}}, \\"wwan\\": {\\"radio\\": {\\"0\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": null, \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"tt-desk-wifi\\", \\"wpapsk\\": \\"$1$65a38624$kpWgcPoQaspaksP4s1K9Jg==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"tt-desk-wifi:2_4G\\"}}}, \\"1\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": \\"fa:1d:0f:6c:fd:50\\", \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"CGNM-FD58-5G\\", \\"wpapsk\\": \\"$1$2590edbb$10C10l2R5JmHxiqeHknNkA==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"fa:1d:0f:6c:fd:50\\"}}}}}}, []], \\"native_runtime_defaults\\": [{\\"system\\": {\\"system_id\\": \\"IBR900-58f\\", \\"users\\": {\\"0\\": {\\"username\\": \\"admin\\", \\"password\\": \\"\\", \\"group\\": \\"admin\\"}}}, \\"wlan\\": {\\"radio\\": {\\"0\\": {\\"bss\\": {\\"0\\": {\\"ssid\\": \\"IBR900-58f\\", \\"wpapsk\\": \\"$1$d0e4c324$L0CcCRN+7a115//BvfBKQQ==\\", \\"radius0key\\": \\"$1$26cc935c$02le7S8kl9ZvL8yZ1ZOeAQ==\\"}, \\"1\\": {\\"ssid\\": \\"Public-58f\\", \\"wpapsk\\": \\"$1$d0e4c324$L0CcCRN+7a115//BvfBKQQ==\\", \\"radius0key\\": \\"$1$26cc935c$02le7S8kl9ZvL8yZ1ZOeAQ==\\"}}}, \\"1\\": {\\"bss\\": {\\"0\\": {\\"ssid\\": \\"IBR900-58f-5g\\", \\"wpapsk\\": \\"$1$d0e4c324$L0CcCRN+7a115//BvfBKQQ==\\", \\"radius0key\\": \\"$1$26cc935c$02le7S8kl9ZvL8yZ1ZOeAQ==\\"}, \\"1\\": {\\"ssid\\": \\"Public-58f-5g\\", \\"wpapsk\\": \\"$1$d0e4c324$L0CcCRN+7a115//BvfBKQQ==\\", \\"radius0key\\": \\"$1$26cc935c$02le7S8kl9ZvL8yZ1ZOeAQ==\\"}}}}}}, []], \\"target_config\\": [{\\"ecm\\": {\\"server_host\\": \\"stream-qa4.cradlepointecm.com\\"}, \\"lan\\": {\\"00000000-0d93-319d-8220-4a1fb0372b51\\": {\\"dhcpd\\": {\\"dhcp6_mode\\": \\"slaacdhcp\\", \\"lease6_time\\": 3600, \\"options\\": []}, \\"ip_address\\": \\"192.168.20.1\\", \\"stp\\": {\\"priority\\": 32768}, \\"vrrp\\": {\\"advert_int\\": 1, \\"auth_type\\": \\"none\\", \\"init_state\\": \\"master\\", \\"ipverify\\": {\\"test_id\\": \\"\\"}, \\"priority\\": 100, \\"vrid\\": 10}, \\"wired_8021x\\": {\\"eap\\": {\\"reauth_period\\": 3600}, \\"radius\\": {\\"auth_servers\\": {\\"0\\": {\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1812}}, \\"acct_servers\\": {\\"0\\": {\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1813}}}}, \\"ip6_prefixlen\\": 64, \\"passthrough_cycle_time\\": 10, \\"id\\": \\"00000000-0d93-319d-8220-4a1fb0372b51\\"}}, \\"stats\\": {\\"client_usage\\": {\\"enabled\\": true}}, \\"system\\": {\\"admin\\": {\\"product_name\\": \\"IBR900-600M\\"}, \\"logging\\": {\\"level\\": \\"debug\\"}, \\"pci_dss\\": false, \\"show_cloud_setup\\": false, \\"ui_activated\\": true, \\"users\\": {\\"0\\": {\\"password\\": \\"\\"}}}, \\"wan\\": {\\"rules2\\": {\\"00000005-a81d-3590-93ca-8b1fcfeb8e14\\": {\\"priority\\": 0, \\"trigger_name\\": \\"Ethernet-wan\\", \\"trigger_string\\": \\"type|is|ethernet%uid|is|wan\\", \\"id\\": \\"00000005-a81d-3590-93ca-8b1fcfeb8e14\\", \\"stabilityCheck\\": {\\"enabled\\": true}, \\"def_conn_state\\": \\"alwayson\\"}, \\"00000006-a81d-3590-93ca-8b1fcfeb8e14\\": {\\"priority\\": 3.2500560105000003, \\"trigger_name\\": \\"WWAN-fa:1d:0f:6c:fd:50\\", \\"trigger_string\\": \\"type|is|wwan%uid|is|fa:1d:0f:6c:fd:50\\", \\"id\\": \\"00000006-a81d-3590-93ca-8b1fcfeb8e14\\", \\"def_conn_state\\": \\"standby\\"}}}, \\"wwan\\": {\\"radio\\": {\\"0\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": null, \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"tt-desk-wifi\\", \\"wpapsk\\": \\"$1$65a38624$kpWgcPoQaspaksP4s1K9Jg==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"tt-desk-wifi:2_4G\\"}}}, \\"1\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": \\"fa:1d:0f:6c:fd:50\\", \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"CGNM-FD58-5G\\", \\"wpapsk\\": \\"$1$2590edbb$10C10l2R5JmHxiqeHknNkA==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"fa:1d:0f:6c:fd:50\\"}}}}}}, []], \\"native_config_firmware\\": 2389, \\"actual_config_firmware\\": 2389, \\"last_sync_error\\": null, \\"last_sync_error_at\\": null}}], \\"object_after\\": [{\\"model\\": \\"core.manager\\", \\"pk\\": 1704729, \\"fields\\": {\\"deleted\\": null, \\"account\\": 53751, \\"router\\": 1703693, \\"actual_config\\": [{\\"lan\\": {\\"0\\": {\\"ip_address\\": \\"192.168.20.1\\", \\"dhcpd\\": {\\"lease6_time\\": 3600, \\"dhcp6_mode\\": \\"slaacdhcp\\", \\"options\\": []}, \\"vrrp\\": {\\"ipverify\\": {\\"test_id\\": \\"\\"}, \\"auth_type\\": \\"none\\", \\"init_state\\": \\"master\\", \\"advert_int\\": 1, \\"priority\\": 100, \\"vrid\\": 10}, \\"stp\\": {\\"priority\\": 32768}, \\"wired_8021x\\": {\\"radius\\": {\\"auth_servers\\": [{\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1812}], \\"acct_servers\\": [{\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1813}]}, \\"eap\\": {\\"reauth_period\\": 3600}}, \\"passthrough_cycle_time\\": 10, \\"ip6_prefixlen\\": 64}}, \\"wan\\": {\\"rules2\\": {\\"5\\": {\\"priority\\": 0, \\"trigger_name\\": \\"Ethernet-wan\\", \\"trigger_string\\": \\"type|is|ethernet%uid|is|wan\\", \\"id\\": \\"00000005-a81d-3590-93ca-8b1fcfeb8e14\\", \\"stabilityCheck\\": {\\"enabled\\": true}, \\"def_conn_state\\": \\"alwayson\\"}, \\"6\\": {\\"priority\\": 3.2500560105000003, \\"trigger_name\\": \\"WWAN-fa:1d:0f:6c:fd:50\\", \\"trigger_string\\": \\"type|is|wwan%uid|is|fa:1d:0f:6c:fd:50\\", \\"id\\": \\"00000006-a81d-3590-93ca-8b1fcfeb8e14\\", \\"def_conn_state\\": \\"standby\\"}}}, \\"ecm\\": {\\"server_host\\": \\"stream-qa4.cradlepointecm.com\\", \\"config_version\\": 1}, \\"system\\": {\\"pci_dss\\": false, \\"ui_activated\\": true, \\"show_cloud_setup\\": false, \\"logging\\": {\\"level\\": \\"debug\\"}, \\"users\\": {\\"0\\": {\\"password\\": \\"\\"}}, \\"admin\\": {\\"product_name\\": \\"IBR900-600M\\"}}, \\"wwan\\": {\\"radio\\": {\\"0\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": null, \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"tt-desk-wifi\\", \\"wpapsk\\": \\"$1$65a38624$kpWgcPoQaspaksP4s1K9Jg==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"tt-desk-wifi:2_4G\\"}}}, \\"1\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": \\"fa:1d:0f:6c:fd:50\\", \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"CGNM-FD58-5G\\", \\"wpapsk\\": \\"$1$2590edbb$10C10l2R5JmHxiqeHknNkA==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"fa:1d:0f:6c:fd:50\\"}}}}}, \\"stats\\": {\\"client_usage\\": {\\"enabled\\": true}}}, []], \\"synched\\": true, \\"_suspended\\": false, \\"pending_patch\\": null, \\"expect_diff\\": false, \\"config_version\\": 1, \\"lock\\": \\"unlocked\\", \\"native_config\\": [{\\"ecm\\": {\\"server_host\\": \\"stream-qa4.cradlepointecm.com\\"}, \\"lan\\": {\\"00000000-0d93-319d-8220-4a1fb0372b51\\": {\\"dhcpd\\": {\\"dhcp6_mode\\": \\"slaacdhcp\\", \\"lease6_time\\": 3600, \\"options\\": []}, \\"ip_address\\": \\"192.168.20.1\\", \\"stp\\": {\\"priority\\": 32768}, \\"vrrp\\": {\\"advert_int\\": 1, \\"auth_type\\": \\"none\\", \\"init_state\\": \\"master\\", \\"ipverify\\": {\\"test_id\\": \\"\\"}, \\"priority\\": 100, \\"vrid\\": 10}, \\"wired_8021x\\": {\\"eap\\": {\\"reauth_period\\": 3600}, \\"radius\\": {\\"auth_servers\\": {\\"0\\": {\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1812}}, \\"acct_servers\\": {\\"0\\": {\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1813}}}}, \\"ip6_prefixlen\\": 64, \\"passthrough_cycle_time\\": 10, \\"id\\": \\"00000000-0d93-319d-8220-4a1fb0372b51\\"}}, \\"stats\\": {\\"client_usage\\": {\\"enabled\\": true}}, \\"system\\": {\\"admin\\": {\\"product_name\\": \\"IBR900-600M\\"}, \\"logging\\": {\\"level\\": \\"debug\\"}, \\"pci_dss\\": false, \\"show_cloud_setup\\": false, \\"ui_activated\\": true, \\"users\\": {\\"0\\": {\\"password\\": \\"\\"}}}, \\"wan\\": {\\"rules2\\": {\\"00000005-a81d-3590-93ca-8b1fcfeb8e14\\": {\\"priority\\": 0, \\"trigger_name\\": \\"Ethernet-wan\\", \\"trigger_string\\": \\"type|is|ethernet%uid|is|wan\\", \\"id\\": \\"00000005-a81d-3590-93ca-8b1fcfeb8e14\\", \\"stabilityCheck\\": {\\"enabled\\": true}, \\"def_conn_state\\": \\"alwayson\\"}, \\"00000006-a81d-3590-93ca-8b1fcfeb8e14\\": {\\"priority\\": 3.2500560105000003, \\"trigger_name\\": \\"WWAN-fa:1d:0f:6c:fd:50\\", \\"trigger_string\\": \\"type|is|wwan%uid|is|fa:1d:0f:6c:fd:50\\", \\"id\\": \\"00000006-a81d-3590-93ca-8b1fcfeb8e14\\", \\"def_conn_state\\": \\"standby\\"}}}, \\"wwan\\": {\\"radio\\": {\\"0\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": null, \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"tt-desk-wifi\\", \\"wpapsk\\": \\"$1$65a38624$kpWgcPoQaspaksP4s1K9Jg==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"tt-desk-wifi:2_4G\\"}}}, \\"1\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": \\"fa:1d:0f:6c:fd:50\\", \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"CGNM-FD58-5G\\", \\"wpapsk\\": \\"$1$2590edbb$10C10l2R5JmHxiqeHknNkA==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"fa:1d:0f:6c:fd:50\\"}}}}}}, []], \\"native_runtime_defaults\\": [{\\"system\\": {\\"system_id\\": \\"IBR900-58f\\", \\"users\\": {\\"0\\": {\\"username\\": \\"admin\\", \\"password\\": \\"\\", \\"group\\": \\"admin\\"}}}, \\"wlan\\": {\\"radio\\": {\\"0\\": {\\"bss\\": {\\"0\\": {\\"ssid\\": \\"IBR900-58f\\", \\"wpapsk\\": \\"$1$d0e4c324$L0CcCRN+7a115//BvfBKQQ==\\", \\"radius0key\\": \\"$1$26cc935c$02le7S8kl9ZvL8yZ1ZOeAQ==\\"}, \\"1\\": {\\"ssid\\": \\"Public-58f\\", \\"wpapsk\\": \\"$1$d0e4c324$L0CcCRN+7a115//BvfBKQQ==\\", \\"radius0key\\": \\"$1$26cc935c$02le7S8kl9ZvL8yZ1ZOeAQ==\\"}}}, \\"1\\": {\\"bss\\": {\\"0\\": {\\"ssid\\": \\"IBR900-58f-5g\\", \\"wpapsk\\": \\"$1$d0e4c324$L0CcCRN+7a115//BvfBKQQ==\\", \\"radius0key\\": \\"$1$26cc935c$02le7S8kl9ZvL8yZ1ZOeAQ==\\"}, \\"1\\": {\\"ssid\\": \\"Public-58f-5g\\", \\"wpapsk\\": \\"$1$d0e4c324$L0CcCRN+7a115//BvfBKQQ==\\", \\"radius0key\\": \\"$1$26cc935c$02le7S8kl9ZvL8yZ1ZOeAQ==\\"}}}}}}, []], \\"target_config\\": [{\\"ecm\\": {\\"server_host\\": \\"stream-qa4.cradlepointecm.com\\"}, \\"lan\\": {\\"00000000-0d93-319d-8220-4a1fb0372b51\\": {\\"dhcpd\\": {\\"dhcp6_mode\\": \\"slaacdhcp\\", \\"lease6_time\\": 3600, \\"options\\": []}, \\"ip_address\\": \\"192.168.20.1\\", \\"stp\\": {\\"priority\\": 32768}, \\"vrrp\\": {\\"advert_int\\": 1, \\"auth_type\\": \\"none\\", \\"init_state\\": \\"master\\", \\"ipverify\\": {\\"test_id\\": \\"\\"}, \\"priority\\": 100, \\"vrid\\": 10}, \\"wired_8021x\\": {\\"eap\\": {\\"reauth_period\\": 3600}, \\"radius\\": {\\"auth_servers\\": {\\"0\\": {\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1812}}, \\"acct_servers\\": {\\"0\\": {\\"ip_address\\": \\"127.0.0.1\\", \\"mac\\": \\"00:00:00:00:00:00\\", \\"port\\": 1813}}}}, \\"ip6_prefixlen\\": 64, \\"passthrough_cycle_time\\": 10, \\"id\\": \\"00000000-0d93-319d-8220-4a1fb0372b51\\"}}, \\"stats\\": {\\"client_usage\\": {\\"enabled\\": true}}, \\"system\\": {\\"admin\\": {\\"product_name\\": \\"IBR900-600M\\"}, \\"logging\\": {\\"level\\": \\"debug\\"}, \\"pci_dss\\": false, \\"show_cloud_setup\\": false, \\"ui_activated\\": true, \\"users\\": {\\"0\\": {\\"password\\": \\"\\"}}}, \\"wan\\": {\\"rules2\\": {\\"00000005-a81d-3590-93ca-8b1fcfeb8e14\\": {\\"priority\\": 0, \\"trigger_name\\": \\"Ethernet-wan\\", \\"trigger_string\\": \\"type|is|ethernet%uid|is|wan\\", \\"id\\": \\"00000005-a81d-3590-93ca-8b1fcfeb8e14\\", \\"stabilityCheck\\": {\\"enabled\\": true}, \\"def_conn_state\\": \\"alwayson\\"}, \\"00000006-a81d-3590-93ca-8b1fcfeb8e14\\": {\\"priority\\": 3.2500560105000003, \\"trigger_name\\": \\"WWAN-fa:1d:0f:6c:fd:50\\", \\"trigger_string\\": \\"type|is|wwan%uid|is|fa:1d:0f:6c:fd:50\\", \\"id\\": \\"00000006-a81d-3590-93ca-8b1fcfeb8e14\\", \\"def_conn_state\\": \\"standby\\"}}}, \\"wwan\\": {\\"radio\\": {\\"0\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": null, \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"tt-desk-wifi\\", \\"wpapsk\\": \\"$1$65a38624$kpWgcPoQaspaksP4s1K9Jg==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"tt-desk-wifi:2_4G\\"}}}, \\"1\\": {\\"mode\\": \\"wwan\\", \\"profiles\\": {\\"0\\": {\\"active_scan\\": false, \\"roaming_enabled\\": false, \\"min_link_rssi\\": -70, \\"min_scan_rssi\\": -80, \\"bssid\\": \\"fa:1d:0f:6c:fd:50\\", \\"authmode\\": \\"wpa2psk\\", \\"enabled\\": true, \\"wpacipher\\": \\"aes\\", \\"ssid\\": \\"CGNM-FD58-5G\\", \\"wpapsk\\": \\"$1$2590edbb$10C10l2R5JmHxiqeHknNkA==\\", \\"eapconf\\": {\\"eaptype\\": \\"peap\\", \\"phase1\\": \\"\\", \\"phase2\\": \\"auth=MSCHAPV2\\"}, \\"uid\\": \\"fa:1d:0f:6c:fd:50\\"}}}}}}, []], \\"native_config_firmware\\": 2389, \\"actual_config_firmware\\": 2389, \\"last_sync_error\\": null, \\"last_sync_error_at\\": null}}], \\"action_type\\": \\"UPDATE\\", \\"ip_address\\": null, \\"timestamp\\": \\"2020-07-27T20:46:51.149741\\"}\", \"created\": 1592412411.1501005, \"levelname\": \"INFO\", \"levelno\": 20, \"dd.trace_id\": 1533466686850586945, \"dd.span_id\": 14783603438013284304} {\"timestamp\": \"2020-07-27T20:46:51.161084\", \"module\": \"client_analytics\", \"name\": \"client_analytics\", \"threadname\": null, \"message\": \"Enabling application analytics on Router 1703693: IBR900-58f 00:30:44:37:45:8F\", \"causality\": \"\", \"opid\": \"342e9672\", \"opid_chain\": \"342e9672\", \"dd.trace_id\": 1533466686850586945, \"dd.span_id\": 14783603438013284304, \"level\": \"INFO\"}"

Thanks ahead of time for your help.

[sumo-drosiek]

@cmedley2 Is this from /var/log/containers/{filename}.log? regex doesn't seem to match any of the line 😕

[cmedley2]

No that is the sumo side output- ie everything concatenated together- here is what it looks like in the docker container log, different timestamps but the lines are the same. The line with a timestamp and not json comes out as expected. Tried several ways of trying to get the json lines to be recognized as the first line in a multiline but I was not successful:

testapp.output.txt

[sumo-drosiek]

I need part of the /var/log/containers/{filename}.log. It's hard to reproduce the issue with the output only. Are you able to share part of this file with me?

[sumo-drosiek]

Are you using docker-json logging driver?

[cmedley2]

so the contents of testapp.output.txt is the contents of the /var/log/containers/{filename}.log retrieved with this command: kubectl logs testsumo-deployment-675d4bdb9b-ldkh5 > testapp.output.txt does that not give you what you need? If not I will log onto the node and get docker log file.

I do not think we are using the docker-json logger- we are using the standard aws eks node ami.

[sumo-drosiek]

Docker_mode supports only docker-json logger (rel: https://docs.fluentbit.io/manual/v/1.0/input/tail#docker_mode).

If line is not a json, it is flushed without concatenation, otherwise the Docker_mode_parser is used to validate if the line is beginning of the log or part of it. Given regex doesn't fit to any of the lines, so they are treated as part of one long log.

the docker json format:

{"log":"This is example log\n","stream":"stdout","time":"2020-01-09T22:51:31.549390877Z"}

Docker_mode shouldn't be used for the files which aren't compatible with that format.

[sumo-drosiek]

I think you should check the file on the node to clarify that

[cmedley2]

K- I will get the docker log file from the node, btw it works fine with docker mode when I do not try to use multiline.

[sumo-drosiek]

@cmedley2 Does it do anything? Do you have any splitted logs?

[sumo-drosiek]

For me it looks like the logs are just read without any modification, but I can be wrong

[cmedley2]

When I turn on docker mode and multiline at the same time it does not split anything, the json is concatenated, as well as my non-json multiline logs. When I run without the multiline and just docker mode the json > 16k is correctly together, but my real multiline logs (not json) do not get concantenated.

[sumo-drosiek]

And what happens if you use only multiline functionality (without docker_mode)?

[cmedley2]

Don't remember if I tried that or not- certainly not since your change went in. Without your change there was not much point. Let me try that and get the docker log file for you.

[cmedley2]

So sorry it took so long, I got wrapped up in some other issues but here is the info I promised. I ran 3 tests this morning, the multi_line regex used in each test is the default provided in the sumo collection chart: regex: (?^{"log":"\d{4}-\d{1,2}-\d{1,2}.\d{2}:\d{2}:\d{2}.*)

Test 1: Docker_Mode On Docker_Mode_Parser multi_line Result: Json > 16k correctly concatenated into one log, multiline non-json logs correctly concatenated, all json logs (including the > 16k log), and non json single line logs incorrectly concatenated together.

Test 2: Multiline On Parser_Firstline multi_line Result: Json > 16k not concatenated into one log comes out in two logs, multiline non-json logs correctly concatenated, non-json single line logs correctly output as single logs, json < 16k correctly output as a single log.

Test 3: Docker_Mode On Multiline On Parser_Firstline multi_line Result: The fluent-bit container crashed because Docker_Mode and Multiline are not valid together. I expected this, but tested it anyway to make sure I had not missed something.

Also attaching the container log file from my test app. testsumo-deployment-675d4bdb9b-5j7g4_default_testsumo-f948f529c5247b671fb2c52f6a23d0900d7ca6df49ec36e6a7f89ff9c587e1ad.log

Hope that helps.

[sumo-drosiek]

Yes, that helps a lot. So I have all of the information. Regex for the multiline has to match the first line of the multiline log: regex: (?^{"log":"\d{4}-\d{1,2}-\d{1,2}.\d{2}:\d{2}:\d{2}.*) doesn't match any of the line from the file, so it won't work correctly. We need to figure out regex which matches only first line of log and is true for every log line. It is easy for timestamped logs (as example regex). Unfortunately the logs you provided doesn't have obvious format, but I believe we are able to figure out correct regex.

Do you have any idea for the first line regex as you know possible formats of the logs? We cannot base on the \n as it has special meaning in docker logging (that's why docker mode is needed)

[cmedley2]

That is interesting- It wont match the json obviously or the hello world log lines but when I run it through a regular expression with tester with the regexp it matches on this line: {"log":"2020-08-05 15:47:01,020 [EC2=] [Causality=] [CausalityChain=] [ERROR ] [o.s.s.s.TaskUtils$LoggingErrorHandler]","stream":"stdout","time":"2020-08-05T16:50:16.956076109Z"}

and this line: {"log":"2020-08-05 15:47:01,020 [EC2=] [Causality=] [CausalityChain=] [ERROR ] [o.s.s.s.TaskUtils$LoggingErrorHandler] [pool-4-thread-1] Unexpected error occurred in scheduled task.\n","stream":"stdout","time":"2020-08-05T16:49:16.864654995Z"}

Which also matches the behavior I am seeing in my results. When I turn on multiline and not docker mode it works as expected other than my json log > 16k does not get put together properly (because docker mode is off). The logs that don't match that follow matching lines are correctly concatonated with the matching lines. The lines that dont match that and do not follow a matching line are not concatenated.

That is different behavior from when I turn on docker mode and use the docker mode parser, in that case I see the beginning match concatenated with the following non matching lines, but also all non matching lines are concatenated together whether they follow a beginning match or not. I am not sure I understand the difference in behavior- I would expect multiline in docker mode and out to give the same behavior.

As to the what we could use, that is where I was hoping you could help (and explain the difference in behavior above. Essentially we want something that sees a json log as a beginning log as well, and I was hoping when you apply the multiline parser the check for the dockermode /n was already complete, and those logs already concatenated, so that we could use that in our regexp. That is essentially how we do it in fluentd today- we do the concat for docker mode first, then apply our multiline- which in the fluentd world looks something like this using the concat filter: multiline_start_regexp /^(-?(?:[1-9][0-9]*)?[0-9]{4})-(1[0-2]|0[1-9])-(3[01]|0[1-9]|[12][0-9])(T| )(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])|^{.+}/

Which is how we designated json logs and logs begining with a timestamp as beginning a multiline. I tried to use this as my docker mode multiline parser: (?^{"log":"\d{4}-\d{1,2}-\d{1,2}.\d{2}:\d{2}:\d{2}.|^{"log":"{.}"})

But it does not work- however going back and looking at the raw log file content again I think this might: (?^{"log":"\d{4}-\d{1,2}-\d{1,2}.\d{2}:\d{2}:\d{2}.|^{"log":"{.})

I will try that and see. Please let me know what you think, or if you have a better solution. Also an explanation of why the difference in behavior I am seeing for multiline with docker mode and multiline without.

Thank you.

[cmedley2]

not sure what happened with the cut and paste but in both my regexp above they should have . in between the brackets in my example above- for some reason the did not show up when I pasted it in.

[cmedley2]

So good news it looks like this regex for the dockermode multiline parser will give us parity with our current fluentd implementation: (?<log>^{"log":"\d{4}-\d{1,2}-\d{1,2}.\d{2}:\d{2}:\d{2}.*|^{"log":"{.*})

It is not perfect but it seems to work as well as the regex we use with fluentd. The rest is up to us to clean up and fix going forward. Thank you for getting me on the right track.

Still interested in getting your feedback on why there is a difference in behavior between Multiline, and the multiline support in Docker_Mode. Just to make sure we understand.

Thanks again.

[sumo-drosiek]

@cmedley2 There is multiple reasons why the multiline for docker_mode is implemented as it is. Due to architecture of fluent-bit tail plugin, reusing multiline in the docker_mode wasn't easy (if possible). Because of that I decided to implement simplified version of this feature. Also the requirements which I had to met were basing on the well defined format which should cover most of the cases (as we can see there are exceptions). The last but not least reason is keeping backward compatibility and introduce minimal set of changes to the existing flow.

Here is related PR: https://github.com/fluent/fluent-bit/pull/2043 And issue: https://github.com/fluent/fluent-bit/issues/1115

[cmedley2]

Thank you for the explanation, just wanted to be sure it was not unexpected behavior. I will keep it in mind, but I believe it covers our cases well enough to move forward for now.

And thanks again for all your efforts.

[sumo-drosiek]

@cmedley2 Happy to help 🤗 If you will have more questions in the future, open an issue and we'll respond 🚀

I believe we can close the issue 😉

[cmedley2]

Agreed!

[JulieLily]

Yes, that helps a lot. So I have all of the information. Regex for the multiline has to match the first line of the multiline log: regex: (?^{"log":"\d{4}-\d{1,2}-\d{1,2}.\d{2}:\d{2}:\d{2}.*) doesn't match any of the line from the file, so it won't work correctly. We need to figure out regex which matches only first line of log and is true for every log line. It is easy for timestamped logs (as example regex). Unfortunately the logs you provided doesn't have obvious format, but I believe we are able to figure out correct regex.

Do you have any idea for the first line regex as you know possible formats of the logs? We cannot base on the \n as it has special meaning in docker logging (that's why docker mode is needed)

I'm a little confused about this configuration 'Docker_Mode_Parser'. How does it parse multi line logs into one line according to this regular rule? What I understand is that，if the line match the regular rule, it will be treated as a new 'multi line', if the next line doesn't match the regular rule, it will be added to the 'multi line', otherwise, it wil be treated as a new 'multi line'. Is it right?

[sumo-drosiek]

@JulieLily

What I understand is that，if the line match the regular rule, it will be treated as a new 'multi line', if the next line doesn't match the regular rule, it will be added to the 'multi line', otherwise, it wil be treated as a new 'multi line'. Is it right?

That's correct

[JulieLily]

@JulieLily

What I understand is that，if the line match the regular rule, it will be treated as a new 'multi line', if the next line doesn't match the regular rule, it will be added to the 'multi line', otherwise, it wil be treated as a new 'multi line'. Is it right?

That's correct

Thanks~🤗

[AmanKrSoni]

@sumo-drosiek I have JSON logs generated by Kubernetes so if I want to concate multiline python error logs and spring boot logs which can further be shipped to es and view in kibana so which option has to use like docker_mode_parser and multiline or anything else

[sumo-drosiek]

@AmanKrSoni I would go with docker_mode_parser

[AmanKrSoni]

@sumo-drosiek Thanks but can you have that config example that more relevant and I have used fluent-bit docker parser for parsing in filter and input is tail

I has used below but doesn`t work and error [trackback] logs are not json Docker_Mode On Docker_Mode_Parser docker

[sumo-drosiek]

Lets move the conversation to the new issue #1074