Open rpurdel opened 1 year ago
I have the following segmentation fault in Kubernetes pods when using Docker image fluent/fluentd-kubernetes-daemonset:v1.15.3-debian-cloudwatch-1.0
:
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180: [BUG] Segmentation fault at 0x0000000000000048
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
-- Control frame information -----------------------------------------------
c:0030 p:---- s:0175 e:000174 CFUNC :write
c:0029 p:0036 s:0170 e:000169 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180
c:0028 p:0010 s:0163 e:000162 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1039
c:0027 p:0022 s:0157 e:000156 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196 [FINISH]
c:0026 p:---- s:0153 e:000152 CFUNC :each
c:0025 p:0023 s:0149 e:000148 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195
c:0024 p:0058 s:0139 e:000138 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1038
c:0023 p:0014 s:0127 e:000126 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:922
c:0022 p:0020 s:0119 e:000118 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:852
c:0021 p:0011 s:0113 e:000112 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0020 p:0009 s:0105 e:000104 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/out_relabel.rb:29
c:0019 p:0013 s:0099 e:000098 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840
c:0018 p:0020 s:0093 e:000092 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196
c:0017 p:0011 s:0085 e:000084 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0016 p:0064 s:0077 e:000076 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615 [FINISH]
c:0015 p:---- s:0069 e:000068 IFUNC
c:0014 p:0257 s:0066 e:000063 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104
c:0013 p:0047 s:0056 e:000055 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140
c:0012 p:0026 s:0050 e:000049 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064
c:0011 p:0003 s:0046 e:000045 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006 [FINISH]
c:0010 p:---- s:0043 e:000042 CFUNC :synchronize
c:0009 p:0006 s:0039 e:000038 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006
c:0008 p:0058 s:0035 e:000034 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817
c:0007 p:0004 s:0030 e:000029 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394
c:0006 p:0005 s:0027 e:000026 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749 [FINISH]
c:0005 p:---- s:0022 e:000021 CFUNC :run_once
c:0004 p:0041 s:0017 e:000016 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88
c:0003 p:0033 s:0012 e:000011 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93
c:0002 p:0080 s:0008 e:000007 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]
-- Ruby level backtrace information ----------------------------------------
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749:in `on_timer'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394:in `block in setup_watcher'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `block in on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064:in `handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140:in `with_io'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104:in `block in handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615:in `receive_lines'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196:in `emit_events'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840:in `emit_sync'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/out_relabel.rb:29:in `process'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:852:in `emit_buffered'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:922:in `execute_chunking'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1038:in `handle_stream_simple'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196:in `block in each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1039:in `block in handle_stream_simple'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180:in `format'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180:in `write'
-- Machine register context ------------------------------------------------
RIP: 0x00007f4261b69922 RBP: 0x00007f4260e03b18 RSP: 0x00007f42524fb8b0
RAX: 0x00007f4260eb9101 RBX: 0x00007f4260ea7c00 RCX: 0x0000000000000006
RDX: 0xffffffffffff9000 RDI: 0xffffffffffffffbf RSI: 0x00007f4260ea7c00
R8: 0x000000000000000c R9: 0x0000000000001988 R10: 0x00007f4260eb91a8
R11: 0x0000000000000000 R12: 0x0000000000007000 R13: 0x0000000000000000
R14: 0x0000000000000000 R15: 0x0000000000000006 EFL: 0x0000000000010202
-- C level backtrace information -------------------------------------------
It's hard for me to tell if it's related.
We have also experienced a similar segmentation fault using the image v1.15-debian-cloudwatch-1
:
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371: [BUG] Segmentation fault at 0x0000000000010047
ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux]
-- Control frame information -----------------------------------------------
c:0045 p:---- s:0266 e:000265 CFUNC :dump
c:0044 p:0143 s:0261 e:000259 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_meta
c:0043 p:0010 s:0245 e:000244 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:96
c:0042 p:0011 s:0238 e:000237 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:110
c:0041 p:0031 s:0231 e:000230 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:94
c:0040 p:0008 s:0224 e:000223 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:218 [FINISH]
c:0039 p:---- s:0218 e:000217 IFUNC
c:0038 p:---- s:0215 e:000214 CFUNC :each
c:0037 p:---- s:0212 e:000211 CFUNC :reduce
c:0036 p:0021 s:0207 e:000206 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217
c:0035 p:0009 s:0201 e:000200 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:193
c:0034 p:0011 s:0193 e:000192 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0033 p:0028 s:0185 e:000184 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:106
c:0032 p:0011 s:0178 e:000177 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:103
c:0031 p:0038 s:0173 e:000172 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:338
c:0030 p:0073 s:0164 e:000163 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:288
c:0029 p:0122 s:0156 e:000155 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:107 [FINISH]
c:0028 p:---- s:0151 e:000150 CFUNC :synchronize
c:0027 p:0014 s:0147 e:000146 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:137
c:0026 p:0004 s:0142 e:000141 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:90
c:0025 p:0009 s:0135 e:000134 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:82
c:0024 p:0022 s:0130 e:000129 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196 [FINISH]
c:0023 p:---- s:0126 e:000125 CFUNC :each
c:0022 p:0023 s:0122 e:000121 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195
c:0021 p:0005 s:0112 e:000111 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:81
c:0020 p:0016 s:0105 e:000104 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/compat/output.rb:164
c:0019 p:0013 s:0099 e:000098 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840
c:0018 p:0020 s:0093 e:000092 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196
c:0017 p:0011 s:0085 e:000084 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0016 p:0064 s:0077 e:000076 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615 [FINISH]
c:0015 p:---- s:0069 e:000068 IFUNC
c:0014 p:0257 s:0066 e:000063 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104
c:0013 p:0047 s:0056 e:000055 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140
c:0012 p:0026 s:0050 e:000049 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064
c:0011 p:0003 s:0046 e:000045 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006 [FINISH]
c:0010 p:---- s:0043 e:000042 CFUNC :synchronize
c:0009 p:0006 s:0039 e:000038 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006
c:0008 p:0058 s:0035 e:000034 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817
c:0007 p:0004 s:0030 e:000029 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394
c:0006 p:0005 s:0027 e:000026 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749 [FINISH]
c:0005 p:---- s:0022 e:000021 CFUNC :run_once
c:0004 p:0041 s:0017 e:000016 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88
c:0003 p:0033 s:0012 e:000011 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93
c:0002 p:0080 s:0008 e:000007 BLOCK /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]
-- Ruby level backtrace information ----------------------------------------
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749:in `on_timer'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394:in `block in setup_watcher'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `block in on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064:in `handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140:in `with_io'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104:in `block in handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615:in `receive_lines'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196:in `emit_events'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840:in `emit_sync'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/compat/output.rb:164:in `process'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:81:in `emit'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196:in `block in each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:82:in `block in emit'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:90:in `process_record'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:137:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:137:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:107:in `block in process_record'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:288:in `push'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:338:in `update_buffer'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:103:in `block (2 levels) in process_record'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:106:in `emit'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:193:in `emit_events'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `reduce'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:218:in `block in filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:94:in `filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:110:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:96:in `block in filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371:in `filter'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371:in `dump'
-- Machine register context ------------------------------------------------
RIP: 0x00007f3651497922 RBP: 0x00007f3637a03a98 RSP: 0x00007f3637efb220
RAX: 0x00007f3637ab5401 RBX: 0x00007f3637a9bb00 RCX: 0x0000000000000006
RDX: 0x00007f3637aa7d00 RDI: 0xffffffffffffffbf RSI: 0x00007f3637a9bb00
R8: 0x0000000000000000 R9: 0x00007f3628752000 R10: 0x00007f3637ab54a8
R11: 0x0000000000000000 R12: 0x0000000000007000 R13: 0x0000000000000000
R14: 0x000000000000ffff R15: 0x0000000000000006 EFL: 0x0000000000010206
-- C level backtrace information -------------------------------------------
2023-04-11 18:11:12 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 18:46:53 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 19:30:29 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 20:06:03 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 20:49:46 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days
This issue was automatically closed because of stale in 30 days
FTR: As problem resolution we migrated from Fluentd to Fluentbit.
All of reports are against v1.15 series (ruby 3.1) Are these issue still reproducible in v1.16 or v1.17?
Using
fluentd-kubernetes-daemonset:v1.15-debian-cloudwatch-amd64-1
and sometimes getting a segmentation fault in the k8s metadata filter. Any idea why this is?