UnrecoverableError - encountered parsing namespace watch event

[kojanabzah]

yes, this again :( found few bug that closed including #249 #239 , i still have restarting issues , i think the previews bugs fixed 410 status code , my status code is 401 Unauthorized. not sure why . any information will be helpful.

2022-01-11T11:12:58.094954278Z stdout F 2022-01-11 11:12:58 +0000 [error]: #0 Exception encountered parsing namespace watch event. The connection might have been closed. Retried 10 times yet still failing. Restarting.HTTP status code 401, Unauthorized for GET https://<remove by me>/api/v1/namespaces?resourceVersion=0
2022-01-11T11:12:58.095150533Z stderr F #<Thread:0x00007fdbfbcdc2b8 run> terminated with exception (report_on_exception is true):
2022-01-11T11:12:58.095170684Z stderr F /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:72:in `rescue in block in set_up_namespace_thread': Exception encountered parsing namespace watch event. The connection might have been closed. Retried 10 times yet still failing. Restarting. (Fluent::UnrecoverableError)
2022-01-11T11:12:58.09527144Z stderr F  from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:41:in `block in set_up_namespace_thread'
2022-01-11T11:12:58.095283788Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:40:in `loop'
2022-01-11T11:12:58.095289949Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:40:in `set_up_namespace_thread'
2022-01-11T11:12:58.095297062Z stderr F /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.2/lib/kubeclient/common.rb:130:in `rescue in handle_exception': Unauthorized (Kubeclient::HttpError)
2022-01-11T11:12:58.095303226Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.2/lib/kubeclient/common.rb:120:in `handle_exception'
2022-01-11T11:12:58.095309032Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.2/lib/kubeclient/common.rb:350:in `get_entities'
2022-01-11T11:12:58.0953141Z stderr F   from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.2/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
2022-01-11T11:12:58.095319393Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:95:in `get_namespaces_and_start_watcher'
2022-01-11T11:12:58.095332829Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:41:in `block in set_up_namespace_thread'
2022-01-11T11:12:58.095337787Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:40:in `loop'
2022-01-11T11:12:58.095342965Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:40:in `set_up_namespace_thread'
2022-01-11T11:12:58.095347863Z stderr F /fluentd/vendor/bundle/ruby/2.6.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response': 401 Unauthorized (RestClient::Unauthorized)
2022-01-11T11:12:58.095366982Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
2022-01-11T11:12:58.09537173Z stderr F  from /fluentd/vendor/bundle/ruby/2.6.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
2022-01-11T11:12:58.095376902Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
2022-01-11T11:12:58.095382394Z stderr F     from /usr/local/lib/ruby/2.6.0/net/http.rb:920:in `start'
2022-01-11T11:12:58.095387147Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
2022-01-11T11:12:58.09539165Z stderr F  from /fluentd/vendor/bundle/ruby/2.6.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
2022-01-11T11:12:58.095395749Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
2022-01-11T11:12:58.095399137Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
2022-01-11T11:12:58.095403044Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.2/lib/kubeclient/common.rb:352:in `block in get_entities'
2022-01-11T11:12:58.095406496Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.2/lib/kubeclient/common.rb:121:in `handle_exception'
2022-01-11T11:12:58.095409783Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.2/lib/kubeclient/common.rb:350:in `get_entities'
2022-01-11T11:12:58.09541314Z stderr F  from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.2/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
2022-01-11T11:12:58.095417207Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:95:in `get_namespaces_and_start_watcher'
2022-01-11T11:12:58.095420806Z stderr F     from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:41:in `block in set_up_namespace_thread'
2022-01-11T11:12:58.09542424Z stderr F  from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:40:in `loop'
2022-01-11T11:12:58.09542777Z stderr F  from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:40:in `set_up_namespace_thread'
2022-01-11T11:12:58.095482714Z stdout F 2022-01-11 11:12:58 +0000 [error]: #0 unexpected error error_class=Fluent::UnrecoverableError error="Exception encountered parsing namespace watch event. The connection might have been closed. Retried 10 times yet still failing. Restarting."
2022-01-11T11:12:58.095490543Z stdout F   2022-01-11 11:12:58 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:72:in `rescue in block in set_up_namespace_thread'
2022-01-11T11:12:58.095495324Z stdout F   2022-01-11 11:12:58 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:41:in `block in set_up_namespace_thread'
2022-01-11T11:12:58.095499883Z stdout F   2022-01-11 11:12:58 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:40:in `loop'
2022-01-11T11:12:58.095504627Z stdout F   2022-01-11 11:12:58 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.7.2/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:40:in `set_up_namespace_thread'
2022-01-11T11:12:58.095514114Z stdout F 2022-01-11 11:12:58 +0000 [error]: #0 Exception encountered parsing namespace watch event. The connection might have been closed. Retried 10 times yet still failing. Restarting. error_class=Fluent::UnrecoverableError error="Exception encountered parsing namespace watch event. The connection might have been closed. Retried 10 times yet still failing. Restarting."
2022-01-11T11:12:58.09556769Z stdout F   2022-01-11 11:12:58 +0000 [error]: #0 suppressed same stacktrace
2022-01-11T11:12:58.447258603Z stdout F 2022-01-11 11:12:58 +0000 [info]: [filter_kube_metadata] Exception encountered parsing pod watch event. The connection might have been closed. Sleeping for 1 seconds and resetting the pod watcher.HTTP status code 401, Unauthorized
2022-01-11T11:12:58.608614805Z stdout F 2022-01-11 11:12:58 +0000 [info]: Worker 0 finished unexpectedly with status 2
2022-01-11T11:12:58.608654221Z stdout F 2022-01-11 11:12:58 +0000 [info]: Received graceful stop

[jcantrill]

my status code is 401 Unauthorized

The obvious question here is: Are you authorized to watch namespaces? It's a 401 error meaning try to figure out why you don't have the right permissions. Closing issue. Please re-open if you can provide definitive artifacts demonstrating the plugin is improperly consuming credentials