search

fluent / fluentd-kubernetes-daemonset

Fluentd daemonset for Kubernetes and it Docker image
Apache License 2.0
1.26k stars 978 forks source link

NO_PROXY variable not honored #1372

Open waldner opened 2 years ago

waldner commented 2 years ago

Running fluentd-kubernetes-daemonset:v1.14.6-debian-elasticsearch7-1.0 inside a k8s cluster behind a proxy. The container has the HTTP_PROXY/HTTPS_PROXY (and their lowercase version) variables set, which are honored; however, the proxy should not be used to connect to the k8s API, so I set NO_PROXY (and no_proxy) to kubernetes,10.43.0.1,kubernetes.default.svc, yet it looks like the API is not being accessed directly. Here are some errors from the log:

Successfully installed fluent-plugin-kubernetes-objects-1.1.12
1 gem installed
2022-07-20 13:13:02 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-kubernetes-objects' version '1.1.12'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.9.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluentd' version '1.14.6'
2022-07-20 13:13:02 +0000 [warn]: [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
#<Thread:0x00007fe581a3da90 run> terminated with exception (report_on_exception is true):
/fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:87:in `rescue in start_pod_watch': start_pod_watch: Exception encountered setting up pod watch from Kubernetes API v1 endpoint https://10.43.0.1:443/api: pods is forbidden: User "system:serviceaccount:myns:default" cannot list resource "pods" in API group "" at the cluster scope ({"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \\"system:serviceaccount:myns:default\\" cannot list resource \\"pods\\" in API group \\"\\" at the cluster scope","reason":"Forbidden","details":{"kind":"pods"},"code":403} (Fluent::ConfigError)
)
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:78:in `start_pod_watch'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'
/fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:130:in `rescue in handle_exception': pods is forbidden: User "system:serviceaccount:myns:default" cannot list resource "pods" in API group "" at the cluster scope (Kubeclient::HttpError)
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:120:in `handle_exception'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:350:in `get_entities'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:101:in `method_missing'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:102:in `get_pods_and_start_watcher'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:79:in `start_pod_watch'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'
/fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response': 403 Forbidden (RestClient::Forbidden)
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
    from /usr/local/lib/ruby/2.7.0/net/http.rb:933:in `start'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:352:in `block in get_entities'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:121:in `handle_exception'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:350:in `get_entities'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:101:in `method_missing'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:102:in `get_pods_and_start_watcher'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:79:in `start_pod_watch'
    from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'

On another cluster running without the proxy (everything else the same), no error is produced.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

waldner commented 1 year ago

The issue is still relevant.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

waldner commented 1 year ago

The issue is still relevant.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

waldner commented 1 year ago

The issue is still relevant.

mikaelkrief commented 1 year ago

any solution to this issue ?

vittico commented 1 year ago

Is there any solution available?

github-actions[bot] commented 10 months ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

waldner commented 10 months ago

This is still an issue.

github-actions[bot] commented 7 months ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

waldner commented 7 months ago

This is still an issue.

github-actions[bot] commented 4 months ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

waldner commented 4 months ago

This is still an issue.

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

waldner commented 1 month ago

This is still an issue.