kakao / network-node-manager

network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of kubernetes.
Apache License 2.0
108 stars 20 forks source link

failed to destroy rule externalIP to clusterIP {"error": "exit status 2"} #19

Open heartcored98 opened 1 year ago

heartcored98 commented 1 year ago

안녕하세요. RKE2 기반의 쿠버네티스 배포판을 온프레미스 환경에서 운영 중입니다. yatai라고 하고 BentoML 모델을 쿠버네티스에 쉽게 배포해주는 오픈소스를 쓰는 와중에 100MB 정도되는 모델을 yatai에 올릴려고 하니까 40~50MB 정도 업로드 되었을 때 항상 connection reset by peer에러가 발생했고, 검색하다 보니 감사하게도 공유해주신 이 오픈소스를 찾게 되었습니다.

그런데

Enable Drop Invalid Packet Rule in INPUT chain 옵션은 true, Enable External-IP to Cluster-IP DNAT Rule 옵션은 false

으로 설치하고 설정했을 때 모든 Pod들에서 Crashloopbackoff 에러가 발생하고 있습니다. env 설정 값은 정상적으로 들어간 것 같습니다. 혹시 어디서부터 손대면 좋을지 조언을 여쭤볼 수 있을까요?

Wed, Mar 1 2023 1:27:50 pm | I0301 04:27:50.915845 1 request.go:655] Throttling request took 1.019118868s, request: GET:https://10.43.0.1:443/apis/coordination.k8s.io/v1?timeout=32s
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.096Z INFO setup starting manager
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.096Z INFO controller-runtime.manager.controller.service Starting EventSource {"reconciler group": "", "reconciler kind": "Service", "source": "kind source: /, Kind="}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controller-runtime.manager.controller.service Starting Controller {"reconciler group": "", "reconciler kind": "Service"}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controller-runtime.manager.controller.service Starting workers {"reconciler group": "", "reconciler kind": "Service", "worker count": 1}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controllers.Service.initalize initalize service contoller
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controllers.Service.initalize config IPv4 pod CIDR {"IPv4 pod cIDR": "10.42.0.0/16"}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controllers.Service.initalize config IPv6 pod CIDR {"IPv6 pod cIDR": ""}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controllers.Service.initalize config for drop invalid packet in INPUT chain {"enabled": true}
Wed, Mar 1 2023 1:27:51 pm | 2023-03-01T04:27:51.298Z INFO controllers.Service.initalize config for externalIP to clusterIP {"enabled": false}
Wed, Mar 1 2023 1:27:52 pm | 2023-03-01T04:27:52.698Z ERROR controllers.Service.initalize {"error": "exit status 2"}
Wed, Mar 1 2023 1:27:52 pm | github.com/go-logr/zapr.(*zapLogger).Error
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
Wed, Mar 1 2023 1:27:52 pm | github.com/kakao/network-node-manager/pkg/rules.DestoryRulesExternalCluster
Wed, Mar 1 2023 1:27:52 pm | /workspace/pkg/rules/rule_external_cluster.go:87
Wed, Mar 1 2023 1:27:52 pm | github.com/kakao/network-node-manager/controllers.(*ServiceReconciler).Reconcile
Wed, Mar 1 2023 1:27:52 pm | /workspace/controllers/service_controller.go:133
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntil
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.UntilWithContext
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
Wed, Mar 1 2023 1:27:52 pm | 2023-03-01T04:27:52.699Z ERROR controllers.Service.initalize failed to destroy rule externalIP to clusterIP {"error": "exit status 2"}
Wed, Mar 1 2023 1:27:52 pm | github.com/go-logr/zapr.(*zapLogger).Error
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
Wed, Mar 1 2023 1:27:52 pm | github.com/kakao/network-node-manager/controllers.(*ServiceReconciler).Reconcile
Wed, Mar 1 2023 1:27:52 pm | /workspace/controllers/service_controller.go:134
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
Wed, Mar 1 2023 1:27:52 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntil
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
Wed, Mar 1 2023 1:27:52 pm | k8s.io/apimachinery/pkg/util/wait.UntilWithContext
Wed, Mar 1 2023 1:27:52 pm | /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
jc01rho commented 1 year ago

동일합니다. kubeadm 온프레미스 v1.25.9 , flannel:v0.22.0 , 환경에서 아래 내용이 발생합니다. 한번 봐주실수 있을까요? @ssup2

2023-06-02T03:58:02.320Z INFO setup starting manager
2023-06-02T03:58:02.320Z INFO controller-runtime.manager.controller.service Starting EventSource {"reconciler group": "", "reconciler kind": "Service", "source": "kind source: /, Kind="}
2023-06-02T03:58:02.420Z INFO controller-runtime.manager.controller.service Starting Controller {"reconciler group": "", "reconciler kind": "Service"}
2023-06-02T03:58:02.420Z INFO controller-runtime.manager.controller.service Starting workers {"reconciler group": "", "reconciler kind": "Service", "worker count": 1}
2023-06-02T03:58:02.421Z INFO controllers.Service.initalize initalize service contoller
2023-06-02T03:58:02.421Z INFO controllers.Service.initalize config IPv4 pod CIDR {"IPv4 pod cIDR": "10.244.0.0/16"}
2023-06-02T03:58:02.421Z INFO controllers.Service.initalize config IPv6 pod CIDR {"IPv6 pod cIDR": ""}
2023-06-02T03:58:02.421Z INFO controllers.Service.initalize config for drop invalid packet in INPUT chain {"enabled": true}
2023-06-02T03:58:02.421Z INFO controllers.Service.initalize config for externalIP to clusterIP {"enabled": false}
2023-06-02T03:58:02.910Z ERROR controllers.Service.initalize  {"error": "exit status 2"}
github.com/go-logr/zapr.(*zapLogger).Error
 /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
github.com/kakao/network-node-manager/pkg/rules.DestoryRulesExternalCluster
 /workspace/pkg/rules/rule_external_cluster.go:87
github.com/kakao/network-node-manager/controllers.(*ServiceReconciler).Reconcile
 /workspace/controllers/service_controller.go:133
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
 /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
 /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
 /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
 /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
 /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
 /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99