Closed manasabsv26 closed 1 month ago
Testing details with new code changes to add validation:
Edited the METRIC_BIND_ADRESS
value to "random" in configMap
apiVersion: v1
data:
METRICS_BIND_ADDRESS: random
controller_manager_config.yaml: |
apiVersion: controller-runtime.sigs.k8s.io/v1alpha1
hostplumber-controller-manager
pods came in error state with these logs:
kc logs -n luigi-system hostplumber-controller-manager-vsn9m
2024-07-08T08:00:15Z ERROR setup METRICS_BIND_ADDRESS is invalid {"error": "address must be in the format IP:port"}
main.main
/workspace/main.go:104
runtime.main
/usr/local/go/src/runtime/proc.go:271
Edited the port value to 77777
in configMap:
apiVersion: v1
data:
METRICS_BIND_ADDRESS: 127.0.0.1:77777
controller_manager_config.yaml: |
apiVersion: [controller-runtime.sigs.k8s.io/v1alpha1](http://controller-runtime.sigs.k8s.io/v1alpha1)
kind: ControllerManagerConfig
luigi-system cert-manager-webhook-68447b9c99-v9nn6 1/1 Running 0 143m
luigi-system hostplumber-controller-manager-fmv5f 1/2 Error 1 (2s ago) 3s
luigi-system hostplumber-controller-manager-nrh67 1/2 Error 1 (2s ago) 3s
hosplumber-controller-manager
pod logs
kc logs -n luigi-system hostplumber-controller-manager-fmv5f
2024-07-08T07:50:01Z ERROR setup METRICS_BIND_ADDRESS is invalid {"error": "port must be between 1 and 65535"}
main.main
/workspace/main.go:104
runtime.main
/usr/local/go/src/runtime/proc.go:271
After adding ipv6 format support in validation ( format - [ipv6 addr]:port), tested following cases:
Valid ipv6 address METRIC_BIND_ADDRESS: '[::1]:9000'
apiVersion: v1
data:
METRICS_BIND_ADDRESS: '[::1]:9000'
controller_manager_config.yaml: |
apiVersion: controller-runtime.sigs.k8s.io/v1alpha1
logs:
kc logs -n luigi-system hostplumber-controller-manager-jffkx
2024-07-08T12:02:47Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "[::1]:9000"}
2024-07-08T12:02:47Z INFO setup starting manager
2024-07-08T12:02:47Z INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::1]:9000"}
2024-07-08T12:02:47Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-07-08T12:02:47Z INFO Starting EventSource {"controller": "hostnetworktemplate", "controllerGroup": "plumber.k8s.pf9.io", "controllerKind": "HostNetworkTemplate", "source": "kind source: *v1.HostNetworkTemplate"}
2024-07-08T12:02:47Z INFO Starting Controller {"controller": "hostnetworktemplate", "controllerGroup": "plumber.k8s.pf9.io", "controllerKind": "HostNetworkTemplate"}
2024-07-08T12:02:47Z INFO Starting workers {"controller": "hostnetworktemplate", "controllerGroup": "plumber.k8s.pf9.io", "controllerKind": "HostNetworkTemplate", "worker count": 1}
Invalid ip address METRIC_BIND_ADDRESS: fe80::1:9000
apiVersion: v1
data:
METRICS_BIND_ADDRESS: fe80::1:9000
controller_manager_config.yaml: |
apiVersion: controller-runtime.sigs.k8s.io/v1alpha1
logs:
kc logs -n luigi-system hostplumber-controller-manager-pr2xl
2024-07-08T11:41:32Z ERROR setup METRICS_BIND_ADDRESS is invalid {"error": "address must be in the format IP:port"}
main.main
/workspace/main.go:122
runtime.main
/usr/local/go/src/runtime/proc.go:271
'[::1]:77777'
apiVersion: v1
data:
METRICS_BIND_ADDRESS: '[::1]:77777'
controller_manager_config.yaml: |
apiVersion: controller-runtime.sigs.k8s.io/v1alpha1
logs:
kc logs -n luigi-system hostplumber-controller-manager-cx2b9
2024-07-08T12:09:28Z ERROR setup METRICS_BIND_ADDRESS is invalid {"error": "port must be between 1 and 65535"}
main.main
/workspace/main.go:122
runtime.main
/usr/local/go/src/runtime/proc.go:271
random:9000
logs:
kc logs -n luigi-system hostplumber-controller-manager-n5nh5
2024-07-08T12:11:34Z ERROR setup METRICS_BIND_ADDRESS is invalid {"error": "invalid IP address"}
main.main
/workspace/main.go:122
runtime.main
/usr/local/go/src/runtime/proc.go:271
127.0.0.1:9000
kc logs -n luigi-system hostplumber-controller-manager-c48z9
2024-07-08T12:13:09Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:9000"}
2024-07-08T12:13:09Z INFO setup starting manager
2024-07-08T12:13:09Z INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:9000"}
2024-07-08T12:13:09Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-07-08T12:13:09Z INFO Starting EventSource {"controller": "hostnetworktemplate", "controllerGroup": "plumber.k8s.pf9.io", "controllerKind": "HostNetworkTemplate", "source": "kind source: *v1.HostNetworkTemplate"}
Removed validation function as discussed with @mithilarun. ctrl.NewManager
should take care of validating the metrics address.
ISSUE(S):
PMK-6484
SUMMARY:
Added the ability to customize hostplumber metrics port by changing the value of
METRICS_BIND_ADDRESS
inhostplumber-manager-config
configmapTESTING:
With these changes, built a new luigi and hostplumber image, and tested these images Pods came up fine:
Events of
luigi-controller-manager-794f54794c-w4qbg
pod:Events of
hostplumber-controller-manager-zxdrz
pod:Environment details in
describe
of one of the hostplumber-controller-manager podsDaemonset:
hostplumber-manager-config
cm:Changed METRICS_BIND_ADDRESS to
127.0.0.1:8085
Deleted the hostplumber-controller-manager pods after making this configMap change, then when new pods came up, checked their logs
Logs of
hostplumber-controller-manager-zxdrz
pod:Could see that it's now using newer metric-bind-address, as provided in configmap