Added metricsPort to hostplumber plugin

[jayanth-tjvrr]

This PR solves PMK-6532

So, Previous solution was not persisting over the upgrades. This solution will require you to provide metricsPort as a part of networkplugins.yaml. Example of the file:

apiVersion: plumber.k8s.pf9.io/v1
kind: NetworkPlugins
metadata:
  name: networkplugins-sample-nosriov
spec:
  # Add fields here
  #privateRegistryBase: "localhost:5100"
  plugins:
    hostPlumber:
      metricsPort: "8090"
      namespace: default
    nodeFeatureDiscovery:
      namespace: default
    multus:
      namespace: default
    whereabouts:
      namespace: default
      ipReconcilerSchedule: "* * * * *"
    #sriov: {}
    #ovs: {}

Testing: Tested this on a DU and the change persisted over the upgrade from 1.28 to 1.29. Attaching the snippet of hostplumber CM:

apiVersion: v1
data:
  METRICS_BIND_ADDRESS: 127.0.0.1:8090
  controller_manager_config.yaml: |
    apiVersion: controller-runtime.sigs.k8s.io/v1alpha1
    kind: ControllerManagerConfig
    health:
      healthProbeBindAddress: :8081
    webhook:
      port: 9443
    leaderElection:
      leaderElect: true
      resourceName: 52f205ce.k8s.pf9.io
kind: ConfigMap
metadata:
  creationTimestamp: "2024-08-13T12:05:49Z"
  name: hostplumber-manager-config
  namespace: default
  resourceVersion: "5972"
  uid: 3fb2c295-c29f-4902-b3aa-79bacf80f528