search

travisghansen / kubernetes-pfsense-controller

Integrate Kubernetes and pfSense
Apache License 2.0
197 stars 22 forks source link

Plugin "pfsense-dns-haproxy-ingress-proxy" sets last host alias only #8

Closed aded closed 4 years ago

aded commented 4 years ago

Hi Travis, when I add more than one ingress on my cluster (created via different Helm charts), I only find an host alias in unbound, the one created through the last ingress. I guess the plugin sets (not adds) the host aliases...? My ingress controller is nginx-ingress.

TIA!

travisghansen commented 4 years ago

Can you share your config along with the sample ingresses?

aded commented 4 years ago

Sure. My kpc config:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-system
  name: kubernetes-pfsense-controller-config
data:
  config: |
    controller-id: "mycluster"
    enabled: true
    plugins:
      metallb:
        enabled: false
      haproxy-declarative:
        enabled: true
      haproxy-ingress-proxy:
        enabled: true
        ingressLabelSelector:
        ingressFieldSelector:
        defaultFrontend: private-http
        defaultBackend: private-http
        # by default anything is allowed
        #allowedHostRegex: "/.*/"
      pfsense-dns-services:
        enabled: false
      pfsense-dns-ingresses:
        enabled: false
        ingressLabelSelector:
        ingressFieldSelector:
        #allowedHostRegex: "/.*/"
        dnsBackends:
          dnsmasq:
            enabled: true
          unbound:
            enabled: true
      pfsense-dns-haproxy-ingress-proxy:
        enabled: true
        #allowedHostRegex: "/.*/"
        dnsBackends:
          dnsmasq:
            enabled: false
          unbound:
            enabled: true
        frontends:
          private-http:
            hostname: private-http.cluster
          public-https:
            hostname: public-https.cluster

Both frontends have the right IP, I'm using only "private-http" anyway. I deploy 2 nginx ingress controllers via-Helm, these are the values files:

controller:
  ingressClass: "nginx-private"
  kind: DaemonSet
  service:
    enableHttp: True
    enableHttps: False
    externalTrafficPolicy: Local
    type: NodePort

and

controller:
  ingressClass: "nginx-public"
  kind: DaemonSet
  service:
    enableHttp: False
    enableHttps: True
    externalTrafficPolicy: Local
    type: NodePort

Let's say I first deploy graylog (via-Helm again): the host I specify in Helm values is correctly added as host alias. Then I deploy another chart (prometheus-operator): I end up with the alertmanager host alias only, the last one specified in prometheus-operator's values.

Anyway if you can't reproduce the issue with a "standard" configuration, I'm definitely doing something wrong... :-)

travisghansen commented 4 years ago

OK, can you try out the same setup with dnsmasq by chance? This very well could be a bug..I'll see if I can get a sample setup going..

travisghansen commented 4 years ago

Any luck trying this with dnsmasq? I'll probably have a chance to dig into this a bit more in the next couple days..

aded commented 4 years ago

Hi Travis, I will be able to try within tomorrow, I'll update you asap!

travisghansen commented 4 years ago

I've found the bug(s) and will have a new release shortly as an FYI.

travisghansen commented 4 years ago

This should be fixed in v0.2.0. Thanks for the report!

aded commented 4 years ago

Hi Travis, thanks for your time: I apologize for the delay in responding, I was able to set up a test env right now. It seems the issue is still there, both in DNS-Resolver and DNS-Forwarder.

Relevant, maybe useful, logs from controller:

2020-04-17T09:59:40+00:00 store successfully initialized
2020-04-17T09:59:40+00:00 waiting for ConfigMap kube-system/kubernetes-pfsense-controller-config to be present and valid
2020-04-17T09:59:46+00:00 controller config loaded/updated
2020-04-17T09:59:46+00:00 loading plugin haproxy-declarative
2020-04-17T09:59:46+00:00 loading plugin haproxy-ingress-proxy
2020-04-17T09:59:46+00:00 loading plugin pfsense-dns-haproxy-ingress-proxy
2020-04-17T09:59:47+00:00 plugin (haproxy-declarative): successfully reloaded HAProxy service
PHP Warning:  Illegal string offset 'host' in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/Utils.php on line 42
PHP Warning:  Illegal string offset 'host' in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/Utils.php on line 42
2020-04-17T09:59:54+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded dnsmasq service
2020-04-17T10:00:01+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded unbound service
2020-04-17T10:00:01+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded DHCP service
2020-04-17T10:00:02+00:00 plugin (haproxy-ingress-proxy): successfully reloaded HAProxy service
2020-04-17T10:00:05+00:00 plugin (haproxy-declarative): successfully reloaded HAProxy service
2020-04-17T10:00:08+00:00 plugin (haproxy-declarative): successfully reloaded HAProxy service
2020-04-17T10:02:07+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /apis/networking.k8s.io/v1beta1/namespaces/pdragon/ingresses/prometheus-operator-prometheus ADDED - 33143
2020-04-17T10:02:07+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /apis/networking.k8s.io/v1beta1/namespaces/pdragon/ingresses/prometheus-operator-alertmanager ADDED - 33144
2020-04-17T10:02:07+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /apis/networking.k8s.io/v1beta1/namespaces/pdragon/ingresses/prometheus-operator-grafana ADDED - 33145
2020-04-17T10:02:17+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /apis/networking.k8s.io/v1beta1/namespaces/pdragon/ingresses/prometheus-operator-prometheus MODIFIED - 33332
2020-04-17T10:02:17+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /apis/networking.k8s.io/v1beta1/namespaces/pdragon/ingresses/prometheus-operator-grafana MODIFIED - 33333
2020-04-17T10:02:17+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /apis/networking.k8s.io/v1beta1/namespaces/pdragon/ingresses/prometheus-operator-alertmanager MODIFIED - 33334
2020-04-17T10:02:18+00:00 plugin (haproxy-ingress-proxy): creating frontend: pdragon-pdragon-prometheus-operator-prometheus
2020-04-17T10:02:18+00:00 plugin (haproxy-ingress-proxy): creating frontend: pdragon-pdragon-prometheus-operator-alertmanager
2020-04-17T10:02:18+00:00 plugin (haproxy-ingress-proxy): creating frontend: pdragon-pdragon-prometheus-operator-grafana
2020-04-17T10:02:19+00:00 plugin (haproxy-ingress-proxy): successfully reloaded HAProxy service
PHP Warning:  Illegal string offset 'host' in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/Utils.php on line 42
PHP Warning:  Illegal string offset 'host' in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/Utils.php on line 42
PHP Warning:  Illegal string offset 'host' in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/Utils.php on line 42
PHP Warning:  Illegal string offset 'host' in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/Utils.php on line 42
PHP Warning:  Illegal string offset 'host' in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/Utils.php on line 42
PHP Warning:  Illegal string offset 'host' in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/Utils.php on line 42
2020-04-17T10:02:25+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded dnsmasq service
2020-04-17T10:02:32+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded unbound service
2020-04-17T10:02:33+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded DHCP service

I'm using v0.2.0 (don't forget to change version in deployment.yml... :-D):

  Normal  Pulling    11m   kubelet, kworker-3  Pulling image "docker.io/travisghansen/kubernetes-pfsense-controller:v0.2.0"
  Normal  Pulled     11m   kubelet, kworker-3  Successfully pulled image "docker.io/travisghansen/kubernetes-pfsense-controller:v0.2.0"

When I deploy prometheus-operator (I deploy it only, as it enables 3 different hosts - alertmanager, grafana and prometheus itself), only one host is present as host alias, both in unbound and dnsmasq. My pfSense version is 2.4.4-RELEASE-p3.

Anyway I will try a completely fresh install asap (I hope in the next few hours!) and I'll write you updates if any. Bye!

aded commented 4 years ago

BTW I noticed that the latest version is 0.2.1: tried it, same result and same logs.

travisghansen commented 4 years ago

Do you 1 ingress with multiple hosts or just multiple ingresses?

aded commented 4 years ago

I have 2 ingresses as described in previous comments, but only one is used. These are my values for prometheus-operator Helm chart:

---
prometheusOperator:
  createCustomResource: false

prometheus:
  ingress:
    annotations:
      kubernetes.io/ingress.class: "nginx-private"
    enabled: true
    hosts: [ "prom.mon.test-current" ]

alertmanager:
  ingress:
    annotations:
      kubernetes.io/ingress.class: "nginx-private"
    enabled: true
    hosts: [ "alerts.mon.test-current" ]

grafana:
  ingress:
    annotations:
      kubernetes.io/ingress.class: "nginx-private"
    enabled: true
    hosts: [ "mon.test-current" ]

KPC config for haproxy-ingress-proxy:

      haproxy-ingress-proxy:
        enabled: true
        ingressLabelSelector:
        ingressFieldSelector:
        defaultFrontend: private-http
        defaultBackend: private-http
        # by default anything is allowed
        #allowedHostRegex: "/.*/"

KPC config for pfsense-dns-haproxy-ingress-proxy:

      pfsense-dns-haproxy-ingress-proxy:
        enabled: true
        #allowedHostRegex: "/.*/"
        dnsBackends:
          dnsmasq:
            enabled: true
          unbound:
            enabled: true
        frontends:
          private-http:
            hostname: private-http.pdragon
          public-https:
            hostname: public-https.pdragon
          private-5141:
            hostname: private-5141.pdragon

And I end up with one alias (prom.mon.test-current) for hostname "private-http.pdragon". Let me build a cluster from scratch and I'll update you if good news...

travisghansen commented 4 years ago

ok, I'm retesting as I likely still have something off with this. The correct rules are created in haproxy for all the ingresses yeah?

aded commented 4 years ago

Yes, the pfSense's HAProxy is ok (frontends/backends), and if I manually add host aliases it works.

travisghansen commented 4 years ago

OK, I'm pretty sure I fixed your issue. Can you give v0.2.2 or latest a try?

aded commented 4 years ago

Hi Travis, it works like a charm! :-) Going to close the issue...

travisghansen commented 4 years ago

Nice! Thanks for the report and patience!

aded commented 4 years ago

Thanks to you for the great job and the controller! I might have more time in the next weeks to take a look at the code, so I might send a PR directly next time... Bye!