search

travisghansen / kubernetes-pfsense-controller

Integrate Kubernetes and pfSense
Apache License 2.0
197 stars 22 forks source link

CrashLoopBackOff: invalid username or password #15

Closed remkolems closed 3 years ago

remkolems commented 3 years ago

First of all I can't express my gratitude enough for the Kubernetes pfSense controller!

System details

k3s -version

k3s version v1.21.3+k3s1 (1d1f220f) go version go1.16.6

Ubuntu version

Linux 5.8.0-63-generic #71~20.04.1-Ubuntu SMP Thu Jul 15 17:46:08 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

pfSense version

2.5.2-RELEASE (amd64) built on Fri Jul 02 15:33:00 EDT 2021 FreeBSD 12.2-STABLE

Installation details

kubectl method

i.e. kubectl apply -f secret.yaml, in which secret.yaml was edited and adjusted according the pfSense admin password. The password was generated via the following command within the OS system where k3s is running.

To generate a base64 encoded password: echo 'blabla' | base64

YmxhYmxhCg==

And to decode the base64 code into a human readable form echo 'YmxhYmxhCg==' | base64 -d

blabla

The password is than defined in secret.yaml as follows:

apiVersion: v1
kind: Secret
metadata:
  name: kubernetes-pfsense-controller
  namespace: kube-system
type: Opaque
data:
  pfsense-password: YmxhYmxhCg==

Log details

kubectl logs kubernetes-pfsense-controller-668c59c454-rlxbs -n kube-system

PHP Deprecated:  Required parameter $callback follows optional parameter $params in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-client-php/src/KubernetesClient/Client.php on line 170
PHP Warning:  Undefined array key "data" in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Store.php on line 128
PHP Deprecated:  Required parameter $callback follows optional parameter $params in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-client-php/src/KubernetesClient/Watch.php on line 137
2021-08-01T11:47:05+00:00 store successfully initialized
2021-08-01T11:47:05+00:00 controller config loaded/updated
2021-08-01T11:47:05+00:00 loading plugin metallb
PHP Warning:  Undefined array key "configMap" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php on line 30
2021-08-01T11:47:05+00:00 loading plugin haproxy-declarative
2021-08-01T11:47:05+00:00 loading plugin haproxy-ingress-proxy
2021-08-01T11:47:05+00:00 loading plugin pfsense-dns-services
PHP Warning:  Undefined array key "serviceLabelSelector" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSIngresses.php on line 32
PHP Warning:  Undefined array key "serviceFieldSelector" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSIngresses.php on line 33
2021-08-01T11:47:05+00:00 loading plugin pfsense-dns-ingresses
2021-08-01T11:47:05+00:00 loading plugin pfsense-dns-haproxy-ingress-proxy
PHP Warning:  Trying to access array offset on value of type null in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Store.php on line 218
2021-08-01T11:47:05+00:00 plugin (metallb): /v1/namespaces/metallb-system/ConfigMap/config ADDED - 135950
PHP Fatal error:  Uncaught Laminas\XmlRpc\Client\Exception\FaultException: Authentication failed: Invalid username or password in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/laminas/laminas-xmlrpc/src/Client.php:324
Stack trace:
#0 phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/XmlRpc/Client.php(59): Laminas\XmlRpc\Client->call('pfsense.backup_...', Array)
#1 phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/PfSenseConfigBlock.php(96): KubernetesPfSenseController\XmlRpc\Client->call('pfsense.backup_...', Array)
#2 phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php(141): KubernetesPfSenseController\Plugin\PfSenseConfigBlock::getInstalledPackagesConfigBlock(Object(KubernetesPfSenseController\XmlRpc\Client), 'frrbgpneighbors')
#3 phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php(117): KubernetesPfSenseController\Plugin\MetalLB->doActionGeneric()
#4 phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Plugin/AbstractPlugin.php(108): KubernetesPfSenseController\Plugin\MetalLB->doAction()
#5 phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Controller.php(532): KubernetesController\Plugin\AbstractPlugin->invokeAction()
#6 phar:///usr/local/bin/kubernetes-pfsense-controller/controller.php(87): KubernetesController\Controller->main()
#7 /usr/local/bin/kubernetes-pfsense-controller(2): include('phar:///usr/loc...')
#8 {main}
  thrown in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/laminas/laminas-xmlrpc/src/Client.php on line 324

In other words

PHP Fatal error: Uncaught Laminas\XmlRpc\Client\Exception\FaultException: Authentication failed: Invalid username or password in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/laminas/laminas-xmlrpc/src/Client.php:324

Questions

  1. How do you define, generate and/or structure the base64 password of pfSense within the secret.yaml? Or even perhaps extract the password from config.xml?
  2. Could you give an exact example of the secret.yaml file?
  3. Could you also provide an extra section with applicable tested against or an overview on the first github landing page. As of now it unclear which version of kubernetes and pfSense the kubernetes-pfsense-controlller is succesfully tested against or working with? Changelog.md is somewhat hidden.
travisghansen commented 3 years ago

Are you just using this for the first time? Or was it working before the updates I made yesterday and now stopped working?

  1. you may need to echo -n (this is likely your issue)
  2. What you’ve got seems fine
  3. That’s a good idea.

Just yesterday I took some time to make some updates and during that process realized that 2.5.2 wasn’t working. I fixed it up and it should be working against any version at least of 2.4 and 2.5 with any k8s version going back I’d guess a long ways (ie: anything someone is likely to still be running).

travisghansen commented 3 years ago

Also, make sure you’re running v0.5.1 and not v0.5.0…I had some bad (likely not harmful to pfSense) logic in there.

remkolems commented 3 years ago

Are you just using this for the first time? Or was it working before the updates I made yesterday and now stopped working?

  1. you may need to echo -n (this is likely your issue)
  2. What you’ve got seems fine
  3. That’s a good idea.

Just yesterday I took some time to make some updates and during that process realized that 2.5.2 wasn’t working. I fixed it up and it should be working against any version at least of 2.4 and 2.5 with any k8s version going back I’d guess a long ways (ie: anything someone is likely to still be running).

First time user. Although, I tried to run it day before yesterday with v0.40, couldn't get it to work due to pfSense version. First thing I did after your update today was installing the latest version 0.5.1.

Will try with -n notation. I will report back you asap. Thank you for the quick reply!

Also I had to adjust some ApiVersion as the yaml seems to be out of date regarding the latest status of Kubernetes apiversion.

rbac.yaml

from

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1

and 

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1

to

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1

and 

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
remkolems commented 3 years ago

Base64 encoded paswword was indeed slightly different with the -n notation. Last character is being replaced with =

However no succes yet, still CrashLoopBackOff with the following errror messages:

PHP Deprecated:  Required parameter $callback follows optional parameter $params in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-client-php/src/KubernetesClient/Client.php on line 170
PHP Warning:  Undefined array key "data" in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Store.php on line 128
PHP Deprecated:  Required parameter $callback follows optional parameter $params in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-client-php/src/KubernetesClient/Watch.php on line 137
2021-08-01T14:37:17+00:00 store successfully initialized
2021-08-01T14:37:17+00:00 controller config loaded/updated
2021-08-01T14:37:17+00:00 loading plugin metallb
PHP Warning:  Undefined array key "configMap" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php on line 30
2021-08-01T14:37:17+00:00 loading plugin haproxy-declarative
2021-08-01T14:37:17+00:00 loading plugin haproxy-ingress-proxy
2021-08-01T14:37:17+00:00 loading plugin pfsense-dns-services
PHP Warning:  Undefined array key "serviceLabelSelector" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSIngresses.php on line 32
2021-08-01T14:37:17+00:00 loading plugin pfsense-dns-ingresses
PHP Warning:  Undefined array key "serviceFieldSelector" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSIngresses.php on line 33
2021-08-01T14:37:17+00:00 loading plugin pfsense-dns-haproxy-ingress-proxy
PHP Warning:  Trying to access array offset on value of type null in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Store.php on line 218
2021-08-01T14:37:17+00:00 plugin (metallb): /v1/namespaces/metallb-system/ConfigMap/config ADDED - 135950
PHP Warning:  Trying to access array offset on value of type null in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Store.php on line 218
PHP Fatal error:  Uncaught TypeError: array_keys(): Argument #1 ($array) must be of type array, null given in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php:218
Stack trace:
#0 phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php(218): array_keys(NULL)
#1 phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php(117): KubernetesPfSenseController\Plugin\MetalLB->doActionGeneric()
#2 phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Plugin/AbstractPlugin.php(108): KubernetesPfSenseController\Plugin\MetalLB->doAction()
#3 phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Controller.php(532): KubernetesController\Plugin\AbstractPlugin->invokeAction()
#4 phar:///usr/local/bin/kubernetes-pfsense-controller/controller.php(87): KubernetesController\Controller->main()
#5 /usr/local/bin/kubernetes-pfsense-controller(2): include('phar:///usr/loc...')
#6 {main}
  thrown in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php on line 218
remkolems commented 3 years ago

Executed kubectl delete -f config.yaml -f deployment.yaml -f rbac.yaml -f secret.yaml

configmap "kubernetes-pfsense-controller-config" deleted
deployment.apps "kubernetes-pfsense-controller" deleted
serviceaccount "kubernetes-pfsense-controller" deleted
clusterrole.rbac.authorization.k8s.io "kubernetes-pfsense-controller" deleted
clusterrolebinding.rbac.authorization.k8s.io "kubernetes-pfsense-controller" deleted
secret "kubernetes-pfsense-controller" deleted

To start again with a fresh install via kubectl apply -f config.yaml -f deployment.yaml -f rbac.yaml -f secret.yaml

Keep you posted.

remkolems commented 3 years ago

Nope, still the same initial error followed with subsequent CrashLoopBackOff:

PHP Deprecated:  Required parameter $callback follows optional parameter $params in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-client-php/src/KubernetesClient/Client.php on line 170
PHP Warning:  Undefined array key "data" in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Store.php on line 128
PHP Deprecated:  Required parameter $callback follows optional parameter $params in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-client-php/src/KubernetesClient/Watch.php on line 137
2021-08-01T14:46:16+00:00 store successfully initialized
PHP Warning:  Undefined array key "configMap" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php on line 30
2021-08-01T14:46:16+00:00 controller config loaded/updated
2021-08-01T14:46:16+00:00 loading plugin metallb
2021-08-01T14:46:16+00:00 loading plugin haproxy-declarative
2021-08-01T14:46:16+00:00 loading plugin haproxy-ingress-proxy
2021-08-01T14:46:16+00:00 loading plugin pfsense-dns-services
2021-08-01T14:46:16+00:00 loading plugin pfsense-dns-ingresses
PHP Warning:  Undefined array key "serviceLabelSelector" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSIngresses.php on line 32
PHP Warning:  Undefined array key "serviceFieldSelector" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSIngresses.php on line 33
2021-08-01T14:46:16+00:00 loading plugin pfsense-dns-haproxy-ingress-proxy
PHP Warning:  Trying to access array offset on value of type null in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Store.php on line 218
2021-08-01T14:46:16+00:00 plugin (metallb): /v1/namespaces/metallb-system/ConfigMap/config ADDED - 135950
PHP Warning:  Trying to access array offset on value of type null in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Store.php on line 218
PHP Fatal error:  Uncaught TypeError: array_keys(): Argument #1 ($array) must be of type array, null given in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php:218
Stack trace:
#0 phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php(218): array_keys(NULL)
#1 phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php(117): KubernetesPfSenseController\Plugin\MetalLB->doActionGeneric()
#2 phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Plugin/AbstractPlugin.php(108): KubernetesPfSenseController\Plugin\MetalLB->doAction()
#3 phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Controller.php(532): KubernetesController\Plugin\AbstractPlugin->invokeAction()
#4 phar:///usr/local/bin/kubernetes-pfsense-controller/controller.php(87): KubernetesController\Controller->main()
#5 /usr/local/bin/kubernetes-pfsense-controller(2): include('phar:///usr/loc...')
#6 {main}
  thrown in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php on line 218
remkolems commented 3 years ago

Let me check this out first regarding PHP Fatal error: Uncaught TypeError: array_keys(): Argument #1 ($array) must be of type array, null given in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php:218

source code: line 218 == $managedNeighborNames = @array_keys($store[$pluginConfig['bgp-implementation']]['managed_neighbors']);

Seems to me that my configuration could be incorrect.

I'm all fresh ;-) to this stuff FRR-BGP + kubernetes.

Update: Installed a NGINX test.yaml and via k3s kubectl get rc,services I'm getting a correct external ip from ???. With another device in the same network via the external ip address I get a correct nginx site (Welcome to nginx).

Muhh... pfSense + frr/bgp is correctly setup so it seems.

travisghansen commented 3 years ago

Yeah I’ll fix it up. That’s the first release with php8 and it appears to be more stringent on some things. It will be a few hours but I’ll get a new release in a bit.

I’ll also update the static yaml to more recent versions!

remkolems commented 3 years ago

No hurry! This debugging is a nice change of pace! Thanks in advance.

travisghansen commented 3 years ago

Try v0.5.2. https://github.com/travisghansen/kubernetes-pfsense-controller/commit/c02fc67a045e98d09647dc4d46434681568f4ec1

remkolems commented 3 years ago

Restarted with new install and double checked version of container. Downloaded v0.5.2.

Some errors did shift, but unfortunately still a CrashLoopBackOff. See below.

Check Just a sanity and clarity check. On pfSense the openbgd package is not available anymore since v2.5.0. FRR instead is more than strongly advised. Within FRR BGP can be used or OSPF, OSPF6 and BFD. Could that be the cause of the issues?

I'm using the FRR -> BGP implementation.

config.yaml (part of what I'm using)

...
        #bgp-implementation: openbgp
        bgp-implementation: frr
        options:
          # openbgp:
          #   # pass through to config.xml
          #   template:
          #     md5sigkey:
          #     md5sigpass:
          #     groupname: metallb
          #     row:
          #       - parameters: announce all
          #         parmvalue:
          frr:
            template:
              peergroup: metallb-rancher
...
PHP Deprecated:  Required parameter $callback follows optional parameter $params in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-client-php/src/KubernetesClient/Client.php on line 170
PHP Deprecated:  Required parameter $callback follows optional parameter $params in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-client-php/src/KubernetesClient/Watch.php on line 137
2021-08-01T19:34:12+00:00 store successfully initialized
PHP Warning:  Undefined array key "configMap" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/MetalLB.php on line 30
2021-08-01T19:34:12+00:00 controller config loaded/updated
2021-08-01T19:34:12+00:00 loading plugin metallb
2021-08-01T19:34:12+00:00 loading plugin haproxy-declarative
2021-08-01T19:34:12+00:00 loading plugin haproxy-ingress-proxy
2021-08-01T19:34:12+00:00 loading plugin pfsense-dns-services
2021-08-01T19:34:12+00:00 loading plugin pfsense-dns-ingresses
PHP Warning:  Undefined array key "serviceLabelSelector" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSIngresses.php on line 32
PHP Warning:  Undefined array key "serviceFieldSelector" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSIngresses.php on line 33
2021-08-01T19:34:12+00:00 loading plugin pfsense-dns-haproxy-ingress-proxy
PHP Warning:  Undefined array key "haproxy-ingress-proxy" in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Store.php on line 218
2021-08-01T19:34:12+00:00 plugin (metallb): /v1/namespaces/metallb-system/ConfigMap/config ADDED - 135950
2021-08-01T19:34:13+00:00 plugin (metallb): successfully reloaded frr bgp service
2021-08-01T19:34:14+00:00 plugin (haproxy-declarative): successfully reloaded HAProxy service
PHP Warning:  Undefined array key "haproxy-ingress-proxy" in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Store.php on line 218
2021-08-01T19:34:14+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /v1/namespaces/kube-system/ConfigMap/kubernetes-pfsense-controller-store ADDED - 311808
PHP Warning:  Undefined array key "pfsense-dns-haproxy-ingress-proxy" in phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Store.php on line 218
PHP Fatal error:  Uncaught TypeError: key_exists(): Argument #2 ($array) must be of type array, null given in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSHAProxyIngressProxy.php:133
Stack trace:
#0 phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSHAProxyIngressProxy.php(133): key_exists('managed_hosts', NULL)
#1 phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Plugin/AbstractPlugin.php(108): KubernetesPfSenseController\Plugin\DNSHAProxyIngressProxy->doAction()
#2 phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Controller.php(532): KubernetesController\Plugin\AbstractPlugin->invokeAction()
#3 phar:///usr/local/bin/kubernetes-pfsense-controller/controller.php(87): KubernetesController\Controller->main()
#4 /usr/local/bin/kubernetes-pfsense-controller(2): include('phar:///usr/loc...')
#5 {main}
  thrown in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSHAProxyIngressProxy.php on line 133
travisghansen commented 3 years ago

I don't think there's anything wrong with how pfSense is setup at this point. These error are all related to php 8 being more stringent than php 7. I'll have another release shortly to clean this one up (the errors are simply a result of starting the first time every so I don't see them generally).

travisghansen commented 3 years ago

OK, let's try this again v0.5.3.

remkolems commented 3 years ago

Almost! Ik had it running for almost a minute. Ended up in an error followed by a crashloopbackoff.

k3s kubectl -n kube-system get pods

NAME                                             READY   STATUS    RESTARTS   AGE
calico-node-2l4k5                                1/1     Running   4          8d
metrics-server-86cbb8457f-mr9d7                  1/1     Running   4          8d
local-path-provisioner-5ff76fc89d-74t5f          1/1     Running   4          8d
coredns-7448499f4d-shqrf                         1/1     Running   4          8d
calico-kube-controllers-78d6f96c7b-jjh25         1/1     Running   4          8d
kubernetes-pfsense-controller-668c59c454-gqt8f   1/1     Running   2          58s
NAME                                             READY   STATUS    RESTARTS   AGE
calico-node-2l4k5                                1/1     Running   4          8d
metrics-server-86cbb8457f-mr9d7                  1/1     Running   4          8d
local-path-provisioner-5ff76fc89d-74t5f          1/1     Running   4          8d
coredns-7448499f4d-shqrf                         1/1     Running   4          8d
calico-kube-controllers-78d6f96c7b-jjh25         1/1     Running   4          8d
kubernetes-pfsense-controller-668c59c454-gqt8f   0/1     Error     2          66s

kubectl logs kubernetes-pfsense-controller-668c59c454-gqt8f -n kube-system

2021-08-01T20:34:37+00:00 store successfully initialized
2021-08-01T20:34:37+00:00 controller config loaded/updated
2021-08-01T20:34:37+00:00 loading plugin metallb
2021-08-01T20:34:37+00:00 loading plugin haproxy-declarative
2021-08-01T20:34:37+00:00 loading plugin haproxy-ingress-proxy
2021-08-01T20:34:37+00:00 loading plugin pfsense-dns-services
2021-08-01T20:34:37+00:00 loading plugin pfsense-dns-ingresses
2021-08-01T20:34:37+00:00 loading plugin pfsense-dns-haproxy-ingress-proxy
2021-08-01T20:34:37+00:00 plugin (metallb): /v1/namespaces/metallb-system/ConfigMap/config ADDED - 135950
2021-08-01T20:34:38+00:00 plugin (metallb): successfully reloaded frr bgp service
2021-08-01T20:34:39+00:00 plugin (haproxy-declarative): successfully reloaded HAProxy service
2021-08-01T20:34:40+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /v1/namespaces/kube-system/ConfigMap/kubernetes-pfsense-controller-store ADDED - 317779
PHP Warning:  Trying to access array offset on value of type null in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSHAProxyIngressProxy.php on line 141
2021-08-01T20:34:40+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded dnsmasq service
2021-08-01T20:34:43+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded unbound service
2021-08-01T20:34:45+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded DHCP service
PHP Fatal error:  Uncaught TypeError: array_keys(): Argument #1 ($array) must be of type array, null given in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/HAProxyIngressProxy.php:282
Stack trace:
#0 phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/HAProxyIngressProxy.php(282): array_keys(NULL)
#1 phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Plugin/AbstractPlugin.php(108): KubernetesPfSenseController\Plugin\HAProxyIngressProxy->doAction()
#2 phar:///usr/local/bin/kubernetes-pfsense-controller/vendor/travisghansen/kubernetes-controller-php/src/KubernetesController/Controller.php(532): KubernetesController\Plugin\AbstractPlugin->invokeAction()
#3 phar:///usr/local/bin/kubernetes-pfsense-controller/controller.php(87): KubernetesController\Controller->main()
#4 /usr/local/bin/kubernetes-pfsense-controller(2): include('phar:///usr/loc...')
#5 {main}
  thrown in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/HAProxyIngressProxy.php on line 282
travisghansen commented 3 years ago

OK, we're close! v0.5.4 when this build finishes: https://github.com/travisghansen/kubernetes-pfsense-controller/actions/runs/1088116364

remkolems commented 3 years ago

This is looking good! Pod is running like a charm without any restarts so far. Let me debug/test further today. Keep you posted. 

2021-08-02T09:47:41+00:00 store successfully initialized
2021-08-02T09:47:41+00:00 controller config loaded/updated
2021-08-02T09:47:41+00:00 loading plugin metallb
2021-08-02T09:47:41+00:00 loading plugin haproxy-declarative
2021-08-02T09:47:41+00:00 loading plugin haproxy-ingress-proxy
2021-08-02T09:47:41+00:00 loading plugin pfsense-dns-services
2021-08-02T09:47:41+00:00 loading plugin pfsense-dns-ingresses
2021-08-02T09:47:41+00:00 loading plugin pfsense-dns-haproxy-ingress-proxy
2021-08-02T09:47:41+00:00 plugin (metallb): /v1/namespaces/metallb-system/ConfigMap/config ADDED - 135950
2021-08-02T09:47:42+00:00 plugin (metallb): successfully reloaded frr bgp service
2021-08-02T09:47:43+00:00 plugin (haproxy-declarative): successfully reloaded HAProxy service
2021-08-02T09:47:44+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /v1/namespaces/kube-system/ConfigMap/kubernetes-pfsense-controller-store ADDED - 317779
PHP Warning:  Undefined array key "managed_frontends" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSHAProxyIngressProxy.php on line 145
2021-08-02T09:47:45+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded dnsmasq service
2021-08-02T09:47:48+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded unbound service
2021-08-02T09:47:49+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded DHCP service
PHP Warning:  Undefined array key "managed_hosts" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSResourceTrait.php on line 59
PHP Warning:  Undefined array key "hosts" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSResourceTrait.php on line 75
2021-08-02T09:47:52+00:00 plugin (pfsense-dns-services): successfully reloaded dnsmasq service
2021-08-02T09:47:55+00:00 plugin (pfsense-dns-services): successfully reloaded unbound service
2021-08-02T09:47:56+00:00 plugin (pfsense-dns-services): successfully reloaded DHCP service
PHP Warning:  Undefined array key "managed_hosts" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSResourceTrait.php on line 59
PHP Warning:  Undefined array key "hosts" in phar:///usr/local/bin/kubernetes-pfsense-controller/src/KubernetesPfSenseController/Plugin/DNSResourceTrait.php on line 75
2021-08-02T09:47:57+00:00 plugin (pfsense-dns-ingresses): successfully reloaded dnsmasq service
2021-08-02T09:48:00+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2021-08-02T09:48:01+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2021-08-02T09:48:01+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /v1/namespaces/kube-system/ConfigMap/kubernetes-pfsense-controller-store MODIFIED - 351413
2021-08-02T09:48:01+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /v1/namespaces/kube-system/ConfigMap/kubernetes-pfsense-controller-store MODIFIED - 351417
2021-08-02T09:48:06+00:00 plugin (haproxy-ingress-proxy): successfully reloaded HAProxy service
2021-08-02T09:48:07+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /v1/namespaces/kube-system/ConfigMap/kubernetes-pfsense-controller-store MODIFIED - 351423
2021-08-02T09:48:09+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded dnsmasq service
2021-08-02T09:48:12+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded unbound service
2021-08-02T09:48:13+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded DHCP service
travisghansen commented 3 years ago

Send over your config now for me to review. Enabling both dnsmasq and unbound is likely not the desired behavior.

In the meantime I’ll clean up those additional warnings in the output and snap a new release.

travisghansen commented 3 years ago

v0.5.5 snapped. Give it a try and see if those remaining warnings go away.

remkolems commented 3 years ago

Send over your config now for me to review. Enabling both dnsmasq and unbound is likely not the desired behavior.

In the meantime I’ll clean up those additional warnings in the output and snap a new release.

Here you go (something came in between today, so a bit later than expected):

config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-system
  name: kubernetes-pfsense-controller-config
data:
  config: |
    controller-id: "kpc-slm"
    enabled: true
    plugins:
      metallb:
        enabled: true
        nodeLabelSelector:
        nodeFieldSelector:
        #configMap: "metallb-system/config"
        # pick 1 implementation
        #bgp-implementation: openbgp
        bgp-implementation: frr
        options:
          # openbgp:
          #   # pass through to config.xml
          #   template:
          #     md5sigkey:
          #     md5sigpass:
          #     groupname: metallb
          #     row:
          #       - parameters: announce all
          #         parmvalue:
          frr:
            template:
              peergroup: metallb-rancher
      haproxy-declarative:
        enabled: true
      haproxy-ingress-proxy:
        enabled: true
        ingressLabelSelector:
        ingressFieldSelector:
        # works in conjunction with the ingress annotation 'haproxy-ingress-proxy.pfsense.org/enabled'
        # if defaultEnabled is empty or true, you can disable specific ingresses by setting the annotation to false
        # if defaultEnabled is false, you can enable specific ingresses by setting the annotation to true
        defaultEnabled: true
        defaultFrontend: slam-http-80
        defaultBackend: slam
        # by default anything is allowed
        #allowedHostRegex: "/.*/"
      pfsense-dns-services:
        enabled: true
        serviceLabelSelector:
        serviceFieldSelector:
        #allowedHostRegex: "/.*/"
        dnsBackends:
          dnsmasq:
            enabled: false
          unbound:
            enabled: true
      pfsense-dns-ingresses:
        enabled: true
        ingressLabelSelector:
        ingressFieldSelector:
        # works in conjunction with the ingress annotation 'dns.pfsense.org/enabled'
        # if defaultEnabled is empty or true, you can disable specific ingresses by setting the annotation to false
        # if defaultEnabled is false, you can enable specific ingresses by setting the annotation to true
        defaultEnabled: true
        #allowedHostRegex: "/.*/"
        dnsBackends:
          dnsmasq:
            enabled: false
          unbound:
            enabled: true
      pfsense-dns-haproxy-ingress-proxy:
        enabled: true
        #allowedHostRegex: "/.*/"
        dnsBackends:
          dnsmasq:
            enabled: false
          unbound:
            enabled: true
        frontends:
          slam-http-80:
            hostname: http-80.k3s
          primary_frontend_name2:
            hostname: slam.k3s

deployment.yaml Notice port 5443 due to the use SSL off-loading by HAProxy for various docker containers (I'm decommisioning them on short term when k3s is up and running). 

kind: Deployment
apiVersion: apps/v1
metadata:
  name: kubernetes-pfsense-controller
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-pfsense-controller
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: kubernetes-pfsense-controller
    spec:
      serviceAccountName: kubernetes-pfsense-controller
      containers:
        - name: kubernetes-pfsense-controller
          image: docker.io/travisghansen/kubernetes-pfsense-controller:latest
          env:
            - name: PFSENSE_URL
              value: "https://xxx.xxx.xxx.xxx:5443"
              # allow self-signed certs
            - name: PFSENSE_INSECURE
              value: "true"
            - name: PFSENSE_USERNAME
              value: "admin"
            - name: PFSENSE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: kubernetes-pfsense-controller
                  key: pfsense-password
remkolems commented 3 years ago

v0.5.5 snapped. Give it a try and see if those remaining warnings go away.

No warnings anymore! Output is based upon v0.5.5.

2021-08-02T20:03:25+00:00 store successfully initialized
2021-08-02T20:03:26+00:00 controller config loaded/updated
2021-08-02T20:03:26+00:00 loading plugin metallb
2021-08-02T20:03:26+00:00 loading plugin haproxy-declarative
2021-08-02T20:03:26+00:00 loading plugin haproxy-ingress-proxy
2021-08-02T20:03:26+00:00 loading plugin pfsense-dns-services
2021-08-02T20:03:26+00:00 loading plugin pfsense-dns-ingresses
2021-08-02T20:03:26+00:00 loading plugin pfsense-dns-haproxy-ingress-proxy
2021-08-02T20:03:26+00:00 plugin (metallb): /v1/namespaces/metallb-system/ConfigMap/config ADDED - 135950
2021-08-02T20:03:27+00:00 plugin (metallb): successfully reloaded frr bgp service
2021-08-02T20:03:28+00:00 plugin (haproxy-declarative): successfully reloaded HAProxy service
2021-08-02T20:03:28+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): /v1/namespaces/kube-system/ConfigMap/kubernetes-pfsense-controller-store ADDED - 351423
2021-08-02T20:03:31+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded unbound service
2021-08-02T20:03:32+00:00 plugin (pfsense-dns-haproxy-ingress-proxy): successfully reloaded DHCP service
2021-08-02T20:03:39+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2021-08-02T20:03:41+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2021-08-02T20:03:43+00:00 plugin (haproxy-ingress-proxy): successfully reloaded HAProxy service
2021-08-02T20:03:46+00:00 plugin (pfsense-dns-services): successfully reloaded unbound service
2021-08-02T20:03:47+00:00 plugin (pfsense-dns-services): successfully reloaded DHCP service
travisghansen commented 3 years ago

Pretty sure that output is not from the latest version of the image. Did you apply the deployment.yaml and go back to v0.5.2?

Do you have appropriate peers in your frr config etc now? Are things generally behaving as desired at this point?

remkolems commented 3 years ago

Wow you're right. Strong catch. I thought I installed the latest version.... v0.5.5. Indeed I hard copied apparently version v0.5.2 (wrongfully pushed to my test node). This has been corrected now to latest.

Command k3s kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |tr -s '[[:space:]]' '\n' |sort |uniq -c

docker.io/travisghansen/kubernetes-pfsense-controller:v0.5.2

Reinstalled everything

docker.io/travisghansen/kubernetes-pfsense-controller:latest

Note 1: I'll update both previous replies with the correct info (for future's sake)

Note 2: I installed the GitHub app on my phone just now, so that I can get push notifications instead of relying on email or page refreshes.

remkolems commented 3 years ago

Do you have appropriate peers in your frr config etc now? Are things generally behaving as desired at this point?

This is where my knowledge is still lacking or not up to par yet. Mainly a k8s thing. Experimenting with Rancher install and config at this point.

To the best of my understanding now and to what I was expecting up until now: things are behaving as desired.

For now this issue can be closed as the CrashLoopBackOff has been solved. Thank you!

travisghansen commented 3 years ago

That's great! Thanks for your patience working through all those fun errors. It's good to get the projected updated to run on php8 and work with 2.5.2.