Issues with persistence under Kubernetes

Hi, I'm trying to figure out whether docker-nginx-controller can work under k8s as well. I have a k8s 1.19.1 cluster and a private registry.

I've built the ubuntu/no-nap image using:

cd docker-nginx-controller/ubuntu/no-nap/
docker build --no-cache --build-arg STORE_UUID=True --build-arg CONTROLLER_URL=https://192.168.1.14/install/controller-agent --build-arg API_KEY='ada99c5d5176389c4413ead8110bc02d' -t registry.mynetwork.xyz/nginx-nonap-withagent:23 .

pushed it to the private registry and started it with:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginxplus-quota
  namespace: test
  labels:
    app: nginxplus-quota
spec:
  selector:
    matchLabels:
      app: nginxplus-quota
  replicas: 1
  template:
    metadata:
      labels:
        app: nginxplus-quota
    spec:
      containers:
      - name: nginxplus-quota
        image: registry.mynetwork.xyz/nginx-nonap-withagent:23
        env:
        - name: ENV_CONTROLLER_INSTANCE_NAME
          value: "nginxplus-quota"
        ports:
        - containerPort: 80
        - containerPort: 8080
        - containerPort: 8999
        [...]

apparently the STORE_UUID=True build argument and the ENV_CONTROLLER_INSTANCE_NAME environment variable in the deployment should enable persistence across restarts.

What I see in the NGINX Controller (release 3.13), instead, is:

Under infrastructure - Instances the instance name is correctly reported as "nginxplus-quota", but in the instance properties we have:

Hostname nginxplus-quota-7bf454896f-c7h2d
Version 1.19.5
Agent Version 3.13.0-2764907.release-3-13
Connected Yes
Hypervisor container

Operating System:
Name Ubuntu
Version 18.04

"hostname" is set to the actual pod name

everything runs fine. Deleting and restarting the deployment, though, seems to break things: the instance name in the controller doesn't get updated with the new hostname and the nginx instance to controller communication doesn't seem to work. Pod logs report:

time="Mar  6 2021 10:10:26.574" level="info" msg="Commencing streaming (id: 521)" feature="configurator"
time="Mar  6 2021 10:10:26.574" level="info" msg="Initialize streaming from endpoint: https://192.168.1.14:8443/1.4/ada99c5d5176389c4413ead8110bc02d/configs/stream/" feature="configurator"
time="Mar  6 2021 10:10:26.574" level="info" msg="Streaming HTTPEndpoint: https://192.168.1.14:8443/1.4/ada99c5d5176389c4413ead8110bc02d/configs/stream/" feature="configurator/streaming"
time="Mar  6 2021 10:10:26.590" level="info" msg="Streaming instance 521 ended" feature="configurator"
time="Mar  6 2021 10:10:26.590" level="info" msg="Streaming terminated remotely. attempting to restart" feature="configurator"

manually deleting the instance from the controller makes things work again, and pod logs report:

time="Mar  6 2021 10:11:36.688" level="info" msg="Commencing streaming (id: 528)" feature="configurator"
time="Mar  6 2021 10:11:36.688" level="info" msg="Initialize streaming from endpoint: https://192.168.1.14:8443/1.4/ada99c5d5176389c4413ead8110bc02d/configs/stream/" feature="configurator"
time="Mar  6 2021 10:11:36.688" level="info" msg="Streaming HTTPEndpoint: https://192.168.1.14:8443/1.4/ada99c5d5176389c4413ead8110bc02d/configs/stream/" feature="configurator/streaming"

Any clue about this? Should I use a StatefulSet instead of a Deployment? In such a case the pod name would change anyway based on the replica size.

Thanks.

nginxinc / docker-nginx-controller

Issues with persistence under Kubernetes #49