[uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask

[rolinux]

This is more of a cry for help than a bug with the image as I've spend the last few days trying to debug why the docker image worked in docker on my old host but after migration when trying to use Kubernetes and Volumes is not able to see the path or record anything.

I can see the live cameras, connect to the cloud and make see the old recordings but not able to record new ones.

The K8s namespace:

# kubectl get all -n unifi-video
NAME                              READY   STATUS    RESTARTS   AGE
pod/controller-79b6bbd79d-9hlzl   1/1     Running   0          36m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/unifi-video-svc   ClusterIP   10.43.94.182   <none>        80/TCP    64m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller   1/1     1            1           64m

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/controller-65545b6659   0         0         0       64m
replicaset.apps/controller-79b6bbd79d   1         1         1       36m

The folder inside the container:

# kubectl exec -it controller-79b6bbd79d-9hlzl -n unifi-video -- /bin/bash
root@controller-79b6bbd79d-9hlzl:/# ls -l /usr/lib/unifi-video/data/videos
total 0
drwxrwxr-x 3 unifi-video users 18 Jun 15 10:01 545fae7e-dc23-3a86-89a4-eaae4105ed09
drwxrwxr-x 3 unifi-video users 18 Jun 15 06:32 6cbf9a74-d507-369d-8e82-2074920e6748
drwxrwxr-x 3 unifi-video users 18 Jun 14 23:35 deeb1b31-1527-371c-881a-4ddf1c33fafa
root@controller-79b6bbd79d-9hlzl:/# 

There are 2 volumes in the deployment:

...
        volumeMounts:
        - name: data
          mountPath: /var/lib/unifi-video
        - name: videos
          mountPath: /usr/lib/unifi-video/data/videos
...
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: unifi-video-data-ssd-pvc
        - name: videos
          persistentVolumeClaim:
            claimName: unifi-video-videos-hdd-pvc
...

I can provide the full code for the StorageClass, PersistentVolume, PersistenVolumeClaim, Deployment and so on.

Unsure if the problem is both the volumes or only the path to record.

The logs:

1570545474.495 2019-10-08 15:37:54.495/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask
1570545414.495 2019-10-08 15:36:54.495/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask
1570545354.495 2019-10-08 15:35:54.495/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask
1570545294.495 2019-10-08 15:34:54.495/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask
1570545234.495 2019-10-08 15:33:54.495/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask
1570545174.491 2019-10-08 15:32:54.491/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask
1570545114.491 2019-10-08 15:31:54.491/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask
1570545054.491 2019-10-08 15:30:54.491/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask
1570544994.491 2019-10-08 15:29:54.491/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask
1570544933.891 2019-10-08 15:28:53.891/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask
1570544874.490 2019-10-08 15:27:54.490/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask
1570544814.490 2019-10-08 15:26:54.490/BST: ERROR  [uv.utils] Unable to get disk usage:/usr/lib/unifi-video/data/videos : No such file or directory in StatsUpdaterTask

The Deployment is:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller
  namespace: unifi-video
  labels:
    k8s-app: controller
spec:
  #replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: controller
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/controller: unconfined
      labels:
        k8s-app: controller
    spec:
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        beta.kubernetes.io/os: linux
      containers:
      - name: controller
        image: pducharme/unifi-video-controller:3.10.8.3.6
        imagePullPolicy: IfNotPresent
        env:
          - name: TZ
            value: "Europe/London"
          - name: PUID
            value: "99"
          - name: PGID
            value: "100"
          - name: DEBUG
            value: "1"
        resources:
          limits:
            cpu: "1"
            memory: "2Gi"
          requests:
            cpu: "100m"
            memory: "512Mi"
        volumeMounts:
        - name: data
          mountPath: /var/lib/unifi-video
        - name: videos
          mountPath: /usr/lib/unifi-video/data/videos
        ports:
        - containerPort: 6666
          hostPort: 6666
          hostIP: 192.168.3.9
          name: inbound-streams
          protocol: TCP
        - containerPort: 7442
          hostPort: 7442
          hostIP: 192.168.3.9
          name: camera-mgmt
          protocol: TCP
        - containerPort: 7080
          port: 7080
          name: client-nvr
          protocol: TCP
        readinessProbe:
          tcpSocket:
            port: client-nvr
          initialDelaySeconds: 20
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: client-nvr
          initialDelaySeconds: 40
          periodSeconds: 20
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - SYS_ADMIN
            - DAC_READ_SEARCH
      dnsPolicy: Default
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: unifi-video-data-ssd-pvc
        - name: videos
          persistentVolumeClaim:
            claimName: unifi-video-videos-hdd-pvc

The 192.168.3.9 is for the VLAN where the cameras run.

Thank you

[rolinux]

Also in Tools -> Database -> Recording DB Analysis triggers Unexpected Error The Server responded with a status code of 400.

[fryfrog]

So instead of passing in a path, you have a volume and pass that in? How does the volume get passed around the cluster?

[rolinux]

It is a single host cluster. I tried with direct volumes and still the same. Even using docker directly (outside the kubernetest) I get the same 'Unexpected Error The Server responded with a status code of 400.'

[fryfrog]

Is it possible this is just somehow the weird :3.10.8 issues? Can you try on :3.10.6?

[rolinux]

Ended up running in docker (not kubernetes) with image 3.10.4 (as on the original host). Had to reset 2 cameras and downgrade firmware to get it working.

I spent too much time to try again :(

[floreseken]

@rolinux, did you ever got this working? I'm running into exactly the same problem..

[ghost]

@rolinux, did you ever got this working? I'm running into exactly the same problem..

Did you manage to solve the issue? I'm having the same issue.

[floreseken]

Nope. Ended up buying the Cloudkey Gen2

[ootagoo]

@rolinux, did you ever got this working? I'm running into exactly the same problem..

Did you manage to solve the issue? I'm having the same issue.

Did YOU ever end up figuring this out? I'm also stuck here...