search

k8up-io / k8up

Kubernetes and OpenShift Backup Operator
https://k8up.io/
Apache License 2.0
681 stars 66 forks source link

Nullpointer Exception on Backup #857

Closed sashokbg closed 1 year ago

sashokbg commented 1 year ago

Description

When trying to run a manual backup (see yaml bellow) I get a strange null pointer exception.

Additional Context

✗ kubectl get nodes
NAME    STATUS   ROLES                         AGE     VERSION
node1   Ready    control-plane,master,worker   7d19h   v1.23.7
node2   Ready    worker                        7d19h   v1.23.7
node3   Ready    worker                        7d19h   v1.23.7

Logs

2023-05-21T11:49:11Z    INFO    k8up.restic.restic.backup   starting backup
2023-05-21T11:49:11Z    INFO    k8up.restic.restic.backup   starting backup for folder  {"foldername": "data-nextcloud"}
2023-05-21T11:49:11Z    INFO    k8up.restic.restic.backup.command   restic command  {"path": "/usr/local/bin/restic", "args": ["backup", "--option", "", "--json", "--host", "home-cloud", "/data/data-nextcloud", "--tag", "manual"]}
2023-05-21T11:49:11Z    INFO    k8up.restic.restic.backup.command   Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}
2023-05-21T11:49:16Z    INFO    k8up.restic.restic.backup.progress  progress of backup  {"percentage": "0.00%"}
2023-05-21T11:49:23Z    INFO    k8up.restic.restic.backup.progress  backup finished {"new files": 0, "changed files": 0, "errors": 0}
2023-05-21T11:49:23Z    INFO    k8up.restic.restic.backup.progress  stats   {"time": 10.738905407, "bytes added": 353, "bytes processed": 4063429439}
2023-05-21T11:49:23Z    INFO    k8up.restic.statsHandler.promStats  sending prometheus stats    {"url": "http://127.0.0.1/"}
2023-05-21T11:49:23Z    ERROR   k8up.restic.restic  prometheus send failed  {"error": "Post \"http://127.0.0.1/metrics/job/restic_backup/instance/home-cloud\": dial tcp 127.0.0.1:80: connect: connection refused"}
github.com/k8up-io/k8up/v2/restic/cli.(*Restic).sendBackupStats
    /home/runner/work/k8up/k8up/restic/cli/backup.go:95
github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out
    /home/runner/work/k8up/k8up/restic/logging/logging.go:162
github.com/k8up-io/k8up/v2/restic/logging.writer.Write
    /home/runner/work/k8up/k8up/restic/logging/logging.go:103
io.copyBuffer
    /opt/hostedtoolcache/go/1.19.9/x64/src/io/io.go:429
io.Copy
    /opt/hostedtoolcache/go/1.19.9/x64/src/io/io.go:386
os/exec.(*Cmd).writerDescriptor.func1
    /opt/hostedtoolcache/go/1.19.9/x64/src/os/exec/exec.go:407
os/exec.(*Cmd).Start.func1
    /opt/hostedtoolcache/go/1.19.9/x64/src/os/exec/exec.go:544
2023-05-21T11:49:23Z    INFO    k8up.restic.restic.backup   backup finished, sending snapshot list
2023-05-21T11:49:23Z    INFO    k8up.restic.restic.snapshots    getting list of snapshots
2023-05-21T11:49:23Z    INFO    k8up.restic.restic.snapshots.command    restic command  {"path": "/usr/local/bin/restic", "args": ["snapshots", "--option", "", "--json"]}
2023-05-21T11:49:23Z    INFO    k8up.restic.restic.snapshots.command    Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x141887e]

goroutine 1 [running]:
github.com/k8up-io/k8up/v2/restic/kubernetes.filterByRepo(...)
    /home/runner/work/k8up/k8up/restic/kubernetes/snapshots.go:123
github.com/k8up-io/k8up/v2/restic/kubernetes.SyncSnapshotList({0x1a42388, 0xc00060cf80}, {0xc0004e5500, 0x87, 0x0?}, {0xc000044009, 0xa}, {0xc000048012, 0x29})
    /home/runner/work/k8up/k8up/restic/kubernetes/snapshots.go:29 +0x35e
github.com/k8up-io/k8up/v2/restic/cli.(*Restic).sendSnapshotList(0xc0002ac080)
    /home/runner/work/k8up/k8up/restic/cli/backup.go:116 +0x19b
github.com/k8up-io/k8up/v2/restic/cli.(*Restic).Backup(0xc0002ac080, {0x17bcee5, 0x5}, {0xc000173b90, 0x1, 0x1})
    /home/runner/work/k8up/k8up/restic/cli/backup.go:46 +0x2da
github.com/k8up-io/k8up/v2/cmd/restic.doBackup({0x1a423c0?, 0xc0000c8000?}, 0xc000380810?, {{0x1a44e50?, 0xc000380810?}, 0xc000380900?})
    /home/runner/work/k8up/k8up/cmd/restic/main.go:234 +0x105
github.com/k8up-io/k8up/v2/cmd/restic.run({0x1a423c0, 0xc0000c8000}, 0x1a44e50?, {{0x1a44e50?, 0xc000380810?}, 0x1a30ec8?})
    /home/runner/work/k8up/k8up/cmd/restic/main.go:129 +0xbf
github.com/k8up-io/k8up/v2/cmd/restic.resticMain(0xc00060c440)
    /home/runner/work/k8up/k8up/cmd/restic/main.go:113 +0x2a7
github.com/urfave/cli/v2.(*Command).Run(0x2598960, 0xc00060c440, {0xc000380750, 0x3, 0x3})
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.23.7/command.go:271 +0xa42
github.com/urfave/cli/v2.(*Command).Run(0xc00038c640, 0xc0003b0e80, {0xc0000c6040, 0x4, 0x4})
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.23.7/command.go:264 +0xc97
github.com/urfave/cli/v2.(*App).RunContext(0xc000388000, {0x1a423c0?, 0xc0000c8000}, {0xc0000c6040, 0x4, 0x4})
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.23.7/app.go:333 +0x665
github.com/urfave/cli/v2.(*App).Run(...)
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.23.7/app.go:310
main.main()
    /home/runner/work/k8up/k8up/cmd/k8up/main.go:30 +0x45

Expected Behavior

Backup should work

Steps To Reproduce

apiVersion: k8up.io/v1
kind: Backup
metadata:
  name: backup-after-server-change
spec:
  tags:
    - manual
  promURL: ""
  podSecurityContext:
    runAsUser: 0
    runAsGroup: 0
  backend:
    envFrom:
      - secretRef:
          name: "backup-secret"
    repoPasswordSecretRef:
      name: "restic-repository-password"
      key: "password"
    swift:
      path: /archives
      container: home-cloud-backup-storage
  failedJobsHistoryLimit: 2
  successfulJobsHistoryLimit: 2

Version of K8up

v4.2.2

Version of Kubernetes

v1.23.7

Distribution of Kubernetes

Bare metal

Kidswiss commented 1 year ago

Hi @sashokbg

Do you have any snapshot objects in that namespace? kubectl get snapshots? If you do, could you please post the yaml output of one of those?

It looks to me that a field is not correctly populated in the snapshot object when using the swift backend.

sashokbg commented 1 year ago

Hello @Kidswiss thank you for your message here are some yamls from the snapshots:

apiVersion: k8up.io/v1
kind: Snapshot
metadata:
  creationTimestamp: "2023-05-14T02:00:26Z"
  generation: 1
  name: f4856c6f
  namespace: home-cloud
  resourceVersion: "85254"
  uid: cc1e2de6-35e1-4dcd-934c-e36b06914cb3
spec:
  date: "2023-03-27T02:00:20Z"
  id: f4856c6fe842d6421187f9368171089fde4316bcf5c7a9053cceba10f192f339
  paths:
  - /home-cloud-postgresql.sql
apiVersion: k8up.io/v1
kind: Snapshot
metadata:
  creationTimestamp: "2023-05-14T02:00:28Z"
  generation: 1
  name: f0cd4ef4
  namespace: home-cloud
  resourceVersion: "85267"
  uid: d5a29422-c0b5-4d79-b4e4-a3f573592155
spec:
  date: "2023-04-18T02:00:12Z"
  id: f0cd4ef4c1f4d26b0eb823310303c2357ef14e7dbaf0f714b18b26abb2603ae0
  paths:
  - /data/data-nextcloud
sashokbg commented 1 year ago

Hello @Kidswiss I have worked on adding the swift support via EnvFrom variables in #657 can you please give me some directions where the NPE might be comming from ? I am not well versed in GO unfortunately :(

Kidswiss commented 1 year ago

Hi @sashokbg

Sorry was a bit busy...

I just checked your snapshots and I see that the field spec.repository is missing. That's what's causing the nilpointer.

It should take the value of RESTIC_REPOSITORY in the backup pod to populate this field. Could you please post a yaml of the backup pod? I'd like to check if all environment variables are available as expected.

EDIT: it might start to work if you use envFrom to inject following envVar:

RESTIC_REPOSITORY=swift:home-cloud-backup-storage/archives

Then you delete the existing snapshot objects in the namespace and trigger a backup again.

sashokbg commented 1 year ago

Hello @Kidswiss

Good news your suggestion worked perfectly !!

I checked the snapshots and saw they did not have the repository field that was added recently so I checked my CRDs and found out I forgot to update them (silly me).

I updated the CRDs and then deleted all snapshots as you suggested and everything worked.

Thank you very much for your kind help