search

k8up-io / k8up

Kubernetes and OpenShift Backup Operator
https://k8up.io/
Apache License 2.0
636 stars 63 forks source link

K8up is unable to restore a single file snapshot #803

Open bfeher-kx opened 1 year ago

bfeher-kx commented 1 year ago

Description

I've created an instance from ghcr.io/k8up-io/k8up:v2 and set the followings:

  - args:
    - -restore
    - -restoreSnap
    - 207d6a74
    - -restoreType
    - folder

The Snapshot id points to a single .sql file which has been created via K8up application aware backup from a PostgreSQL instance. It fails with with the following error: panic: runtime error: slice bounds out of range [:3] with capacity 2.

The snapshot above can be restored manually via the Restic binary packed in the same image.

Additional Context

My goal was to restore to a RWO volume thus used podAffinity spec to get my pod scheduled beside the target pod.

Tried with the operator as well, pointing to a RWX volume, it fails too.

Logs

1.6745719441801126e+09    INFO    k8up.restic    initializing                                                                                                                                  
1.674571944180123e+09    INFO    k8up.restic    setting up a signal handler                                                                                                                    
1.6745719441801894e+09    INFO    k8up.restic.restic    using the following restic options    {"options": [""]}                                                                                
1.6745719441802113e+09    INFO    k8up.restic.restic.RepoInit.command    restic command    {"path": "/usr/local/bin/restic", "args": ["init", "--option", ""]}                                 
1.6745719441802182e+09    INFO    k8up.restic.restic.RepoInit.command    Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}                                                   
1.6745719442500741e+09    INFO    k8up.restic.restic.unlock    unlocking repository    {"all": false}                                                                                          
1.6745719442501287e+09    INFO    k8up.restic.restic.unlock.command    restic command    {"path": "/usr/local/bin/restic", "args": ["unlock", "--option", ""]}                                 
1.6745719442501488e+09    INFO    k8up.restic.restic.unlock.command    Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}                                                     
1.6745719448515744e+09    INFO    k8up.restic.restic.unlock.restic.stderr    successfully removed locks                                                                                        
1.6745719448527825e+09    INFO    k8up.restic.restic.snapshots    getting list of snapshots                                                                                                    
1.6745719448528092e+09    INFO    k8up.restic.restic.snapshots.command    restic command    {"path": "/usr/local/bin/restic", "args": ["snapshots", "--option", "", "--json"]}                 
1.6745719448528461e+09    INFO    k8up.restic.restic.snapshots.command    Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}                                                  
1.674571945789223e+09    INFO    k8up.restic.restic.restore    restore initialised                                                                                                             
1.674571945789251e+09    INFO    k8up.restic.restic.restore    loading all snapshots from repositoy                                                                                            
1.6745719457892542e+09    INFO    k8up.restic.restic.snapshots    getting list of snapshots                                                                                                    
1.6745719457892606e+09    INFO    k8up.restic.restic.snapshots.command    restic command    {"path": "/usr/local/bin/restic", "args": ["snapshots", "--option", "", "--json"]}                 
1.674571945789275e+09    INFO    k8up.restic.restic.snapshots.command    Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}                                                   
panic: runtime error: slice bounds out of range [:3] with capacity 2                                                                                                                           

goroutine 1 [running]:                                                                                                                                                                         
github.com/k8up-io/k8up/v2/restic/cli.(*Restic).linkRestorePaths(0x14a05f9?, {{0xc000054700, 0x40}, {0x230ca086, 0xedb60a0fa, 0x0}, {0xc0000546c0, 0x40}, {0xc000698380, 0x1, ...}, ...}, ...) 
    /home/runner/work/k8up/k8up/restic/cli/restore.go:200 +0x265                                                                                                                               
github.com/k8up-io/k8up/v2/restic/cli.(*Restic).folderRestore(0xc000056c80, {0xc00004806c, 0x8}, {{0xc000054700, 0x40}, {0x230ca086, 0xedb60a0fa, 0x0}, {0xc0000546c0, 0x40}, ...}, ...)       
    /home/runner/work/k8up/k8up/restic/cli/restore.go:144 +0xf8                                                                                                                                
github.com/k8up-io/k8up/v2/restic/cli.(*Restic).Restore(0xc000056c80, {0x7fff01c76681, 0x8}, {{0x7fff01c76697, 0x6}, {0xc00004806c, 0x8}, {0x0, 0x0}, 0x0, ...}, ...)                          
    /home/runner/work/k8up/k8up/restic/cli/restore.go:89 +0x4aa                                                                                                                                
github.com/k8up-io/k8up/v2/cmd/restic.doRestore(0x1?)                                                                                                                                          
    /home/runner/work/k8up/k8up/cmd/restic/main.go:189 +0x1c7                                                                                                                                  
github.com/k8up-io/k8up/v2/cmd/restic.doNonBackupTasks(0x0?)                                                                                                                                   
    /home/runner/work/k8up/k8up/cmd/restic/main.go:159 +0x45                                                                                                                                   
github.com/k8up-io/k8up/v2/cmd/restic.run({0x1a56560, 0xc00004c030}, 0x1a58f50?, {{0x1a58f50?, 0xc000200c60?}, 0x1a452b0?})                                                                    
    /home/runner/work/k8up/k8up/cmd/restic/main.go:118 +0x92                                                                                                                                   
github.com/k8up-io/k8up/v2/cmd/restic.resticMain(0xc0006991c0)                                                                                                                                 
    /home/runner/work/k8up/k8up/cmd/restic/main.go:105 +0x25e                                                                                                                                  
github.com/urfave/cli/v2.(*Command).Run(0x25c6f40, 0xc0006991c0, {0xc0000a9500, 0x6, 0x6})                                                                                                     
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.20.2/command.go:271 +0xa8f                                                                                                             
github.com/urfave/cli/v2.(*Command).Run(0xc00031e140, 0xc000698b80, {0xc00003e070, 0x7, 0x7})                                                                                                  
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.20.2/command.go:264 +0xced                                                                                                             
github.com/urfave/cli/v2.(*App).RunContext(0xc0001e88c0, {0x1a56560?, 0xc00004c030}, {0xc00003e070, 0x7, 0x7})                                                                                 
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.20.2/app.go:305 +0x658                                                                                                                 
github.com/urfave/cli/v2.(*App).Run(...)                                                                                                                                                       
    /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.20.2/app.go:282                                                                                                                        
main.main()                                                                                                                                                                                    
    /home/runner/work/k8up/k8up/cmd/k8up/main.go:28 +0x45

Expected Behavior

K8up should be able to restore a single file snapshot.

Steps To Reproduce

spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
         matchExpressions:
         - key: statefulset.kubernetes.io/pod-name
           operator: In
           values:
            - podname
        topologyKey: kubernetes.io/hostname
  containers:
  - args:
    - -restore
    - -restoreSnap
    - 207d6a74
    - -restoreType
    - folder
    command:
    - /usr/local/bin/k8up
    - restic

Version of K8up

v2.5

Version of Kubernetes

v1.24.6

Distribution of Kubernetes

AKS

Kidswiss commented 1 year ago

Hi @bfeher-kx

Thanks for reporting this! I thought we're already tracking that, but was not able to find an issue for that.

bfeher-kx commented 1 year ago

Hi @Kidswiss

We have checked the code and it seems its expecting 3 sections for the path. In this case the last section is missing as we don't have a directory.

Kidswiss commented 1 year ago

hi @bfeher-kx

Yes that is correct, the current restore code to PVC expects that the backup also came from a PVC and not from a pre-backup command. So it's a use-case that has not yet been implemented as such.

The main issue is, that in most cases a dump file in a PVC isn't of much use to someone. They then still have to get the dump out of the PVC and into the DB instance somehow. But it would make for a better user-experience than just crashing.

There's the idea of restoreCommands: https://github.com/k8up-io/k8up/issues/637

Kidswiss commented 1 year ago

BTW: just found in the docs that we have a note about this case: https://k8up.io/k8up/2.5/how-tos/restore.html#_restore_from_s3_to_pvc

I knew we had written down something about this exact issue.

bfeher-kx commented 1 year ago

Hi @Kidswiss

Does that mean we should not wait for a fix? Is there a reason why its done like that?

Kidswiss commented 1 year ago

Hi @bfeher-kx

I think to implement a fix would be reasonable.

As to your second question, basically what I wrote above:

The main issue is, that in most cases a dump file in a PVC isn't of much use to someone. They then still have to get the dump out of the PVC and into the DB instance somehow.

That's why we have the descriptions how to do manual restores.

bfeher-kx commented 1 year ago

HI @Kidswiss

Can this be expected to be fixed? If not, please let me know and we will find a workaround.