search

openshift / oadp-operator

OADP Operator
Apache License 2.0
78 stars 71 forks source link

Podvolumerestore failed with EOF error #1161

Closed Malsmi closed 1 year ago

Malsmi commented 1 year ago

Describe issue

Installed the OADP Operator, 1.1.6, into OpenShift 4.12.22. The restic restore failed with error pod volume restore failed

Error

  Velero:  pod volume restore failed: error running restic restore, cmd=restic restore --repo=s3:http://minio-velero.apps.cpd473-baroffline11.cp.fyre.ibm.com/velero/cpdbackup/restic/zen --password-file=/tmp/credentials/openshift-adp/velero-restic-credentials-repository-password --cache-dir=/scratch/.cache/restic e30caf9d --target=. --delete --skip-unchanged, stdout=Skip Unchanged True
restoring <Snapshot e30caf9d of [/host_pods/637dcb34-1723-47ae-b8ab-3fdca7d3396c/volumes/kubernetes.io~nfs/pvc-72a209db-ea13-4596-b8ed-2b8caf25e9f9] at 2023-09-15 17:29:45.047755158 +0000 UTC by root@velero> to .
, stderr=Load(<data/fb78845427>, 16826112, 0) returned error, retrying after 552.330144ms: ReadFull: unexpected EOF
Load(<data/fb78845427>, 16826112, 0) returned error, retrying after 1.080381816s: ReadFull: unexpected EOF
Load(<data/fb78845427>, 16826112, 0) returned error, retrying after 1.31013006s: ReadFull: unexpected EOF
Load(<data/fb78845427>, 16826112, 0) returned error, retrying after 1.582392691s: ReadFull: unexpected EOF
Load(<data/fb78845427>, 16826112, 0) returned error, retrying after 2.340488664s: ReadFull: unexpected EOF
Load(<data/fb78845427>, 16826112, 0) returned error, retrying after 4.506218855s: ReadFull: unexpected EOF
Load(<data/fb78845427>, 16826112, 0) returned error, retrying after 3.221479586s: ReadFull: unexpected EOF
Load(<data/fb78845427>, 16826112, 0) returned error, retrying after 5.608623477s: ReadFull: unexpected EOF
Load(<data/fb78845427>, 16826112, 0) returned error, retrying after 7.649837917s: ReadFull: unexpected EOF
Load(<data/fb78845427>, 16826112, 0) returned error, retrying after 15.394871241s: ReadFull: unexpected EOF
ignoring error for /blumeta0/bigsql/c-bigsql-1694729814184242-db2u-0/version.txt: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/BIGSQL/T0000006/C0000000.LRG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000164.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000170.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/home/db2inst1/sqllib/spmlog/SPMLOGSD/SPM00002.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2uadm/.ssh/id_rsa: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000158.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000163.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000175.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000179.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000157.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2instusr/.ssh/config: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2uhausr/.ssh/id_rsa.pub: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/bigsql/security/.bigsql-scheduler-ssl-pass.pw: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/bigsql/security/bigsql-scheduler-truststore.pfx: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/BIGSQL/T0000004/C0000000.LRG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000150.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000151.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2instusr/.ssh/authorized_keys: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000172.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000177.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000160.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2uhausr/.ssh/config: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/bigsql/analyze/.sync: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000154.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000161.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000183.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2uadm/.ssh/config: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000156.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000165.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000180.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000189.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/bigsql/security/.bigsql-credential-provider.jceks.crc: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/bigsql/security/bigsql-metastore-truststore.jks: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/bigsql/security/bigsql-scheduler-certificate.cert: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000162.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000171.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000182.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000186.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000193.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/bigsql/security/.bigsql-metastore-ssl-pass.pw: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/BIGSQL/T0000000/C0000000.CAT: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000155.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000199.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/bigsql/security/bigsql-metastore-certificate.cert: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/bigsql/c-bigsql-1694729814184242-db2u-0/tls.crt: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000152.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000188.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2instusr/.ssh/id_rsa: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000166.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000173.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000178.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000181.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000191.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/BIGSQL/T0000001/C0000000.TMP/SQLTAG.NAM: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000184.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000167.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2uadm/.ssh/id_rsa.pub: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/BIGSQL/T0000007/C0000000.LRG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000153.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000169.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000185.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000187.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000194.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2instusr/.ssh/known_hosts: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/BIGSQL/T0000002/C0000000.UTM/SQLTAG.NAM: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/BIGSQL/T0000003/C0000000.LRG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/BIGSQL/T0000005/C0000000.LRG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000159.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000174.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000190.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000192.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/SystemConfig/global.reg: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2uhausr/.ssh/id_rsa: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000176.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000197.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/home/db2inst1/sqllib/spmlog/SPMLOGSD/SPM00001.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2instusr/.ssh/id_rsa.pub: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2uhausr/.ssh/authorized_keys: StreamPack: ReadFull: unexpected EOF
ignoring error for /bludata0/sshkeys/db2uadm/.ssh/authorized_keys: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000198.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000168.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000195.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/db2/databases/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/S0000196.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/home/db2inst1/sqllib/spmlog/SPMLOGSD/SPM00000.LOG: StreamPack: ReadFull: unexpected EOF
ignoring error for /blumeta0/bigsql/security/bigsql-credential-provider.jceks: StreamPack: ReadFull: unexpected EOF
Fatal: There were 86 errors

: exit status 1

OADP Version 1.1.6

OpenShift Version 4.12.22

Must-gather Log must-gather.tar (1).gz

How to resolve this EOF errors in restic?

weshayutin commented 1 year ago

Greetings, It looks like your restic repo is corrupted or restic is unable to download from the repo. Suggestion:

  1. Take a fresh backup
  2. Ensure the restic repo created for the backup is sane by running restic check
  3. Restore from backup.

Additional help: