Open veerendra2 opened 1 week ago
Please share the velero log bundle from the time when the maintenance jobs were created to the latest.
velero-server logs -> https://gist.github.com/veerendra2/a25ba2e55070230af7cf791350f327f8
@veerendra2 We need the entire log bundle, you can get it by velero debug
command.
I see there is one running job backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859377kblkt
lasted 10h, is that expected? If not, what happened to that job?
I think that job blocked deletion of other jobs
@veerendra2 Please help to collect the log for job backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859377kblkt and we need to check what happened to the job and why it kept running for 10h
Moreover, you need to keep the Velero server and client in the same version, otherwise, you may see many weird issues due to failures on resource operations since the CRD version mismatch.
As we can see here, the version info seems not correct:
Velero version (use velero version): Server v1.13.2
Therefore, if the version mismatch, please retry it after fixing the version problem.
@Lyndon-Li Sorry, I lost notifications. The velero-server is upgraded and logs are gone. This velero debug
never worked for me
$ k get deployment -n velero
NAME READY UP-TO-DATE AVAILABLE AGE
velero 1/1 1 1 357d
$ velero debug
An error occurred: velero deployment does not exist in namespace: velero
I will remove --keep-latest-maintenance-jobs
in our dev cluster today and keep it for this weekend. On next Monday, I will share the logs.
@Lyndon-Li Here is the info
I see there is one running job backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859377kblkt lasted 10h, is that expected?
No.(But, as I mentioned above, logs are gone. I'm attaching new logs below, please check)
Pods
$ k get pods
NAME READY STATUS RESTARTS AGE
analytics-backup-dev-de1-kopia-2vtqt-maintain-job-1719209077nwg 0/1 Completed 0 137m
analytics-backup-dev-de1-kopia-2vtqt-maintain-job-1719212984k8q 0/1 Completed 0 73m
analytics-backup-dev-de1-kopia-2vtqt-maintain-job-17192167pfgkc 0/1 Completed 0 8m53s
auditlog-backup-dev-de1-kopia-5xq8k-maintain-job-171920721bpp86 0/1 Completed 0 168m
auditlog-backup-dev-de1-kopia-5xq8k-maintain-job-171921107xnktn 0/1 Completed 0 103m
auditlog-backup-dev-de1-kopia-5xq8k-maintain-job-171921471495jt 0/1 Completed 0 43m
auth-backup-dev-de1-kopia-vg4wb-maintain-job-1719208970921z4f55 0/1 Completed 0 139m
auth-backup-dev-de1-kopia-vg4wb-maintain-job-1719212584309vms29 0/1 Completed 0 78m
auth-backup-dev-de1-kopia-vg4wb-maintain-job-1719216477608b6cpt 0/1 Completed 0 13m
backend-backup-dev-de1-kopia-ztv8f-maintain-job-1719205973ss2qj 0/1 Completed 0 3h8m
backend-backup-dev-de1-kopia-ztv8f-maintain-job-17192098702w8tz 0/1 Completed 0 124m
backend-backup-dev-de1-kopia-ztv8f-maintain-job-1719213473pf596 0/1 Completed 0 63m
backstage-backup-dev-de1-kopia-hvj2t-maintain-job-17192059ngb8p 0/1 Completed 0 3h8m
backstage-backup-dev-de1-kopia-hvj2t-maintain-job-17192098fgb6c 0/1 Completed 0 123m
backstage-backup-dev-de1-kopia-hvj2t-maintain-job-1719213428s8k 0/1 Completed 0 63m
dagster-backup-dev-de1-kopia-92dmg-maintain-job-1719207197r6vrk 0/1 Completed 0 168m
dagster-backup-dev-de1-kopia-92dmg-maintain-job-17192108099qqhd 0/1 Completed 0 108m
dagster-backup-dev-de1-kopia-92dmg-maintain-job-1719214712msr7h 0/1 Completed 0 43m
email-backup-dev-de1-kopia-ffzw7-maintain-job-171920598522wf8gb 0/1 Completed 0 3h8m
email-backup-dev-de1-kopia-ffzw7-maintain-job-171920988447nk7hx 0/1 Completed 0 123m
email-backup-dev-de1-kopia-ffzw7-maintain-job-171921348742xxhnw 0/1 Completed 0 63m
logging-backup-dev-de1-kopia-66qfg-maintain-job-1719207770g9dg2 0/1 Completed 0 159m
logging-backup-dev-de1-kopia-66qfg-maintain-job-1719211377nm96w 0/1 Completed 0 98m
logging-backup-dev-de1-kopia-66qfg-maintain-job-1719215270g5nvw 0/1 Completed 0 34m
mediamgmt-backup-dev-de1-kopia-sktdp-maintain-job-17192059rdf9j 0/1 Completed 0 3h8m
mediamgmt-backup-dev-de1-kopia-sktdp-maintain-job-17192098gz684 0/1 Completed 0 123m
mediamgmt-backup-dev-de1-kopia-sktdp-maintain-job-17192134k7pcd 0/1 Completed 0 63m
node-agent-256mw 1/1 Running 0 2d22h
node-agent-2whq8 1/1 Running 0 3d20h
node-agent-5dxdq 1/1 Running 0 3d20h
node-agent-6grrp 1/1 Running 0 3d20h
node-agent-7mjg6 1/1 Running 0 3d20h
node-agent-bv8qv 1/1 Running 0 3d20h
node-agent-hz4bw 1/1 Running 0 3d20h
node-agent-kt7cr 1/1 Running 0 3d20h
node-agent-kz8l9 1/1 Running 0 3d20h
node-agent-mctdc 1/1 Running 0 3d20h
node-agent-mmbmb 1/1 Running 0 3d20h
node-agent-n2v96 1/1 Running 0 3d20h
node-agent-q7n6s 1/1 Running 0 3d20h
node-agent-t662b 1/1 Running 0 3d20h
node-agent-xbt6b 1/1 Running 0 3d20h
pacman-backup-dev-de1-kopia-4srqc-maintain-job-17192074794lkckk 0/1 Completed 0 163m
pacman-backup-dev-de1-kopia-4srqc-maintain-job-17192113912h7svd 0/1 Completed 0 98m
pacman-backup-dev-de1-kopia-4srqc-maintain-job-17192152857g7ftt 0/1 Completed 0 33m
pacman-backup-dev-de1-restic-q8mmf-maintain-job-1718877154qwr47 0/1 Completed 0 3d22h
pact-backup-dev-de1-kopia-g4xcz-maintain-job-1719207485450rsc7h 0/1 Completed 0 163m
pact-backup-dev-de1-kopia-g4xcz-maintain-job-1719211370929nc6zw 0/1 Completed 0 99m
pact-backup-dev-de1-kopia-g4xcz-maintain-job-1719214972980rt77w 0/1 Completed 0 38m
poodle-backup-dev-de1-kopia-bg5vv-maintain-job-17192072064j59qd 0/1 Completed 0 168m
poodle-backup-dev-de1-kopia-bg5vv-maintain-job-1719210822422lph 0/1 Completed 0 108m
poodle-backup-dev-de1-kopia-bg5vv-maintain-job-17192147061x5vbz 0/1 Completed 0 43m
postman-backup-dev-de1-kopia-4f7v6-maintain-job-1719206886n974t 0/1 Completed 0 173m
postman-backup-dev-de1-kopia-4f7v6-maintain-job-1719210770hb97v 0/1 Completed 0 109m
postman-backup-dev-de1-kopia-4f7v6-maintain-job-1719214373zj4nh 0/1 Completed 0 48m
profiling-backup-dev-de1-kopia-mqqx9-maintain-job-17192060nctd8 0/1 Completed 0 3h8m
profiling-backup-dev-de1-kopia-mqqx9-maintain-job-17192099gmpb4 0/1 Completed 0 123m
profiling-backup-dev-de1-kopia-mqqx9-maintain-job-17192137wzkdn 0/1 Completed 0 59m
repo-maintain-job-1719206270915-2vft4 0/1 Completed 0 3h4m
repo-maintain-job-1719206279558-kpr6m 0/1 Completed 0 3h3m
repo-maintain-job-1719206285243-qgn24 0/1 Completed 0 3h3m
repo-maintain-job-1719206873465-h72vf 0/1 Completed 0 173m
repo-maintain-job-1719206880264-bn649 0/1 Completed 0 173m
repo-maintain-job-1719206892334-k4lzz 0/1 Completed 0 173m
repo-maintain-job-1719207173175-gm87k 0/1 Completed 0 168m
repo-maintain-job-1719207179310-nrj4f 0/1 Completed 0 168m
repo-maintain-job-1719207185326-t4c78 0/1 Completed 0 168m
repo-maintain-job-1719207191363-9gjvm 0/1 Completed 0 168m
repo-maintain-job-1719207470917-qk2v2 0/1 Completed 0 164m
repo-maintain-job-1719207777445-r59sg 0/1 Completed 0 158m
repo-maintain-job-1719208073179-sjzpr 0/1 Completed 0 153m
repo-maintain-job-1719208370920-9f8rb 0/1 Completed 0 149m
repo-maintain-job-1719208672868-m45l7 0/1 Completed 0 143m
repo-maintain-job-1719208977856-tp5r6 0/1 Completed 0 138m
repo-maintain-job-1719208983888-dhc7x 0/1 Completed 0 138m
repo-maintain-job-1719208989916-g2ngg 0/1 Completed 0 138m
repo-maintain-job-1719209004987-ldzvg 0/1 Completed 0 138m
repo-maintain-job-1719209011005-mv9p9 0/1 Completed 0 138m
repo-maintain-job-1719209022060-gf6tx 0/1 Completed 0 138m
repo-maintain-job-1719209028072-ddsws 0/1 Completed 0 138m
repo-maintain-job-1719209042143-f2phj 0/1 Completed 0 137m
repo-maintain-job-1719209273427-kd97d 0/1 Completed 0 133m
repo-maintain-job-1719209280087-8gkjf 0/1 Completed 0 133m
repo-maintain-job-1719209908574-z7sxn 0/1 Completed 0 123m
repo-maintain-job-1719210173336-v79t6 0/1 Completed 0 118m
repo-maintain-job-1719210179714-qvcxf 0/1 Completed 0 118m
repo-maintain-job-1719210473506-8sq2p 0/1 Completed 0 113m
repo-maintain-job-1719210777234-gj25r 0/1 Completed 0 108m
repo-maintain-job-1719210784436-xccc2 0/1 Completed 0 108m
repo-maintain-job-1719210790243-7l4xr 0/1 Completed 0 108m
repo-maintain-job-1719210796299-qsxxh 0/1 Completed 0 108m
repo-maintain-job-1719210803307-klmf5 0/1 Completed 0 108m
repo-maintain-job-1719210816386-tg5lv 0/1 Completed 0 108m
repo-maintain-job-1719211080497-92kcn 0/1 Completed 0 103m
repo-maintain-job-1719211383817-7rs7q 0/1 Completed 0 98m
repo-maintain-job-1719211970930-xnzd9 0/1 Completed 0 89m
repo-maintain-job-1719211978495-qssmt 0/1 Completed 0 88m
repo-maintain-job-1719212570931-rvs4b 0/1 Completed 0 79m
repo-maintain-job-1719212578228-xmhj6 0/1 Completed 0 78m
repo-maintain-job-1719212591336-7mg8h 0/1 Completed 0 78m
repo-maintain-job-1719212597369-m2vl7 0/1 Completed 0 78m
repo-maintain-job-1719212872898-vq5jn 0/1 Completed 0 73m
repo-maintain-job-1719212879622-x89cs 0/1 Completed 0 73m
repo-maintain-job-1719212890695-jcd6f 0/1 Completed 0 73m
repo-maintain-job-1719212895758-bljjw 0/1 Completed 0 73m
repo-maintain-job-1719212907794-cl4ql 0/1 Completed 0 73m
repo-maintain-job-1719212914813-sp8fb 0/1 Completed 0 73m
repo-maintain-job-1719212920871-tz62j 0/1 Completed 0 73m
repo-maintain-job-1719213777779-d5vbw 0/1 Completed 0 58m
repo-maintain-job-1719213786316-g698w 0/1 Completed 0 58m
repo-maintain-job-1719213792884-fmzxq 0/1 Completed 0 58m
repo-maintain-job-1719214073073-qqtcg 0/1 Completed 0 53m
repo-maintain-job-1719214380513-bmxf2 0/1 Completed 0 48m
repo-maintain-job-1719214673508-4ftdv 0/1 Completed 0 43m
repo-maintain-job-1719214679939-rhqwl 0/1 Completed 0 43m
repo-maintain-job-1719214685986-pwr2r 0/1 Completed 0 43m
repo-maintain-job-1719214693020-dtb25 0/1 Completed 0 43m
repo-maintain-job-1719214699105-ft7fn 0/1 Completed 0 43m
repo-maintain-job-1719214726196-svzcb 0/1 Completed 0 43m
repo-maintain-job-1719215277346-76mxk 0/1 Completed 0 33m
repo-maintain-job-1719215570939-wwz7m 0/1 Completed 0 29m
repo-maintain-job-1719215578298-rzhss 0/1 Completed 0 28m
repo-maintain-job-1719216170941-tbkkt 0/1 Completed 0 19m
repo-maintain-job-1719216470942-h4k5s 0/1 Completed 0 14m
repo-maintain-job-1719216483663-pr57g 0/1 Completed 0 13m
repo-maintain-job-1719216489668-rghrf 0/1 Completed 0 13m
repo-maintain-job-1719216770943-lbtqs 0/1 Completed 0 9m1s
repo-maintain-job-1719216784542-9c8zv 0/1 Completed 0 8m47s
repo-maintain-job-1719216790579-sfsr9 0/1 Completed 0 8m41s
repo-maintain-job-1719216796601-hvfrr 0/1 Completed 0 8m35s
repo-maintain-job-1719216804991-k4p5q 0/1 Completed 0 8m26s
repo-maintain-job-1719216811667-5kcsg 0/1 Completed 0 8m20s
repo-maintain-job-1719216818757-9crlx 0/1 Completed 0 8m13s
search-backup-dev-de1-kopia-n5khx-maintain-job-17192089979k5xfz 0/1 Completed 0 138m
search-backup-dev-de1-kopia-n5khx-maintain-job-17192126075jl9v7 0/1 Completed 0 78m
search-backup-dev-de1-kopia-n5khx-maintain-job-17192164969nwps5 0/1 Completed 0 13m
temporal-backup-dev-de1-kopia-x7btp-maintain-job-1719209019nkmg 0/1 Completed 0 138m
temporal-backup-dev-de1-kopia-x7btp-maintain-job-17192128878k4t 0/1 Completed 0 73m
temporal-backup-dev-de1-kopia-x7btp-maintain-job-1719216829jr8v 0/1 Completed 0 8m7s
theming-backup-dev-de1-kopia-5d687-maintain-job-1719205997d5sqw 0/1 Completed 0 3h8m
theming-backup-dev-de1-kopia-5d687-maintain-job-1719209896cgmwb 0/1 Completed 0 123m
theming-backup-dev-de1-kopia-5d687-maintain-job-1719213500kx57j 0/1 Completed 0 63m
velero-995fb7c64-hp6m5 1/1 Running 1 (4h44m ago) 2d22h
velero-watchdog-28653570-xh5pz 0/1 Completed 0 51m
wams-backup-dev-de1-kopia-tkrdg-maintain-job-1719208377640q5wgk 0/1 Completed 0 148m
wams-backup-dev-de1-kopia-tkrdg-maintain-job-1719211984534ww6pt 0/1 Completed 0 88m
wams-backup-dev-de1-kopia-tkrdg-maintain-job-1719215584331xfptr 0/1 Completed 0 28m
Also, you can see repo-maintain-job-*
is one has more than 3 and rest are only 3 by the way!
Version
$ velero version
Client:
Version: v1.14.0
Git commit: -
Server:
Version: v1.14.0
logs velero-logs.txt
@veerendra2 Could you share the entire log bundle but running velero debug
command?
@Lyndon-Li I tied it again, it seems not working for me
$ velero debug
An error occurred: velero deployment does not exist in namespace: velero
$ velero debug -n velero
An error occurred: velero deployment does not exist in namespace: velero
$ k get deployment -n velero
NAME READY UP-TO-DATE AVAILABLE AGE
velero 1/1 1 1 360d
I'm using macOS 14.5 23F79 arm64
@veerendra2
Here velero debug
command is trying to find velero deployment with two conditions:
velero
component=velero
Could you check if your installation meets these two conditions?
What steps did you take and what happened:
Deploy velero server with default options(i.e don't set
--keep-latest-maintenance-jobs
) and uploader typekopia
. Create backups with--snapshot-move-data
like belowFrom help, the default value for
--keep-latest-maintenance-jobs
is 3When
--keep-latest-maintenance-jobs
not set, there should be 3 job per backup repository, butWhat did you expect to happen: There should be only 3
maintain-job
pods per backup repositoriesThe following information will help us better understand what's going on:
If you are using velero v1.7.0+:
Please use
velero debug --backup <backupname> --restore <restorename>
to generate the support bundle, and attach to this issue, more options please refer tovelero debug --help
If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)
kubectl logs deployment/velero -n velero
velero backup describe <backupname>
orkubectl get backup/<backupname> -n velero -o yaml
velero backup logs <backupname>
velero restore describe <restorename>
orkubectl get restore/<restorename> -n velero -o yaml
velero restore logs <restorename>
Anything else you would like to add:
velero-server
logs -> https://gist.github.com/veerendra2/a25ba2e55070230af7cf791350f327f8Environment:
velero-server
node-agent
velero version
): Serverv1.13.2
velero client config get features
):kubectl version
):Server Version: v1.28.5
/etc/os-release
): UbuntuVote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.