Chase up teams with erroring jobs

[sj-williams]

Background

We seem to have a group of usual suspects when it comes to cronjob pods in error state.

Reach out to the owners of the common namespaces/pods and see whether they have an understanding of what is happening/whether they have a fix in place.

We should do this as a prerequisite for looking at the more cleanup solution in this ticket: https://github.com/ministryofjustice/cloud-platform/issues/5672

Example of pods:

❯ kubectl get pods -A --field-selector status.phase=Failed
NAMESPACE                                   NAME                                                            READY   STATUS   RESTARTS   AGE
claim-criminal-injuries-compensation-dev    claim-criminal-injuries-dev-cleardown-28645988-62hsz            0/1     Error    0          10h
claim-criminal-injuries-compensation-dev    claim-criminal-injuries-dev-push-to-gateway-28645988-hmq2p      0/1     Error    0          10h
claim-criminal-injuries-compensation-dev    claim-criminal-injuries-dev-resubmit-failed-28645988-c5k26      0/1     Error    0          10h
claim-criminal-injuries-compensation-prod   claim-criminal-injuries-prod-cleardown-28645988-qsnpp           0/1     Error    0          10h
claim-criminal-injuries-compensation-prod   claim-criminal-injuries-prod-push-to-gateway-28645988-547jc     0/1     Error    0          10h
claim-criminal-injuries-compensation-prod   claim-criminal-injuries-prod-resubmit-failed-28645988-566d6     0/1     Error    0          10h
court-probation-preprod                     analytics-data-extractor-28645980-bbrzt                         0/1     Error    0          10h
court-probation-preprod                     analytics-data-extractor-28645980-dqhzc                         0/1     Error    0          10h
court-probation-preprod                     analytics-data-extractor-28645980-jq4mn                         0/1     Error    0          10h
court-probation-preprod                     analytics-data-extractor-28645980-kb598                         0/1     Error    0          11h
court-probation-preprod                     analytics-data-extractor-28645980-lkx49                         0/1     Error    0          10h
court-probation-preprod                     analytics-data-extractor-28645980-nn4d9                         0/1     Error    0          11h
court-probation-preprod                     analytics-data-extractor-28645980-thg2j                         0/1     Error    0          10h
hmpps-audit-dev                             queue-housekeeping-cronjob-28646250-6prpf                       0/1     Error    0          6h27m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646250-jnqpp                       0/1     Error    0          6h22m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646250-l7fpp                       0/1     Error    0          6h27m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646250-n4dsh                       0/1     Error    0          6h26m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646250-nct57                       0/1     Error    0          6h16m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646250-svpvp                       0/1     Error    0          6h24m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646250-x7svz                       0/1     Error    0          6h27m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646270-b4qxl                       0/1     Error    0          6h13m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646270-bdsd9                       0/1     Error    0          6h15m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646270-bthz6                       0/1     Error    0          6h16m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646270-h8ptt                       0/1     Error    0          6h16m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646270-m5wrd                       0/1     Error    0          6h15m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646270-v8tq8                       0/1     Error    0          6h10m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646270-xm8r5                       0/1     Error    0          6h5m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646280-52dl6                       0/1     Error    0          6h4m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646280-7wszh                       0/1     Error    0          5h59m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646280-k4h86                       0/1     Error    0          6h5m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646280-k7fs7                       0/1     Error    0          5h54m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646280-llk9x                       0/1     Error    0          6h2m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646280-txg7h                       0/1     Error    0          6h5m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646280-xmpbt                       0/1     Error    0          6h3m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646290-4n57p                       0/1     Error    0          5h53m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646290-8hr4t                       0/1     Error    0          5h43m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646290-8jmxh                       0/1     Error    0          5h53m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646290-9r2l6                       0/1     Error    0          5h52m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646290-b4ptn                       0/1     Error    0          5h54m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646290-vvt4z                       0/1     Error    0          5h51m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646290-xfmnf                       0/1     Error    0          5h48m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646300-28qkx                       0/1     Error    0          5h42m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646300-2f6pn                       0/1     Error    0          5h42m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646300-b4wmm                       0/1     Error    0          5h37m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646300-dl94w                       0/1     Error    0          5h39m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646300-t5s29                       0/1     Error    0          5h31m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646300-v97nq                       0/1     Error    0          5h41m
hmpps-audit-dev                             queue-housekeeping-cronjob-28646300-xtmbz                       0/1     Error    0          5h42m
hmpps-education-and-work-plan-dev           export-database-to-analytical-platform-cronjob-28645980-2p8gz   0/1     Error    0          11h
hmpps-education-and-work-plan-dev           export-database-to-analytical-platform-cronjob-28645980-bwvvz   0/1     Error    0          11h
hmpps-education-and-work-plan-dev           export-database-to-analytical-platform-cronjob-28645980-h26ps   0/1     Error    0          11h
hmpps-education-and-work-plan-dev           export-database-to-analytical-platform-cronjob-28645980-kx6r8   0/1     Error    0          11h
hmpps-education-and-work-plan-preprod       export-database-to-analytical-platform-cronjob-28645980-4j5nf   0/1     Error    0          11h
hmpps-education-and-work-plan-preprod       export-database-to-analytical-platform-cronjob-28645980-6w94c   0/1     Error    0          11h
hmpps-education-and-work-plan-preprod       export-database-to-analytical-platform-cronjob-28645980-jdpnl   0/1     Error    0          11h
hmpps-education-and-work-plan-preprod       export-database-to-analytical-platform-cronjob-28645980-kf7q8   0/1     Error    0          11h
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646455-4fw26                       0/1     Error    0          3h11m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646455-5jz65                       0/1     Error    0          3h11m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646455-6nnb9                       0/1     Error    0          3h9m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646455-9mntx                       0/1     Error    0          3h12m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646455-lnzqv                       0/1     Error    0          3h6m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646455-xnzvs                       0/1     Error    0          3h10m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646455-zmmbj                       0/1     Error    0          3h
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646465-7wvdt                       0/1     Error    0          3h
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646465-8kgzp                       0/1     Error    0          3h
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646465-hfrn9                       0/1     Error    0          179m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646465-hkh56                       0/1     Error    0          169m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646465-kbgqc                       0/1     Error    0          175m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646465-pm5xv                       0/1     Error    0          3h
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646465-sz67c                       0/1     Error    0          177m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646475-25mfk                       0/1     Error    0          168m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646475-4vcsb                       0/1     Error    0          163m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646475-fw4dx                       0/1     Error    0          166m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646475-hlh5x                       0/1     Error    0          158m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646475-jw7g6                       0/1     Error    0          169m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646475-p5wcr                       0/1     Error    0          169m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646475-rhwvj                       0/1     Error    0          168m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646495-5g9cz                       0/1     Error    0          151m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646495-d6mgh                       0/1     Error    0          152m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646495-fw7t7                       0/1     Error    0          151m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646495-hglck                       0/1     Error    0          149m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646495-jzcls                       0/1     Error    0          140m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646495-tn789                       0/1     Error    0          146m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646495-x2sxr                       0/1     Error    0          150m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646505-9lkxx                       0/1     Error    0          129m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646505-f92kq                       0/1     Error    0          137m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646505-jnh2g                       0/1     Error    0          140m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646505-kgglp                       0/1     Error    0          140m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646505-qthjh                       0/1     Error    0          139m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646505-sz47r                       0/1     Error    0          140m
hmpps-launchpad-dev                         purge-sso-requests-cronjob-28646505-zdbxr                       0/1     Error    0          135m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646550-659dp                       0/1     Error    0          73m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646550-7vjfd                       0/1     Error    0          79m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646550-9nr98                       0/1     Error    0          82m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646550-cg5kf                       0/1     Error    0          88m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646550-dx4s7                       0/1     Error    0          84m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646550-kdw8r                       0/1     Error    0          87m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646550-mwktq                       0/1     Error    0          86m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646565-jbssr                       0/1     Error    0          71m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646565-js4rh                       0/1     Error    0          72m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646565-k6m76                       0/1     Error    0          67m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646565-kzsv4                       0/1     Error    0          70m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646565-ltvqx                       0/1     Error    0          69m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646565-tnzt2                       0/1     Error    0          57m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646565-zwf5d                       0/1     Error    0          63m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646595-cgllw                       0/1     Error    0          51m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646595-gwv9z                       0/1     Error    0          37m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646595-gzgbd                       0/1     Error    0          50m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646595-hg67v                       0/1     Error    0          46m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646595-hkcmm                       0/1     Error    0          43m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646595-kfzdl                       0/1     Error    0          51m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646595-kj8hs                       0/1     Error    0          48m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646610-28zv2                       0/1     Error    0          36m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646610-747bb                       0/1     Error    0          33m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646610-75gm9                       0/1     Error    0          31m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646610-ckd6b                       0/1     Error    0          34m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646610-dq54j                       0/1     Error    0          27m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646610-drk4c                       0/1     Error    0          35m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646610-qg7jx                       0/1     Error    0          21m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646625-2jtxn                       0/1     Error    0          20m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646625-ddg7f                       0/1     Error    0          6m7s
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646625-fnlfv                       0/1     Error    0          15m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646625-jdfnt                       0/1     Error    0          12m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646625-jnmrv                       0/1     Error    0          17m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646625-mpf7b                       0/1     Error    0          18m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646625-qkjgh                       0/1     Error    0          20m
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646640-7q9n4                       0/1     Error    0          2m3s
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646640-lkzz5                       0/1     Error    0          4m29s
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646640-nvk95                       0/1     Error    0          5m22s
hmpps-workload-preprod                      queue-housekeeping-cronjob-28646640-sdcjb                       0/1     Error    0          3m26s
kuberhealthy                                namespace-kh-check-1718726527                                   0/1     Error    0          20h
laa-apply-for-legalaid-uat                  apply-ap-5096-sca-means-te-export-digest-28646115-54bkk         0/1     Error    0          8h
laa-apply-for-legalaid-uat                  apply-ap-5096-sca-means-te-export-digest-28646115-b49g8         0/1     Error    0          8h
laa-apply-for-legalaid-uat                  apply-main-export-digest-28646115-55897                         0/1     Error    0          8h
laa-apply-for-legalaid-uat                  apply-main-export-digest-28646115-gs9vm                         0/1     Error    0          8h
pathfinder-dev                              moj-data-platform-extractor-28646040-9bm9q                      0/1     Error    0          10h
pathfinder-dev                              moj-data-platform-extractor-28646040-c7xn5                      0/1     Error    0          10h
pathfinder-dev                              moj-data-platform-extractor-28646040-cg87r                      0/1     Error    0          10h
pathfinder-dev                              moj-data-platform-extractor-28646040-d4ht5                      0/1     Error    0          9h
pathfinder-dev                              moj-data-platform-extractor-28646040-fshkq                      0/1     Error    0          10h
pathfinder-dev                              moj-data-platform-extractor-28646040-pjbdj                      0/1     Error    0          10h
pathfinder-dev                              moj-data-platform-extractor-28646040-vns4h                      0/1     Error    0          10h
pathfinder-dev                              push-extract-28645965-75249                                     0/1     Error    0          6h14m
pathfinder-dev                              push-extract-28645965-7cwdh                                     0/1     Error    0          9h
pathfinder-dev                              push-extract-28645965-f6vwx                                     0/1     Error    0          11h
pathfinder-dev                              push-extract-28645965-j77dn                                     0/1     Error    0          8h
pathfinder-dev                              push-extract-28645965-srg4t                                     0/1     Error    0          7h17m
pathfinder-dev                              push-extract-28645965-tgzwb                                     0/1     Error    0          10h

Proposed user journey

Approach

Which part of the user docs does this impact

Communicate changes

• [ ] post for #cloud-platform-update
• [ ] Weeknotes item
• [ ] Show the Thing/P&A All Hands/User CoP
• [ ] Announcements channel

Questions / Assumptions

Definition of done

• [ ] readme has been updated
• [ ] user docs have been updated
• [ ] another team member has reviewed
• [ ] smoke tests are green
• [ ] prepare demo for the team

Reference

How to write good user stories

[sj-williams]

contacted pathfinder team - they are investigating manage-soc-cases-dev & pathfinder-dev