PX Backup 2.7.0 Sign off

kshithijiyer-px commented 6 months ago

PXB Sign-off-Template Following are the tasks which has to be completed by following teams

NOTE: This issue should not be closed until PX-Backup build/version pushed to IBM Cloud Catalog

Sign-off process Step 1:

Portworx Team:

[x] Planned PX-Backup Release Version: https://docs.portworx.com/release-notes/portworx-backup-on-prem
[x] Lists of Tests executed:
[x] Screenshots/Summary of Test Execution:
[x] IKS/ROKS Support Matrix for new version: IKS: 1.29.x, 1.28.8, 1.27.x ROKS: 4.15.x, 4.14.x, 4.13.x, 4.12.x
[x] Steps for existing users to migrate to new versions: Same as 2.5.1
[x] Bug fixes V/S TestCase mapping:
[ ] Please attach logs from both the PX-Backup and stork(source and destination cluster) components that were used during the execution of the tests described above:

IBM Team:

[ ] Planned PX-Backup Release Version:
[ ] Provide information about the IKS cluster(including BOM version) where integration tests were conducted:
[ ] Provide information about the ROKS cluster(including BOM version) where integration tests were conducted:
[ ] Torpedo Test Result's Details from IKS Cluster:
[ ] Torpedo Test Result's Details from ROKS Cluster:
[ ] IBM Team need to verify tasks and sign-off PX_Backup build

Sign-off process Step 2:

NOTE: Once above tasks completed and marked as done including sign-off then PX-Backup and IBM Team need to complete following tasks as well

PX-Backup Team:

[ ] Push sign-off PX-Backup build to IBM Cloud Catalog in one region i.e us-east or so
[ ] Verify torpedo tests(via automation) on default IKS and ROKS version - In case of test failure build has to be rollback and discussed with IBM Team.
[ ] Push sign-off PX-Backup build to all IBM Cloud Catalog regions.

IBM Team:

[ ] Run PX-Backup automated tests on one region for IKS and ROKS default version
[ ] In case of tests failure discuss with PX-Backup team and take action accordingly
[ ] Close this issue in case of success test result.

kshithijiyer-px commented 6 months ago

Screenshot 2024-05-22 at 10 54 35 AM

The test cases which have failed are due to PB-6896 PB-6819 which are Known issues which have been release noted in our documentation: https://docs.portworx.com/release-notes/portworx-backup-on-prem#known-issues

kshithijiyer-px commented 6 months ago

Testcase details have attached: px_backup_2.7.0_system_test.csv

ambiknai commented 6 months ago

Test Suite Passed !!
BasicBackupCreation | CancelAllRunningRestoreJobs | ScaleMongoDBWhileBackupAndRestore | BasicSelectiveRestore | BackupMultipleNsWithSameLabel 
PX-BACKUP-VERSION: 2.7.0

k8s: default (1.29.4 )

ambiknai commented 6 months ago

Test Suite Passed !!

BasicBackupCreation | CancelAllRunningRestoreJobs | ScaleMongoDBWhileBackupAndRestore | BasicSelectiveRestore | BackupMultipleNsWithSameLabel 
PX-BACKUP-VERSION: 2.7.0

ocp: default

ROKS: 4.14.24_openshift

ambiknai commented 6 months ago

@kshithijiyer-px Torpedo test on 4.15 cluster failed with connection error. Did you create cluster by disabling secure by default and then trigger tests

kshithijiyer-px commented 6 months ago

@kshithijiyer-px Torpedo test on 4.15 cluster failed with connection error. Did you create cluster by disabling secure by default and then trigger tests

Yes @ambiknai the images aren't push to catalog yet hence we would need to disable secure by default to allow to pull from docker.io.

ambiknai commented 5 months ago

@kshithijiyer-px Alright. When can we expect the images in catalog. Is 2.7.0 push planned in IBM Catalog once the images are in place.

Balachandar-Pan commented 5 months ago

@ambiknai Test catalog is updated and verified with ROKS 4.15.14_1537 with secure enabled via upgrade scenarios from PX-B 2.6.0 to 2.7.0.

[root@ip-10-13-8-162 ~]# helm ls -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION px-backup pxb270 2 2024-06-04 05:04:05.41204968 +0000 UTC deployed px-central-2.7.0 2.7.0

[root@ip-10-13-8-162 ~]# kubectl -n pxb270 get po NAME READY STATUS RESTARTS AGE alertmanager-px-backup-alertmanager-0 2/2 Running 0 30h alertmanager-px-backup-alertmanager-1 2/2 Running 0 30h prometheus-operator-6fc9bb4db-p7tbv 1/1 Running 0 30h prometheus-px-backup-dashboard-prometheus-0 2/2 Running 0 30h prometheus-px-backup-dashboard-prometheus-1 2/2 Running 0 30h px-backup-5cfc5858d7-lhxrv 1/1 Running 0 30h pxc-backup-mongodb-0 1/1 Running 0 30h pxc-backup-mongodb-1 1/1 Running 0 30h pxc-backup-mongodb-2 1/1 Running 0 30h pxcentral-apiserver-6ccffdbbd6-f2rsb 1/1 Running 0 30h pxcentral-backend-7c977b978b-jl4pq 1/1 Running 0 30h pxcentral-frontend-648b698dd7-lwrn6 1/1 Running 0 30h pxcentral-keycloak-0 1/1 Running 0 30h pxcentral-keycloak-postgresql-0 1/1 Running 0 31h pxcentral-lh-middleware-85dcd8559c-hzh7h 1/1 Running 0 30h pxcentral-mysql-0 1/1 Running 0 30h pxcentral-post-install-hook-5btt7 0/1 Completed 0 30h

[root@ip-10-13-8-162 ~]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 10.240.0.52 Ready master,worker 6d2h v1.28.9+8ca71f7 10.240.0.52 10.240.0.52 Red Hat Enterprise Linux 8.9 (Ootpa) 4.18.0-513.24.1.el8_9.x86_64 cri-o://1.28.6-5.rhaos4.15.gita02fb1e.el8 10.240.128.51 Ready master,worker 6d2h v1.28.9+8ca71f7 10.240.128.51 10.240.128.51 Red Hat Enterprise Linux 8.9 (Ootpa) 4.18.0-513.24.1.el8_9.x86_64 cri-o://1.28.6-5.rhaos4.15.gita02fb1e.el8 10.240.64.58 Ready master,worker 6d2h v1.28.9+8ca71f7 10.240.64.58 10.240.64.58 Red Hat Enterprise Linux 8.9 (Ootpa) 4.18.0-513.24.1.el8_9.x86_64 cri-o://1.28.6-5.rhaos4.15.gita02fb1e.el8 [root@ip-10-13-8-162 ~]#

pxb270_image version.txt

ambiknai commented 5 months ago

@Balachandar-Pan Thank you for sharing this result. May I know if I can execute same from torpedo. Is secure by default tests supported in torpedo test suite. Few days back when I ran the tests, it failed for 4.15

kshithijiyer-px commented 5 months ago

@ambiknai to run torpedo with Secure by Default enabled we need to push the app images to custom ICR repo and then Pass it to torpedo by using -e INTERNAL_DOCKER_REGISTRY=docker.pwx.dev.purestorage.com -e IMAGE_PULL_SERVER=docker.pwx.dev.purestorage.com -e IMAGE_PULL_USERNAME=**** -e IMAGE_PULL_PASSWORD=**** parameters. We haven't tried this approach we'll need time till next week as we are stuck with this release to build and get a pipeline ready with these changes.

@ambiknai If you have bandwidth can you please try in parallel and let us know if this approach works?

CC: @trenukarya-px

ambiknai commented 5 months ago

@kshithijiyer-px

What I understand is we have to push TORPEDO_IMG=portworx/torpedo-backup:master to icr.io. Unfortunately we are not allowed to manually push any of the images to icr.io registry . We need an approved pipeline which we do not have as of now for pushing any of the portworx images. Could you post an update here once test is done from your end.

I know for PXE, images are pushed, to icr.io/ext/portworx by PX team.

kshithijiyer-px commented 5 months ago

@ambiknai We have already completed the tests which are documented in https://github.com/IBM/signoff-pxb/issues/22#issuecomment-2149787434 and https://github.com/IBM/signoff-pxb/issues/22#issuecomment-2123893816. Are you looking for something else as well?

ambiknai commented 5 months ago

@kshithijiyer-px I was talking about the approach you mentioned here

IBM / signoff-pxb

PX Backup 2.7.0 Sign off #22