Depricating all python based services deployed in `prod` and `preprod` Kubernetes clusters

[todor-ivanov]

The problem:

We have recently discovered that in our prod and preprod Kubernetes clusters we still have the old python based services deployed. Here is the status of those deployemets:

• Prod:

  [user@lxplus9109 deploymentK8]$ k8init prod
  [user@lxplus9109 deploymentK8]$ kubectl get deploy -n dbs  |grep ^dbs
  dbs-global-r      1/1     1            1           601d
  dbs-global-w      1/1     1            1           601d
  dbs-phys03-r      1/1     1            1           601d
  dbs-phys03-w      1/1     1            1           601d
  dbs2go-global-r   5/5     5            5           601d
  dbs2go-global-w   3/3     3            3           601d
  dbs2go-phys03-r   3/3     3            3           601d
  dbs2go-phys03-w   3/3     3            3           601d

• Preprod:

[user@lxplus9109 deploymentK8]$ k8init preprod
[user@lxplus9109 deploymentK8]$ kubectl get deploy -n dbs  |grep ^dbs
dbs-global-r              5/5     5            5           142d
dbs-global-w              5/5     5            5           142d
dbs-migrate               5/5     5            5           142d
dbs-phys03-r              5/5     5            5           142d
dbs-phys03-w              5/5     5            5           142d
dbs2go-global-m           1/1     1            1           142d
dbs2go-global-migration   1/1     1            1           142d
dbs2go-global-r           5/5     5            5           142d
dbs2go-global-w           3/3     3            3           142d
dbs2go-migrate            2/3     3            2           142d
dbs2go-migration          0/3     3            0           142d
dbs2go-phys03-m           1/1     1            1           142d
dbs2go-phys03-migration   1/1     1            1           142d
dbs2go-phys03-r           5/5     5            5           142d

• Out of them these are the old deployments:

  [user@lxplus9109 deploymentK8]$ k8init preprod
  [user@lxplus9109 deploymentK8]$ kubectl get deploy -n dbs -o wide  |grep ^dbs |grep HG
  dbs-global-r              5/5     5            5           142d   dbs-global-r,dbs-global-r-filebeat                         registry.cern.ch/cmsweb/dbs:HG2205e-stable,docker.elastic.co/beats/filebeat:7.12.0   app=dbs-global-r
  dbs-global-w              5/5     5            5           142d   dbs-global-w,dbs-global-w-filebeat                         registry.cern.ch/cmsweb/dbs:HG2205e-stable,docker.elastic.co/beats/filebeat:7.12.0   app=dbs-global-w
  dbs-migrate               5/5     5            5           142d   dbs-migrate,dbs-migrate-filebeat                           registry.cern.ch/cmsweb/dbs:HG2205e-stable,docker.elastic.co/beats/filebeat:7.12.0   app=dbs-migrate
  dbs-phys03-r              5/5     5            5           142d   dbs-phys03-r,dbs-phys03-r-filebeat                         registry.cern.ch/cmsweb/dbs:HG2205e-stable,docker.elastic.co/beats/filebeat:7.12.0   app=dbs-phys03-r
  dbs-phys03-w              5/5     5            5           142d   dbs-phys03-w,dbs-phys03-w-filebeat                         registry.cern.ch/cmsweb/dbs:HG2205e-stable,docker.elastic.co/beats/filebeat:7.12.0   app=dbs-phys03-w

Desired solution:

Deprecate all python based dbs instances in both clusters prod and preprod

Additional information:

None

[todor-ivanov]

All python based services in preprod has been scaled to 0 replicas with the following set of commands:

k8init preprod

for depl in `kubectl get deploy -n dbs -o wide  |grep HG |awk '{print $1}'`
> do
> echo "=== Scaling down to 0 pods deployment: $depl ==="
> echo -n "y/n: "; read y
>  [[ $y == y ]] && kubectl -n dbs scale deploy $depl --replicas=0 || echo NOT scaling down
> done 

=== Scaling down to 0 pods deployment: dbs-global-r ===
y/n: y
deployment.apps/dbs-global-r scaled

=== Scaling down to 0 pods deployment: dbs-global-w ===
y/n: y
deployment.apps/dbs-global-w scaled

=== Scaling down to 0 pods deployment: dbs-migrate ===
y/n: y
deployment.apps/dbs-migrate scaled

=== Scaling down to 0 pods deployment: dbs-phys03-r ===
y/n: y
deployment.apps/dbs-phys03-r scaled

=== Scaling down to 0 pods deployment: dbs-phys03-w ===
y/n: y
deployment.apps/dbs-phys03-w scaled

Here is the result:

kubectl -n dbs  get pods |grep ^dbs
dbs2go-global-m-766ddb8c7f-9cnwh           2/2     Running            0                  37h
dbs2go-global-migration-5f54f44d6c-zwm8p   2/2     Running            0                  37h
dbs2go-global-r-5b55b4cb5f-4x7fn           2/2     Running            0                  37h
dbs2go-global-r-5b55b4cb5f-9rzxp           2/2     Running            0                  37h
dbs2go-global-r-5b55b4cb5f-lwdfp           2/2     Running            0                  37h
dbs2go-global-r-5b55b4cb5f-mfrfv           2/2     Running            0                  37h
dbs2go-global-r-5b55b4cb5f-rqrjn           2/2     Running            0                  37h
dbs2go-global-w-79fd47d745-jdfrm           2/2     Running            0                  37h
dbs2go-global-w-79fd47d745-ng7pl           2/2     Running            0                  37h
dbs2go-global-w-79fd47d745-z2bj5           2/2     Running            0                  37h
dbs2go-migrate-6586548f99-9pqsz            1/2     CrashLoopBackOff   31796 (119s ago)   119d
dbs2go-migrate-6586548f99-bctxv            1/2     CrashLoopBackOff   31685 (3m1s ago)   119d
dbs2go-migrate-798c9c847d-bkhbh            1/2     CrashLoopBackOff   442 (2m23s ago)    37h
dbs2go-migrate-798c9c847d-cb2v9            1/2     CrashLoopBackOff   422 (71s ago)      35h
dbs2go-migration-787b9fc5b5-s48sg          1/2     CrashLoopBackOff   31663 (4m8s ago)   119d
dbs2go-migration-869d5c9ff-26qc5           1/2     CrashLoopBackOff   441 (3m56s ago)    37h
dbs2go-migration-869d5c9ff-qlgcs           1/2     CrashLoopBackOff   365 (63s ago)      30h
dbs2go-migration-869d5c9ff-wr9hv           1/2     CrashLoopBackOff   441 (77s ago)      37h
dbs2go-phys03-m-5dc48f5ddf-k245m           2/2     Running            0                  37h
dbs2go-phys03-migration-54dc9ffdb9-wm7l9   2/2     Running            0                  37h
dbs2go-phys03-r-7474967457-9wt6z           2/2     Running            0                  37h
dbs2go-phys03-r-7474967457-j57dz           2/2     Running            0                  37h
dbs2go-phys03-r-7474967457-lg2fp           2/2     Running            0                  37h
dbs2go-phys03-r-7474967457-mp6j8           2/2     Running            0                  37h
dbs2go-phys03-r-7474967457-rbs7p           2/2     Running            0                  37h

If I hear no complains by mid next week, I'll delete those deployments completely and will repeat the procedure in the prod cluster upon proper announcement in all relevant channels.