kvaps / kube-linstor

Containerized LINSTOR SDS for Kubernetes, ready for production use.
Apache License 2.0
130 stars 25 forks source link

upgrade #10

Closed joalmjoalm closed 4 years ago

joalmjoalm commented 4 years ago

If I'm on v 1.1.2 and want to upgrade to later version, a) which are the system requirements for each version? b) how do I upgrade without loosing the current data/lvm configuration? c) any release notes?

kvaps commented 4 years ago

Hi, thanks for your question!

a) which are the system requirements for each version?

The current version is well tested on Kubernetes v1.17 but should also work on older ones except one thing:

In the last release csi-plugin was moved to lisntor namespace, but according https://github.com/kubernetes/kubernetes/issues/60596 the pods with system-cluster-critical priorityClass can work outside kube-system only since v1.17

Of course you can unspecify the priorityClass, but better to upgrade your Kubernetes cluster :)

b) how do I upgrade without loosing the current data/lvm configuration?

First and foremost make a backup, having backup will allow you to rollback the changes and try again, if something will gonna wrong.

  1. If you're using stolon as back-end for your LINSTOR installation you can easily perform the backup of your database:

    kubectl exec -ti -n linstor linstor-db-stolon-keeper-0 -- \
     sh -c 'PGPASSWORD=$(cat $STKEEPER_PG_SU_PASSWORDFILE) pg_dump -h linstor-db-stolon-proxy -U stolon linstor' \
     > linstor-backup.sql

    It always recommended to perform the backup before each LINSTOR upgrade.

  2. If you were using helm template to perform the installation as described in README.md for v1.1.2, I would suggest you switch to Helmv3, however helm template method should also work fine, we're using it with qbec.

    Anyway you can perform upgrade by simple replacing resources in your Kubernetes cluster thus

    Example plan: Upgrade stolon:

    • Remove all stolon resources except generated ones. The generated resources (like PVC's and stolon-cluster-linstor-db-stolon configmap), should remain in the cluster even after you remove mentioned statefulsets for them.
    • Install new resources with the same names, they should start using old PVCs and linstor-db-stolon configmap to reload cluster state

      Upgrade LINSTOR:

      • Remove all LINSTOR resources, it does not store any state in the Kubernetes cluster, so you can do that without fear. The LINSTOR state is stored only in database.
      • Create new LINSTOR resources. check the controller log, it should perform the schema migration for the database.
      • If it goes right, you should see your nodes in Offline state, that's because latest version enables mutual ssl authentification for the linstor-satellites. You can easily fix that by executing:
        linstor n l | awk '/(PLAIN)/ { print "linstor n i m -p 3367 --communication-type SSL " $2 " default" }' | sh -ex

        c) any release notes?

Since your version the these two major features were implemented:


I'll keep this issue open, for any feedback and until creating new documentation where these steps will be recorded.

joalmjoalm commented 4 years ago

Im running Kubernetes 1.17.4 and have deleted all linstor and stolon resources except ConfigMaps and Persistent Volumes. But I get this error when running install, sudo helm install linstor-db stable/stolon --namespace linstor -f examples/linstor-db.yaml Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: namespace: linstor, name: linstor-db-stolon, existing_kind: /v1, Kind=ConfigMap, new_kind: /v1, Kind=ConfigMap

How should I proceed? Is it easier to do a clean install and restore the db-backup?

kvaps commented 4 years ago

Sorry I was mistaken, you need to keep stolon-cluster-linstor-db-stolon configmap (not linstor-db-stolon). It is generated by stolon itself and keeping the cluster state.

Then you can install stolon with --set job.autoCreateCluster=false to not override it.

Is it easier to do a clean install and restore the db-backup?

It's up to you, this is also possible.

joalmjoalm commented 4 years ago

Latest release is up and running even if it took some time to found all resourses to delete in the prev installation. But I think I wiped my db by forgetting the autocreateflag. How do I restore the db from the backup?

kvaps commented 4 years ago

Sorry just noticed your comment, so I think you already found the solution, any way to recover backup just do:

kvaps commented 4 years ago

Fixed in https://github.com/kvaps/kube-linstor/commit/e10816434ef1631e01c79d56bb7900bd4fd4acf9