search

ceph / ceph-iscsi

Ceph iSCSI tools
GNU General Public License v3.0
62 stars 59 forks source link

[Question] How to properly upgrade ceph/rbd packages? #215

Open wwdillingham opened 4 years ago

wwdillingham commented 4 years ago

I would like to upgrade my ceph-iscsi 3.4 environment's underlying ceph/rbd packages to use the latest nautilus release. Going from 14.2.6 to 14.2.11

If I recall correctly in testing I just bounced the tcmu-runner daemon and this would cause "ceph versions" to report the new versions for tcmu-runner. However, now that I have some production workload I wanted to verify this was the proper way to do things?

Googling around I found this: https://bugzilla.redhat.com/show_bug.cgi?id=1568581 which says "Normally, the user should not be starting/stopping tcmu-runner manually. Components like rtslib and rbd-target-gw/ceph-iscsi-config and systemd will handle this."

So my question is, during such upgrades, what is the best practice way for an operator to cause tcmu-runner to begin using the new version of ceph.

Thanks in advance.

dillaman commented 4 years ago

That BZ was referring to a bug in pre-RHEL 7.6 kernels where IO could get permanently stuck in the kernel IO ring requiring a reboot of the box to recover.

wwdillingham commented 4 years ago

Thank you @dillaman .. so simply a restart of tcmu-runner is the appropriate step?

dillaman commented 4 years ago

Correct, shouldn't be an issue if the kernel version isn't an issue. The ceph-ansible rolling upgrade script will stop the iSCSI services on a node, upgrade, and then restart [1]

[1] https://github.com/ceph/ceph-ansible/blob/cec994b973343e05252b4aa0f0be0dd0ee39405b/infrastructure-playbooks/rolling_update.yml#L750