search

ibm-messaging / mq-container

Container images for IBM® MQ
Apache License 2.0
252 stars 188 forks source link

MQ 9.1.4.0 version upgrade to 9.2.1.0 maintaining shared storage results in broken installation #453

Open Jacobreise opened 3 years ago

Jacobreise commented 3 years ago

Utilizing a working IBM certified image for 9.1.4.0 and removing all configmaps, sts and all besides the shared pvcs mounted to /mnt/mqm-data & /mnt/mqm-log and deploying a new instance of the same service and queue manager name utilizing the IBM certified image for 9.2.1.0 results in a broken installation. On the surface it does not appear broken however the pods never even attempt to utilize the updated configmap (primarily focused on mqsc) after this upgrade. We can see in /etc/mqm that the correct mqsc configmap is mounted as specified by the STS however in the MQ logs it never attempts to use it. We can then attempt to force it by bringing down the sts pods to 0 and bringing it back up. The expected behavior for both a fresh 9.1.4.0 and a fresh 9.2.1.0 results in the pod running the mqsc configmap commands, but in this instance (upgrading while keeping storage) it still never even tries to utilize the configmap. I can provide any files requested or run any tests as required but we have a 100% success rate in producing this issue during the upgrade. I also want to note that using a fresh install (new storage et al) of the exact same files for 9.1.4.0 and 9.2.1.0 results in the expected behavior of utilizing and updating the mqsc configmap.

Jacobreise commented 3 years ago

Created the IBM case TS005540815 as it appears 9.2.1.0 image is not working as intended.

Jacobreise commented 3 years ago

Based on https://github.com/ibm-messaging/mq-container/blob/master/docs/usage.md the below is said.

By creating your own image and adding your own MQSC file into the /etc/mqm directory on the image. This file will be run when your queue manager is created.

I can see the updated mqsc file there however MQ never tries to use it. It never tries to use it because it believes it is in recovery mode and never re-creates the queue manager on restart. After multiple scale downs it is always in recovery mode and never attempts to create the queue manager leading to this indefinite loop of never using the updated configmap.

Jacobreise commented 3 years ago

After speaking with an MQ solutions architect we found the issue being in 9.1.4.0 it does not by default utilize the AutoConfig stanza to offload the running of the mqsc and ini files. In version 9.1.5.0 and on it does. So when going from 9.1.4.0 to a higher version while maintaining storage the new files are never run because MQ no longer relies on OCP to run it, MQ runs it itself but without the stanza in the ini from the 9.1.4.0 version install it does not know where to go. If you add the AutoConfig stanza and the locations of you mqsc and ini file to you qm.ini and restart the pod so the updated files are mounted within the 9.1.4.0 version. You can then go about upgrading, while maintaining storage to the new version. This seems to have been lost in the documentation of upgrades and might be helpful to "patch" into 9.1.4.0 base to have that stanza defined with default values.