QA fixes for upgrade cluster

[Xartos]

What this PR does / why we need it:

Which issue this PR fixes (use the format fixes #<issue number>(, fixes #<issue_number>, ...) to automatically close the issue when PR gets merged): fixes #

Public facing documentation PR (if applicable)

Special notes for reviewer:

Checklist:

• [ ] Proper commit message prefix on all commits
• [ ] Updated the public facing documentation
• Is this changeset backwards compatible for existing clusters? Applying:
  • [ ] is completely transparent, will not impact the workload in any way.
  • [ ] requires running a migration script.
  • [ ] requires draining and/or replacing nodes.
  • [ ] will break the cluster. I.e. full cluster migration is required.

[linus-astrom]

You haven't made a WIP-CHANGELOG.md entry.

[Xartos]

You haven't made a WIP-CHANGELOG.md entry.

Yes, I should fix that. However for QA branches you should update CHANGELOG.md directly. But I'll fix that