Vendor applied untested patches and upgrades to production

[davidjb]

In short, an untested patch during business hours from Squiz caused the failure of SSJS across our sites, breaking all of our sites. The patch was reversed, but we're now faced with the bigger issue of managing future changes:

• A back-out plan should be formulated and ready to enact in case of failure
• Patches/fixes/upgrades need to be tested before deployment
• Ideally, this is performed in a staging environment where we can test before giving the go-ahead
• Only when we've tested should the patch be applied to production
• Patch application should be scheduled for a time out of hours
• Notification should be given as to when production can be tested and finalised

Applying fixes/patches directly to production without testing risks the operation of the whole system and this discussion will continue.

[davidjb]

Closing this as wontfix -- the current practices by Squiz make it "too complicated" to do anything but apply patches directly to production without testing. This represents far too great a business risk, even if applied after hours. We'll now wait until major version upgrades to get our patches as then and only then can we test changes in a non-prod environment.