Site upgrade

[blackshadowshade]

The production instance that we are currently running needs updating, for a number of reasons:

• to bring various software products up to a currently-supported version
• to allow us to use new features
• to bring in security patches

Specific issues that may relate to this site upgrade are:

• [ ] #161: Downloading PHPUnit via puppet
• [ ] #224: Using SSL
• [ ] #439: JQuery version
• [ ] #440: JSHint version
• [ ] #575: Puppet version
• [ ] #1083: Manage third-party Javascript libraries
• [ ] #1097: buttonweavers.com should redirect to www.buttonweavers.com
• [x] #1167: Upgrade PHP
• [x] #1168: Upgrade PHPUnit
• [ ] #1339: Backup drive running out of space
• [ ] #1989: Connection errors
• [ ] #2143: MySQL settings to enable temporary backups
• [ ] #2197: Python 3 compatibility
• [ ] #2208: Adding the statistical environment R
• [x] #2342: Changing CircleCI to use the new upgraded versions

[blackshadowshade]

Okay, good and bad news on the audit front.

I've just had a look at the isset() calls. I count 16 that need to be rewritten in the production code, but hundreds that need to be rewritten in the test code. The reason that the test code isn't showing errors is because the asserts are currently not testing anything, since they're similar to

assertFalse(isset($die->max))

which will always pass, due to the change of behaviour of isset().

I'll base a pull request on the 2398_site_upgrade branch.

[cgolubi1]

The replay site has played 13100 each replayed and novel games with no errors or mismatches. How many is enough? I don't know. That certainly seems like enough to tell us that any logic bugs introduced are pretty subtle. So i'll let it keep going, but i'm inclined to say that logically speaking, the site can play games.

We need to figure out what to do about the automated apt package updates:

• I definitely don't want them on replay sites, but i can handle that any number of ways, up to and including just having my replay site setup script remove the cron job.
• The advantage of having them on staging and prod is that those sites will actually get OS updates
• The disadvantage of having them on staging and prod is that those sites will get OS updates at an unpredictable time and in an unpredictable order (e.g. prod before staging).

My inclination is to leave them on, because it gets us patching of some packages with no human time expended, and the downside (a bad update completely toasts the prod site and we have to rebuild it) isn't that bad. Anyone disagree or agree or have any opinion about this at all?

[blackshadowshade]

Is it possible to just accept some level of automatic patching, e.g., critical or mature patches?

[irilyth]

Automatically notifying about updates (all, or a subset) seems like a fine idea; automatically applying them seems less appealing to me.

[cgolubi1]

Okay, i hear y'all --- i'll figure out how to disable automatic patching.

Irilyth: as far as i know, you are the first person to say the words "automatic notification" in this conversation. I don't know how to configure that, and it doesn't sound valuable to me. So i'm not planning to look into it unless someone tells me i'm missing something.

[irilyth]

Oh, if you're in favor of automatic updates and want to try it out, we can. I assumed you were opposed and James was in favor. :^)

The value of automatic notification is that we'd get e-mail from a cron job or something when there were updates available, and then we could go apply them. If you don't have that, and will just install updates in some other way on some other schedule, that's fine with me.

[blackshadowshade]

I think Chaos understands, but I'd be okay with automatic patching of critical fixes. Automatic patching of everything seems like a recipe for disaster. :)

[cgolubi1]

Okay, i pushed a change to disable automated updates.

[cgolubi1]

I'm very tempted to try to get this upgrade done this week. Here's what i think that would look like:

• Right now:
  • Continue to let replay testing go on shadowshade's branch and my initial site
  • Make a clean rebased copy of my branch plus shadowshade's two commits, put up a PR for that, and start some replay testing on that as well just to be on the safe side
• Tuesday:
  • Merge the new branch and rebuild the staging site, in the process writing up a full plan for the rebuild/migrate
  • Notify the prod site about the expected downtime
• Thursday:
  • Rebuild/migrate the prod site

[blackshadowshade]

@cgolubi1, just one update to this issue. In the initial post of this issue, I listed related issues that might be affected by the site upgrade. I reckon that we might be able to close some of those issues now. Did you feel like doing so?