loogart / impact-canada

Canadian Government's Privy Council's Office Impact Canada website.
https://loogart.github.io/impact-canada/
0 stars 0 forks source link

Software upgrade Feb 10 #69

Closed djsutter closed 5 years ago

djsutter commented 5 years ago

@loogart @habernation, I have completed a full software upgrade this weekend, and it would be good if you could check the site over asap to make sure everything is ok. In particular, pay attention to new features.

The main upgrade on Saturday went very well. First I upgraded the operating system on all 4 virtual hosts (the jump server, the web/database server, and two instances of the site). Then I did a database backup and performed the software upgrade for the site. All was looking good.

Today (Sunday) I realized that we also need to run the drush "updatedb" command and this is when the problems started - it failed to get through one of the updates, which prevented it from getting through the remaining updates. I tried same on my own host and had the same problem.

In the end, this is what worked: I took a fresh backup of the database today and transferred it to my host, with the previous version of the software (before the Saturday update). Then I ran "drush updatedb" and it completed successfully. Then I moved the database back to the production host and restored it. Then I ran "drush updatedb" again and it applied all the remaining updates successfully.

After all that, I did a restore of the config files and was a bit surprised to see so many updates being applied. This is my concern now, I am hoping that no data/relationships have been lost in the process.

If things don't look good, then we best revert back to Saturday.

Let me know what you see. The site looks ok to me.

habernation commented 5 years ago

Hi Duncan, the site is missing all of the work that was done on Thursday and Friday. This includes a user role for CrushIT challenge jurors and a separate Challenge judging page. All of this is still on the qualivera site. We need these back on the PROD asap - these are pages for challenge jurors in order for them to access applications and evaluate them.

The missing pages are : https://impact.canada.ca/en/challenge-judging-page-dctc https://impact.canada.ca/en/challenge-judging-page-crushit

Please let me know when these have been restored, Mike

On Sun, Feb 10, 2019 at 3:44 PM djsutter notifications@github.com wrote:

@loogart https://github.com/loogart @habernation https://github.com/habernation, I have completed a full software upgrade this weekend, and it would be good if you could check the site over asap to make sure everything is ok. In particular, pay attention to new features.

The main upgrade on Saturday went very well. First I upgraded the operating system on all 4 virtual hosts (the jump server, the web/database server, and two instances of the site). Then I did a database backup and performed the software upgrade for the site. All was looking good.

Today (Sunday) I realized that we also need to run the drush "updatedb" command and this is when the problems started - it failed to get through one of the updates, which prevented it from getting through the remaining updates. I tried same on my own host and had the same problem.

In the end, this is what worked: I took a fresh backup of the database today and transferred it to my host, with the previous version of the software (before the Saturday update). Then I ran "drush updatedb" and it completed successfully. Then I moved the database back to the production host and restored it. Then I ran "drush updatedb" again and it applied all the remaining updates successfully.

After all that, I did a restore of the config files and was a bit surprised to see so many updates being applied. This is my concern now, I am hoping that no data/relationships have been lost in the process.

If things don't look good, then we best revert back to Saturday.

Let me know what you see. The site looks ok to me.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/loogart/impact-canada/issues/69, or mute the thread https://github.com/notifications/unsubscribe-auth/Asab5FZiJUlW0ws7hmWGLeMnfHRtZT8Vks5vMITCgaJpZM4azQ0Q .

djsutter commented 5 years ago

Ok I have reverted the whole thing back to Saturday before the upgrade. Check again to see if things are ok.

djsutter commented 5 years ago

Before all this, this was one of the risks I was concerned about. It's the way the update process works, and apparently it resets the configuration to some state as defined in the software. I'm not at all happy with the way "they" did this, referring to the previous developers. I will have to figure out a new update path that doesn't destroy the new database structure.

habernation commented 5 years ago

Ok- looks like the pages are back now. I'll let you know if I find anything missing, but seems fine.

djsutter commented 5 years ago

Alright thanks, it should be ok now. I did run the "drush dbupdate" (EARLIER) command to bring the database up to current. It should not change any content or structure. I'm thinking that we can bring this database over to the new software release and it should work fine. But I'd rather do that another time, and after some testing. It may define our new process going forward. Basically, NEVER let the automated process upgrade the database.

djsutter commented 5 years ago

On Feb 17, we switched to the new software with the current database. No issues.