Move the OVF stack to AWS

[joereddington]

As per #139 - we are gradually moving services from Dreamhost to Amazon web services. WWW is broadly complete, so it's time to start moving OVF.

Thinking about this.

I'm migrating three things.

• The existing aids.
• The code
• The site.

(B) Clone the main code and run the tests. (C) Set up with a different Domain name - Novareddington.com is legit.

• Include putting in the https - because then we'll look for any problems with the actual proocess (C) Check exactly what the rules currently are for ovf site - dose 'www' get added or removed or what? (C) Confirm that WWW and OVF have the same security system slash ports open. (C) Draft blog post for 'ovf will be having downtime during these hours' (D) Move the site

Steps

• [x] Move the Code
• [x] Move the website
  • [x] setup webserver
  • [x] clone repo
  • [x] Redirect the url
  • [ ] Fix the www. redirect.
• [ ] Update the google ads

[joereddington]

13/01/22 10:39 to 11:55, Working on https://github.com/eQualityTime/Public/issues/148 +EQT

Moving OVF to AWS

• Setup vim. (tried to sort for sudo via https://stackoverflow.com/questions/21488357/use-sudo-with-vimrc but didn't get it working)
• Mapped the issue https://github.com/eQualityTime/Public/issues/148
• Installed nginx, can't seem to get the page to load.
  • Changed the security group, still NO joy.
  • Hmmm. What could be missing?
  • Ah, I hadn't changed the security group. Something had gone wrong.
• Cloned https://github.com/equalityTime/TheOpenVoiceFactory-site/ into the server
• Jekyll is going to be a pain right? It was a pain last time but apparently I fixed it.
  • Tried this script: https://uly.me/install-jekyll-on-amazon-linux-2/

Sprint stack:

• Save the commands and properly write up the issue

Done installing documentation for unicode-display_width, terminal-table, safe_yaml, rouge, forwardable-extended, pathutil, mercenary, liquid, rexml, kramdown, kramdown-parser-gfm, ffi, rb-inotify, rb-fsevent, listen, jekyll-watch, sassc, jekyll-sass-converter, concurrent-ruby, i18n, http_parser.rb, eventmachine, em-websocket, colorator, public_suffix, addressable, jekyll after 46 seconds 27 gems installed Ignoring bigdecimal-1.3.2 because its extensions are not built. Try: gem pristine bigdecimal --version 1.3.2 Ignoring io-console-0.4.6 because its extensions are not built. Try: gem pristine io-console --version 0.4.6 Ignoring json-2.0.4 because its extensions are not built. Try: gem pristine json --version 2.0.4 Ignoring openssl-2.0.9 because its extensions are not built. Try: gem pristine openssl --version 2.0.9 Ignoring psych-2.2.2 because its extensions are not built. Try: gem pristine psych --version 2.2.2 jekyll 4.2.1 | Error: Permission denied @ dir_s_mkdir - /usr/share/nginx/html/myblog [ec2-user@ip-172-31-34-189 html]$

Nope, tried those commands and got nothing

NA: ssh into old server and see if the jekyll commands are available

[joereddington]

[ec2-user@jekyll ~]$ history 1 sudo yum install git 2 ls 3 pwd 4 git clone https://github.com/joereddington/dotvimdirectory 5 mv dotvimdirectory/ .vim 6 ls 7 cp .vim/vimrc .vimrc 8 vim hope.txt 9 cd .vim/ 10 ls 11 cd bundle/ 12 ls 13 cd .. 14 more setup.sh 15 cd .. 16 rm .vimrc 17 cd .vim/ 18 ./setup.sh 19 ls 20 sudoedit vimrc 21 cd .. 22 ls 23 sudo -E vim vimrc 24 ls 25 sudo -E vim hope.txt 26 git 27 pwd 28 ls 29 rm hope.txt 30 sudo apt-get install nginx 31 sudo yum install nginx 32 sudo amazon-linux-extras install nginx1 33 cd /usr/share/nginx/html/ 34 ls 35 more index.html 36 pwd 37 sudo vim /etc/nginx/nginx.conf 38 sudo systemctl status nginx.service 39 sudo systemctl start nginx.service 40 sudo systemctl status nginx.service 41 ls -lt 42 sudo chmod 777 index.html 43 sudo git clone https://github.com/equalityTime/TheOpenVoiceFactory-site/ 44 ls 45 jekyll 46 history 47 history >historysofar.txt 48 sudo history >historysofar.txt 49 history >~/historysofar.txt 50 sudo yum install ruby 51 sudo vim jeykllinstall.sh 52 sudo chmod a+x jeykllinstall.sh 53 ./jeykllinstall.sh 54 pwd 55 ls 56 cd TheOpenVoiceFactory-site/ 57 jekyll build 58 gem pristine bigdecimal --version 1.3.2 59 sudo gem pristine bigdecimal --version 1.3.2 60 sudo jekyll build 61 jekyll build 62 sudo gem pristine io-console --version 0.4.6 63 sudo gem pristine json --version 2.0.4 64 jekyll build 65 sudo gem pristine psych --version 2.2.2 66 jekyll build 67 sudo gem pristine openssl --version 2.0.9 68 jekyll build 69 bundle exec jekyll build 70 sudo bundle exec jekyll build 71 bundle install 72 sudo bundle install 73 cd .. 74 jekyll new myblog 75 sudo jekyll new myblog 76 exit 77 history

[joereddington]

There are the missing ifles /root/.bash_history:sudo jekyll build /root/.bash_history:sudo gem install jekyll bundler /root/.bash_history:sudo bundle exec jekyll build /root/.bash_history:jekyll build /root/.bash_history:gem install jekyll bundler /root/.bash_history:gem install jekyll /root/.bash_history:gem install jekyll /root/.bash_history:bundle exec jekyll build /home/ec2-user/.bash_history:sudo jekyll build /home/ec2-user/.bash_history:sudo bundle exec jekyll build /home/ec2-user/.bash_history:jekyll build /home/ec2-user/.bash_history:bundle exec jekyll build

But none of them worked.

Part of the problem was that the ruby version was too low. I bumped it up to 3 with.

sudo amazon-linux-extras install ruby3.0

but that still didn't work until I did. something else that seemed normal. It's now building. Here's the rest of the commands? 163 cd TheOpenVoiceFactory-site/ 164 ls 165 touch hope.txt 166 jeykll build 167 jekyll build 168 bundle exec jekyll build 169 bundle install 170 sudo gem install http_parser.rb -v '0.6.0' --source 'https://rubygems.org/' 171 bundle install 172 ls 173 vim Gemfile 174 rm Gemfile.lock 175 jekyll build 176 gem install ffi 177 sudo gem install ffi 178 sudo amazon-linux-extras install ruby-dev 179 sudo amazon-linux-extras list | grep ruby 180 gem install aws-skd 181 gem install aws-sdk 182 sudo gem install ffi 183 jekyll build 184 gem pristine ffi --version 1.15.5 185 sudo gem pristine ffi --version 1.15.5 186 gem uninstall ffi 187 gem uninstall -i ffi 188 sudo gem uninstall ffi 189 sudo gem pristine ffi --version 1.15.5 190 sudo gem pristine ffi 191 sudo gem install ffi 192 ruby 193 sudo yum install ruby-devel 194 sudo gem install ffi 195 jekyll build 196 bundle install 197 jekyll build 198 bundle exec jekyll build 199 ls 200 ls _site/ 201 ls 202 history

[joereddington]

I'm setting up Novareddington.com as a test.

To register a domain on Amazon.

• Go to route 53
• Create a new hosted zone
• Keep track of the nameserves ns-1297.awsdns-34.org. ns-515.awsdns-00.net. ns-1947.awsdns-51.co.uk. ns-180.awsdns-22.com.
• Assign an elastic IP address to the instance https://www.youtube.com/watch?v=h-yOoHbH_Dw
• Copy the new IP address
• In route 53 create an 'A' record pointing to that IP.
• Put the nameservers into the domain register
• Wait, and see if you can ssh in.

[joereddington]

02/02/22 12:23 to 16:45, Work on https://github.com/eQualityTime/Public/issues/148 +EQT

• Been editing Nginx and route 53. There are now two subdomains that successfully point to folders on the instance.

• Checked out the code into one of those folders.

• Decided NOT to test the designs first but to put the code in and generate

• Realised that I actually did the task already, but I'm pushing through because I think I could really crack this.

• The tests all run (with sudo, ten fail otherwise)

• Installed version 8.0 (www is on 5.4 which is what https://www.thegeekstuff.com/2013/12/nginx-php-fpm/ wanted) Also didn't have to change much config

  306 02/02/22 14:36:48 sudo amazon-linux-extras install php8.0 307 02/02/22 14:37:03 php --version 309 02/02/22 14:38:08 sudo vim /etc/php-fpm.d/www.conf #this was find 313 02/02/22 14:39:32 sudo service php-fpm restart 314 02/02/22 14:39:46 sudo vim /etc/nginx/nginx.conf 315 02/02/22 14:40:14 sudo nginx -t 316 02/02/22 14:40:18 sudo systemctl restart nginx [ec2-user@jekyll TheOpenVoiceFactory]$

• Now fixing the code. I am ready for testing!

  • First time the server hadn't refreshed for the new code.
  • Second time there was a 413 error that I fixed in the nginx code. * Third time I got an error in the upload.php
  • This turned out to be that php has file size limits -fixed with https://stackoverflow.com/a/27192946/170243
  • Four error is that the file won't move - that turns out to be persmissions
  • It works! Sort of! I need to set up https, but I'm quite pleased.

Doing Https

IMPORTANT NOTES:

• Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/novareddington.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/novareddington.com/privkey.pem Your certificate will expire on 2022-05-03. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew all of your certificates, run "certbot renew"

• If you like Certbot, please consider supporting our work by:

  Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le

[ec2-user@jekyll ~]$

I used the /usr/share/nginx/html/TheOpenVoiceFactory-site/_site webroote

I had to follow lots of advice on this page: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SSL-on-amazon-linux-2.html

Here's the commands I used:

439 02/02/22 16:04:35 sudo wget -r --no-parent -A 'epel-release-.rpm' https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/ 440 02/02/22 16:04:42 sudo rpm -Uvh dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-.rpm 441 02/02/22 16:04:48 sudo yum-config-manager --enable epel* 442 02/02/22 16:04:53 sudo yum repolist all 443 02/02/22 16:05:14 sudo amazon-linux-extras install epel -y 444 02/02/22 16:05:38 sudo yum install -y certbot python2-certbot-apache 454 02/02/22 16:06:18 sudo certbot certonly

[joereddington]

It is moved.

09/05/22 06:14 to 07:31, Work on OVF migration +EQT

• I started the process but there were problems because qiq didn't let us update the nameservers. I rased a ticket.
• Fixed a bug on something lese.
• Dealt with some emails.
• Novareddington.com kept working - but now is randomly redirecting to www. Which is super interesting. I went back and deleted the hosted zone.
• Ran the update script for tasks.

09/05/22 09:58 to 10:15, Working out day +EQT

IMPS3 and meeting are the things I'm excited abut but...

09/05/22 10:15 to 10:50, +EQT migration of OVF

• The QIQ people updated the nameservers.
• First thing to fix was the TLS certificates.
  • I used certbot with this webroot

/usr/share/nginx/html/TheOpenVoiceFactory-site/_site

Which worked and I got this

IMPORTANT NOTES:

• Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/theopenvoicefactory.org/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/theopenvoicefactory.org/privkey.pem Your certificate will expire on 2022-08-07. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew all of your certificates, run "certbot renew"

09/05/22 10:51 to 11:55, +EQT migration of OVF

• Fix the novareddington.com redirect (two actually) +EQT

• I made the changes to the html but jekyll wouldn't build without 757 09/05/22 09:56:56 gem install bundler -v 2.3.5 758 09/05/22 09:57:00 jekyll build

• OVF can now set up new aids.

• I have switched off the github pages version

• Some of the old pageset files are missing. Upon proper investigation it turns out that a large proportion of pageset.obz files didn't make the transfer.

  • 17 gig against 3.7