search

eQualityTime / Public

1 stars 0 forks source link

Move the WWW stack to AWS #139

Closed joereddington closed 2 years ago

joereddington commented 2 years ago

Dreamhost has been increasingly unpleasent to use.

We've been moving sites off there as and when we could: http://www.projectreal.co.uk/, https://www.whitewaterwriters.com/, OVF's homepage and Flowers for Turing are all hosted on Github

But there is code running in other places: OVF's code, the WWW driver, Supertitle, the blog lists, that prevent us from dropping Dreamhost completely.

There are also things that we want to do that I've been unable to do on Dreamhost (mostly WWW):

So here is what I want:

Resouces

Comands

Main ssh command:

Useful file locations Html files live at /usr/share/nginx/html
Config is at /etc/nginx/nginx.something

ssh -i ~/.ssh/Webserver1-keypair.pem ec2-user@ec2-3-8-118-44.eu-west-2.compute.amazonaws.com

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash curl -sL https://rpm.nodesource.com/setup | sudo bash - curl -sL https://rpm.nodesource.com/setup_10.x | sudo bash - curl -sL https://rpm.nodesource.com/setup_16.x | sudo bash - curl -sL https://rpm.nodesource.com/setup_current.x | sudo bash - git clone http://watchtower.equalitytime.co.uk/Driver/ journalctl -f -u etherpad.service mkdir /etc/nginx/sites-available mkdir server;cd server mysql -uroot -p npm init npm install -g npm@latest npm install express --save-dev nvm install 12.16.3 sudo adduser --home /opt/etherpad --shell /bin/bash etherpad sudo amazon-linux-extras install nginx1 sudo apt-get install nginx sudo curl http://watchtower.equalitytime.co.uk/driver.tar --output driver.tar sudo curl http://watchtower.equalitytime.co.uk/hmp.tar --output hmp.tar sudo install -d -m 755 -o etherpad -g etherpad /opt/etherpad sudo mkdir /etc/nginx/sites-available sudo mysql_secure_installation sudo npm install -g npm@latest sudo su - etherpad sudo systemctl enable etherpad.service sudo systemctl enable mariadb.service sudo systemctl restart httpd sudo systemctl restart nginx sudo systemctl start etherpad.service sudo systemctl start mariadb.service sudo yum install -y nodejs sudo yum install curl vim gcc-c++ make sudo yum install git -y sudo yum install jekyll -y sudo yum install mariadb-server sudo yum install nginx sudo yum install nginx1 sudo yum install php php-common php-opcache php-mcrypt php-cli php-gd php-curl php-mysqlnd sudo yum install ruby -y sudo yum remove nodejs sudo yum remove nodejs -y sudo yum uninstall -y nodejs sudo yum-config-manager --enable remi-php73 systemctl enable etherpad-lite

The code in the original instance:

yum update -y yum install httpd -y service httpd start chkconfig httpd on cd /var/www/html echo "Your Internal IP Address is: " >> index.html curl http://169.254.169.254/latest/meta-data/local-ipv4 >> index.html echo "
Your External IP Address is: "$'\r' >> index.html curl http://169.254.169.254/latest/meta-data/public-ipv4 >> index.html echo "
Your DNS Address is: "$'\r' >> index.html curl http://169.254.169.254/latest/meta-data/public-hostname >> index.html

Plan

joereddington commented 2 years ago

Spent a lot of today on this.

I have now an instance that correctly runs Driver. That includes

That's really good. It effectively entirely replaces the Dreamhost, once I point the domain name in the right place.

However I also went to a lot of effort to get etherpad working. Mostly via this https://www.rosehosting.com/blog/install-etherpad-on-a-centos-7-vps/ page, but also helped by https://stackoverflow.com/a/36340441/170243

I reached "verify the unit started, run journalctl -f -u etherpad.service and you should see something like below:" and very much didn't get that.

Tried https://nodesource.com/blog/running-your-node-js-app-with-systemd-part-1/ but couldn't get that to work either...

Next step is to spin up another instance but without the custom code.

joereddington commented 2 years ago

Did a sprint on this last night.

The problem with "verify the unit started, run journalctl -f -u etherpad.service and you should see something like below:" was that "sudo node" was accessing the wrong version of node.

I ran lots of slight variations of commands, including from this excellent youtube video (stuff at about 3:00) and got that working. I think the one that made the difference was 'sudo yum clean'

The next problem was that the '3000' port that I had opened was only for IP6, and I should have opened it for IP4 and IP6.

I used: 

361  sudo fuser -k 80/tcp
  362  sudo fuser -k 443/tcp
  363  sudo service nginx restart

to stop the httpd server.

Finally, Nginx only started working after I put the config in /etc/nginx/conf.d/ and removed the 'homepage' bit from sudo vim /etc/nginx/nginx.conf The config that I ended up using was https://www.digitalocean.com/community/tutorials/how-to-install-the-etherpad-collaborative-web-editor-on-ubuntu-20-04

joereddington commented 2 years ago

Yesterday's work:

Commands that I might want to remember:

journalctl -xe sudo nginx -t sudo systemctl restart etherpad.service sudo systemctl restart nginx sudo vim /etc/nginx/sites-available/etherpad.conf sudo vim /etc/systemd/system/etherpad.service sudo vim /var/log/nginx/etherpad.access.log sudo vim /var/log/nginx/etherpad.error.log ~

Maddeningly - the issue is now to get nginx to work with php.

joereddington commented 2 years ago 
server {
    listen       80;
    listen       [::]:80;
    server_name  ec2-3-8-118-44.eu-west-2.compute.amazonaws.com;

    access_log  /var/log/nginx/etherpad.access.log;
    error_log   /var/log/nginx/etherpad.error.log;

    index index.html index.htm index.php index.nginx-debian.html;

    location ~ \.php$ {
             fastcgi_pass unix:/var/run/php-fpm/www.sock;
         fastcgi_index index.php;

    }

    location /watchtower {
    root /var/www/html/;
    }

    location /{
    root /var/www/html/whitewaterwriters-site/docs/;
    }

    location /etherpad {
        proxy_pass         http://127.0.0.1:3000;
    rewrite /etherpad/(.*) /$1  break; #from https://serverfault.com/a/379679
        proxy_buffering    off;
        proxy_set_header   Host $host;
        proxy_pass_header  Server;

        # proxy headers
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $remote_addr;
        proxy_set_header    X-Forwarded-Proto $scheme;
        proxy_http_version  1.1;
        # websocket proxying
        proxy_set_header  Upgrade $http_upgrade;
        proxy_set_header  Connection "upgrade";
    }
}
joereddington commented 2 years ago

18/11/21 06:19 to 07:58, Working on +EQT https://github.com/eQualityTime/Public/issues/139

sudo yum install -y nginx php-fpm works for the php but NOT the nginx?

5 Google for nginx and jekyll (or try this: https://gist.github.com/rickharrison/6410194 ) 7 Use the youtube link on a fresh install to work out the php of it all.
8 Back up the key somewhere

18/11/21 09:43 to 11:43, +EQT https://github.com/eQualityTime/Public/issues/139

18/11/21 11:43 to 12:59, +EQT

Okay next things are to set up etherpad (again) and finally fix the jeykll links.

joereddington commented 2 years ago

22/11/21 20:10 to 21:22, Working on +EQT https://github.com/eQualityTime/Public/issues/139

Next morning I asked this quesiton: Okay, via a SE rubberducking here: https://serverfault.com/questions/1084322/how-to-fix-invalidinput-400-record-name-is-not-valid-for-hosted-zone-on-aw/1084325#1084325, I have a working domain name. and self answered it shortly afterwards. Still need to fix 'www.' and 'https::/'

joereddington commented 2 years ago

This note belongs both here and in #140.

This is probably useful later: Installing to: /usr/local/texlive/2021 o

It failed because of lack of space.

Later I used this command: 


[ec2-user@ip-172-31-26-130 install-tl-20211129]$ df -h 
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        484M     0  484M   0% /dev
tmpfs           492M     0  492M   0% /dev/shm
tmpfs           492M  412K  491M   1% /run
tmpfs           492M     0  492M   0% /sys/fs/cgroup
/dev/xvda1      8.0G  8.0G   11M 100% /
tmpfs            99M     0   99M   0% /run/user/1000
[ec2-user@ip-172-31-26-130 install-tl-20211129]$

To confirm usage was 100%

I've since increased the size of the volumne to 32gb, and once that has finnished I'll update.

joereddington commented 2 years ago

IMPORTANT NOTES:

I think the next thing is to watch that youtube video again.

01/12/21 19:25 to 20:15, The AWS. What exactly is the problem? +EQT

01/12/21 20:15 to 20:43, now to latex +EQT

finished with package-specific postactions

Welcome to TeX Live!

See /usr/local/texlive/2021/index.html for links to documentation. The TeX Live web site (https://tug.org/texlive/) contains any updates and corrections. TeX Live is a joint project of the TeX user groups around the world; please consider supporting it by joining the group best for you. The list of groups is available on the web at https://tug.org/usergroups.html.

Add /usr/local/texlive/2021/texmf-dist/doc/man to MANPATH. Add /usr/local/texlive/2021/texmf-dist/doc/info to INFOPATH. Most importantly, add /usr/local/texlive/2021/bin/x86_64-linux to your PATH for current and future sessions. Logfile: /usr/local/texlive/2021/install-tl.log

--- Okay, the important files definately exist in the relevent path --- So I try again

latex]$ /usr/local/texlive/2021/bin/x86_64-linux/tex4ebook book.tex /usr/local/texlive/2021/bin/x86_64-linux/tex4ebook:6: module 'lapp-mk4' not found: no field package.preload['lapp-mk4'] [kpse lua searcher] file not found: 'lapp-mk4' [kpse C searcher] file not found: 'lapp-mk4' [ec2-user@ip-172-31-26-130 latex]$

--- Damint okay,

export PATH="/usr/local/texlive/2021/bin/x86_64-linux:$PATH"

--- That solved the first error, but now I have:

[STATUS] tex4ebook: Conversion started [STATUS] tex4ebook: Input file: book.tex ...021/texmf-dist/scripts/tex4ebook/tex4ebook-exec_epub.lua:105: attempt to index a nil value (local 'm')

Solved by changing the permissions...

However, now pdflatex doens't compile the pdf because it complains about the way that the file is downloaded. Sigh.

02/12/21 06:03 to 07:57, Hello +EQT

joereddington commented 2 years ago

Need to install and connect etherpad onto the current server.

06/01/22 15:42 to 16:50, Install etherpad +EQT

Working on https://www.rosehosting.com/blog/install-etherpad-on-a-centos-7-vps/ and hit

06/01/22 18:59 to 19:29, Etherpad +EQT

I believe I extended the size of the partituion with these commands 1025 du -h 1026 df -h 1027 df -hT 1028 lsblk 1029 sudo growpart /devxvda 1 1030 sudo growpart /dev/xvda 1 1031 lsblk 1032 pwd 1033 df -h 1034 lsblk 1035 df -hT 1036 sudo xfs_growfs -d / 1037 df -hT

The issue now is that www.whitewaterwriters.com/etherpad is sort of working, but the not serving all the (javascript, css) files that would make it really work...

NA: is to write this up as an issue.

joereddington commented 2 years ago

28/01/22 08:28 to 10:57, Working on https://github.com/eQualityTime/Public/issues/139 +EQT

We're fixing a range of bugs in the new version of IMPS.

joereddington commented 2 years ago

As a review - I think we are monitoring this for a couple of weeks and then hopefully closing it.

joereddington commented 2 years ago

Closing.

joereddington commented 2 years ago

Today I closed down the old crons, removed the ability to create camps and delated 19G of pdfs and tar files.

joereddington commented 2 years ago

Bloggged (ish) at https://equalitytime.co.uk/6666/2022/07/28/infrastructure/