arendvw commented 5 years ago

Here are my notes for installing speckle on a vps like server, using Debian 9.

I'm not sure if these instructions should be part of a readme, an install file, or if this should be a general advice. This is how I did it, this may also benefit others.

This document assumes you have DNS setup for speckle.example.com

0. Setup `sudo`

This document assumes all commands are done from a non-root user. Make sure you have a non root user, and this user is allowed to use sudo. We install & configure nano as the default editor.

apt install sudo nano
update-alternatives --set editor /usr/bin/nano
adduser yourname
sudo visudo
yourname ALL=(ALL:ALL) ALL

The rest of this documentation assumes you're signed in as yourname

1. Install `ufw`

Setup ufw (Uncomplicated firewall)

sudo apt install ufw

# Just to be sure, disable first.
sudo ufw disable 
sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https
sudo ufw default deny incoming
sudo ufw enable

Setup fail2ban (Bans ips that try to login too many times for 15 minutes)

sudo apt install fail2ban

3. Install `nodejs`

Add nodejs apt repository

curl -sSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo apt-key add -
VERSION=node_11.x
DISTRO="$(lsb_release -s -c)"
echo "deb https://deb.nodesource.com/$VERSION $DISTRO main" | sudo tee /etc/apt/sources.list.d/nodesource.list
echo "deb-src https://deb.nodesource.com/$VERSION $DISTRO main" | sudo tee -a /etc/apt/sources.list.d/nodesource.list
apt update
apt install nodejs

Install speckle

sudo apt install mongodb redis-server 

# Add speckle user
sudo adduser speckle
# Disable login for this user
passwd -l speckle
cd /home/speckle
sudo -u speckle git clone https://github.com/speckleworks/SpeckleServer.git
cd SpeckleServer

sudo -u speckle npm install

cd plugins
sudo -u speckle git clone https://github.com/speckleworks/SpeckleAdmin.git
sudo -u speckle git clone https://github.com/speckleworks/SpeckleViewer.git

Update configuration

cd /home/speckle
sudo cp .env-base .env
nano .env

# Change listen ip to 127.0.0.1
# Change url to https://speckle.example.com
# Test the speckle server to see if it will start
cd SpeckleServer
sudo -u speckle node server.js

You can close/kill the node server again, because we will use pm2 in the next steps.

TODO: Add documentation to allow the nodjes server only write to the folders it needs to write to? (E.g. logs?)

Install pm2 process manager

This will start the node server again if it crashes, and makes sure it will start on boot.

cd /home/speckle
sudo npm install -g pm2
sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u speckle --hp /home/
sudo -u speckle pm2 start server.js

Install nginx as a reverse proxy

sudo apt install ssl-cert nginx
nano /etc/nginx/sites-available/speckle

Add this config to the file

server {
    listen 80;
    server_name speckle.example.com;
    location /.well-known {
            alias /home/speckle/.well-known;
    }

    location / {
        # redirect to https
        return 301 https://$host$request_uri;
    }
}

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

# theoretically this can be a range of severs to facilitate 
# horizontal scaling.
upstream speckleserver {
    server 127.0.0.1:3000;
}

server {
    # And: as far as I know websockets are not supported by http2 yet.
    listen 443 ssl;

    server_name speckle.example.com;

    # The snake-oil certificate is loaded in the first place, so the letsencrypt
    # process is easier.
    # Once you have requested the certbot certificate you can remove these certificates
    ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
    ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;

    #ssl_certificate /etc/letsencrypt/live/speckle.example.com/fullchain.pem;
    #ssl_certificate_key /etc/letsencrypt/live/speckle.example.com/privkey.pem;

    ssl_stapling on;
    ssl_stapling_verify on;

    access_log /var/log/nginx/speckle.log combined;

    # maintain the .well-known directory alias for renewals
    location /.well-known {
        alias /home/speckle/.well-known;
    }

    # This part might need a bit of tuning later on.
    # I'm not 100% sure about the wide range of timeout options available.
    location / {
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_pass http://speckleserver;
        proxy_read_timeout 86400;
    }
}

Enable the config file:

sudo ln -s /etc/nginx/sites-available/speckle /etc/nginx/sites-enabled/speckle
sudo systemctl restart nginx

Your speckle server should now be reachable through http and https on port 80 and port 443.

Install SSL certificate with letsencrypt

echo "deb http://http.debian.net/debian stretch-backports main contrib non-free\n" | sudo tee /etc/apt/sources.list.d/stretch-backports.list
sudo apt-get install certbot -t stretch-backports
sudo certbot certonly --webroot -w /home/speckle -d speckle.example.com

Replace the commented lines in the nginx config file (ssl_certificate, ssl_certificate_key)

 nano /etc/nginx/sites-available/speckle
 sudo systemcl restart nginx

didimitrie commented 5 years ago

Wow, hello @arendvw! Really good notes, big shout for the fail2ban reference too!!! I can't believe I've missed this for more than a month 😞

It's actually a good question where to put these; i think maybe in the repo wiki and we could link to it from the readme in a discrete but proper way (i got burned - as you might've noticed given a recent pr to said readme).

ionutanton commented 5 years ago

@arendvw Great doc. i did a good and clean install of a speckle server on a VM under windows 10. i am a linux noob i just copy paste commands and a few commands were a little hard to follow or implement. i found the following guides helpfull: for installing nodejs on debian https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-debian-9 for installing backports on debian https://backports.debian.org/Instructions/

also on my debian systemcl command is in fact systectl and on the ngix config file i fount the folllowing typos: server_name speckle.example.com - shoul be - server_name speckle.example.com;

ssl_certificate_key /etc/letsencrypt/speckle.example.com/privkey.pem; - should be - #ssl_certificate_key /etc/letsencrypt/live/speckle.example.com/privkey.pem;

proxy_pass http://speckle_servers; - should be - proxy_pass http://speckleserver;

Otherwise great work.

arendvw commented 5 years ago

It's actually a good question where to put these; i think maybe in the repo wiki and we could link to it from the readme in a discrete but proper way

@didimitrie I would suggest to keep the documentation in the github repository, so you can easily keep the documentation up to date in github, and use something like readthedocs.io to automatically generate html/searchable documentation if needed. It makes it easier to "move fast and break things", since you can easily create versioned documentation from git tags.

That being said, I agree with you that this may be too exaustive for a README. Especially since this is specifically written for a VPS, and I can imagine that a seperate documentation for development versions on Windows WSL, under OSX or for docker/kubernetes can have their own readme's.

In other project I've found the distiction between README, INSTALL.{platform}, and UPGRADE_XX_YY quite useful. But perhaps that's too legacy unix-y, and can also be opted for a docs/ folder with humable readable filenames.

arendvw commented 5 years ago

@ionutanton Great work! I've updated my comment on the top with the typo's. Thanks for the write up.

My only question is about your links for documentation: In most cases I added these apt scripts as commands, since I generally don't like to download bash scripts and execute them as root user, this is why I've added them as commands. Perhaps you can share what went wrong with the backports and nodejs scripts?

markcichy commented 5 years ago

@arendvw I would add that if people are installing in VM (Hyper-V or EXSi), and are doing so far a large organization in production, they should setup a user specifically dedicated to Speckle (as you have done, not sure that was documented) with the correct perms. I would be careful giving this user perms to sudo or root. These are the guides I've used (shared via Slack too) and found extremely helpful -- although they are written for Ubuntu Server, which is in effect Debian 'X':

Download/install Ubuntu Server 18.05.2 LTS (although it may have a higher overhead - about 3GB installed - it does have many useful monitoring tools out of box: htop, ifconfig, ufw, etc.) https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-18-04
Best to edit your /etc/ssh/sshd_config to change default port immediately (prior to UFW config)
UFW setup and port config https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-18-04
MongoDB from apt repo (seems to work best with Speckle) https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-18-04
Redis Server with added security https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-18-04
NodeJS and NPM install, I used the PPA repo and seemed to get better results as it install 10.x as opposed to the 8.x that is available in the APT repo https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-18-04

The rest of your instructions RE Nginx and certs are great! The one note I might add is that the Server Block could be significantly reduced (sample below, mostly same as yours):

SSL server block

server {

    # ssl
    listen  443 ssl  http2;
    listen [::]:443 ssl http2;

    server_name speckle.example.com;

    location / {
            proxy_pass http://ipaddr-speckleserver:3300;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
    }

ssl_certificate /etc/letsencrypt/live/speckle.example.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/speckle.example.com/privkey.pem; # managed by Certbot

}

http redirect server block

server {

if ($host = speckle.example.com) {
    return 301 https://$host$request_uri;
} # managed by Certbot

listen 80;
listen [::]:80;

server_name speckle.example.com;
return 404; # managed by Certbot

}

ionutanton commented 5 years ago

@arendvw I had a problem with the command curl -sSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo apt-key add - it just said OK and the next part i didnt quite understand what to do with it (version= ...). that is why i used the script in the link i provided. and for backports after the echo that writes to the list you need an apt-get update. otherwise all good.

arendvw commented 5 years ago

@markcichy the digitalocean setup tutorial is great. I've added my own 'poor' version of this, but referencing this and taking it as a starting point seems great.

Different port for ssl I know its popular advice, but I don't really believe changing ports will make stuff more secure. Perhaps you will filter out lots of the automated requests. I would highly prioritize not allowing root users and only allowing public key authentication over changing ports.

Unprivileged user The user comment is great. Except I would always do it, not just in "large organisations". In step 3 a non-root, non-sudo user is added for speckle.

UFW UFW Guide, seems like a good starting guide for learning ufw.

Mongo The mongo guide is a bit verbose, but a good reference for using systemctl. (E.g. sudo apt install -y mongodb ) will work for most people out of the box. Perhaps it's a good idea to add a check if mongo is only bound to the local ip.

Nginx Thanks for the updated version of nginx, I've added parts for what I assumed was needed for what I assumed was needed for websocket support, but perhaps this is not needed at all? This was also why I did not add http2 support.

Nginx SSL Will nginx start if an ssl certificate does not exist? I know apache does not.

I added the snake oil certificates to make sure the nginx config will work tutorial style. (Otherwise you have this chicken-egg problem with certbot that need a working nginx config to authenticate, and nginx needs an elasticsearch cerficate to work to start.

markcichy commented 5 years ago

Yes, @arendvw ! Nginx will indeed startup without the cert -- browser will complain initially, but once you create your certs and add them to the Block all is well. Changing SSH port just cuts out bot scanning -- a good start IMHO.

logan12358 commented 5 years ago

These instructions seem pretty detailed and thorough :)

I agree that a Wiki isn't the ideal place for instructions like this. But I also think that they don't all quite fit in the repo. This is because a lot of the instructions are particular to dependency installation and nginx configuration. These are things where the best practice could easily change outside of our repo, and the instructions would then need to be maintained. It also might give the impression that we're vouching for the security and reliability of a system set up in this way, when in reality there are more considerations (updates, backups, password strength/key security).

As a (much larger, but) comparable project, the official NextCloud documentation only provides high level instructions for installing dependencies. They also provide some high level hardening and security guidance.

I think that the current instructions in the readme cover all of the Speckle-specific steps, and everything else is probably out of scope of any officially maintained documentation. We could perhaps add a list with some high level security guidance.

I think that a good place to put instructions like these would be as a blog post over here, with a caveat like "These instructions were tested and worked as of [date], but we can't guarantee they'll work in the future".

arendvw commented 5 years ago

It also might give the impression that we're vouching for the security and reliability of a system set up in this way, when in reality there are more considerations (updates, backups, password strength/key security).

Is this not already solved by the MIT Licence? This contains a very clear notice this software is delivered AS IS. Adding a notice that following these steps does not guarantee the safety of your server if of course a good idea.

Nextcloud only provides high level instructions for installing dependencies.

I think it's an excellent outline for an installation document, but did you read the whole document?

The document you referenced contains:

A reference on where to learn further security concepts and knowledge that is assumed before installing the server
A list of which php modules are required for installation
Different installation and configuration instructions for dependencies. (PHP, Apache, MariaDB, Redis) for CentOS and Ubuntu
Permission instructions (Apache user is used, not a seperate user) and SELinux dependencies
Firewall AND Selinux instructions
Both a web based installation wizard and a commandline installation scripts (with validation of previous steps and permissions)
Apache2 Config and Nginx Config
SSL Configuration
Notes on using PHP FPM (php's cousin of the likes of pm2)
Notes on it's performance and how to tune it.

Also note keep in mind:

All file system / file permissions in nextcloud are scripted. Which folders should be writable by the webserver, and which not? (This is missing for specke). Even if nothing needs to be world writeable, it should be documented.
The comparison is not completely fair, because PHP by definition will run under apache as an unpriviledged user, and nodejs will run once as whatever user you start it with, and when it crashes it's the end.

I think that the current instructions in the readme cover all of the Speckle-specific steps

I cannot disagree more. It took me a day to figure out how to get this to work in a way that any other project I've ever installed has decent documentation for. I would argue that getting this project to work in production does not have to be masochistic undertaking only for seasoned nodejs devops guru's.

Yet all the current readme has on this front is the cynical and non-encouraging notice that if you wish to use it in production and don't get it, it's because you're confused and stupid and should find and ask someone smarter.

didimitrie commented 5 years ago

Sorry if the instructions seem masochistic. They're not meant to be. But, so far, their terseness comes from several reasons, which are mentioned in the PR that modified them. And it seems they had the desired effect so far (with the not desired effect of annoying you @arendvw 😬, but there's a price for everything i guess...).

Is this not already solved by the MIT Licence?

Legally yes, practically no: see, for example, this tweet. I've personally received more feedback and discovering more every day (if i'm lucky I get to hear about it). Other anecdotal stories (but bear in mind i have to deal with this stuff!):

workshops with self-hosted speckle servers that crashed continuously (result: speckle sucks),
rumours, like the above tweet, regarding speckle's security due to bad mongo deployments (result: speckle sucks),
over-priced infrastructure provisioning (result: speckle is expensive and sucks),
a million unstable/insecure/etc in-house deployments at larger companies (including where I'm working at now). Results:
- personal (gentle) spanking from above
- semi-cold war with IT (thawed after much effort)
- warnings from lawyers 👀
- frustrated users that I can't help (A: The server is down! Me: Uh... no? A: Yes it is! Me: Which server? A: The speckle server! ... result: after half an hour of debugging potential company specific firewalls and potentially non-renewed ssl certs on the test server, almost opening a ticket on digital ocean re shoddy networking, the dude who deployed speckle on his personal computer went on holiday and didn't leave his computer on)

The above rant aside, this boils down to a very simple fact: the speckle community, so far, does not have the resources to maintain this level of detail in a potentially dangerous set of instructions. Even more, we can't even maintain basic docs about the core plugins. Not to mention the online clients. Or the sdk. We just have a bunch of old docs that I and a few others wrote ages ago. Nevertheless, with positive contributions like yours, and the rest from the above, and as the project matures, my hope is that this will change for the better.

Next steps (as i see them):

blog post or discourse post on detailed server installation instructions
link the above in the readme
the title should contain development guide in some sort or form, unless we want shoot ourselves in the leg with a free SLA that we can't cover

As an aside, I'm trying to transition the speckle website to something more approachable and editable, with less of a headache inducing front page, and with a proper CD pipeline (right now it's broken, and before it relied on a cron git pull), so perhaps the discourse option is safer. Happy for either options though.

arendvw commented 5 years ago

@didimitrie @logan12358 This makes sense. I can imagine your reluctance in endorsing installation instructions.

I'm willing to maintain a living document on the discourse if that helps you in the short term from endorsing installation instructions and or implied SLA's.

didimitrie commented 5 years ago

@arendvw, thanks, that would be absolutely lovely. Thinking a bit back, i do believe i was a bit drastic in the readme refresh, and didn't allow for a graceful "migration" (ie, move the old instructions somewhere and link to them). Once you've done this, and redeemed my above sin, do throw a PR with the link to it.

🙇

Discourse's also connected to the slack channel, so it gets quite a bit of visibility too.

arendvw commented 5 years ago

Ok, I've editted it to take into account the comments here and posted it on discourse

speckleworks / SpeckleServer

Installation of speckle on Debian 9 on a VPS with PM2, nginx and letsencrypt #126

0. Setup `sudo`

1. Install `ufw`

3. Install `nodejs`

Install speckle

Update configuration

Install pm2 process manager

Install nginx as a reverse proxy

Install SSL certificate with letsencrypt

ssl_certificate_key /etc/letsencrypt/speckle.example.com/privkey.pem; - should be - #ssl_certificate_key /etc/letsencrypt/live/speckle.example.com/privkey.pem;

SSL server block

http redirect server block

speckleworks / SpeckleServer

Installation of speckle on Debian 9 on a VPS with PM2, nginx and letsencrypt #126

0. Setup sudo

1. Install ufw

3. Install nodejs

Install speckle

Update configuration

Install pm2 process manager

Install nginx as a reverse proxy

Install SSL certificate with letsencrypt

ssl_certificate_key /etc/letsencrypt/speckle.example.com/privkey.pem; - should be - #ssl_certificate_key /etc/letsencrypt/live/speckle.example.com/privkey.pem;

SSL server block

http redirect server block

0. Setup `sudo`

1. Install `ufw`

3. Install `nodejs`