etesync / server

The Etebase server (so you can run your own)
https://www.etesync.com
GNU Affero General Public License v3.0
1.53k stars 75 forks source link

I have created an automated installer script for debian #42

Closed BeatLink closed 3 years ago

BeatLink commented 4 years ago

This script will create an etesync user, install etesync to /opt/etesync and add a service to systemd while enabling it. It should be a good start but further configuration is recommended (for example, limiting the allowed hosts that etesync will be authorized on)

# Set needed variables
ETESYNC_HOME=/opt/etesync;
ETESYNC_GITHUB=https://github.com/etesync/server.git;
ETESYNC_APP_FOLDER=$ETESYNC_HOME/server;

# Setup etesync user 
sudo groupadd etesync;
sudo useradd -m -d $ETESYNC_HOME etesync -g etesync;
sudo mkdir -p $ETESYNC_HOME;
sudo chown -R etesync:etesync $ETESYNC_HOME;

# Install Etesync
sudo apt install -y python3-venv git;
sudo -u etesync bash << EOF
    git clone $ETESYNC_GITHUB $ETESYNC_APP_FOLDER;
    python3 -m venv $ETESYNC_APP_FOLDER/venv;
    . $ETESYNC_APP_FOLDER/venv/bin/activate;
    python3 -m pip install wheel;
    python3 -m pip install gunicorn;
    python3 -m pip install -r $ETESYNC_APP_FOLDER/requirements.txt;
    cp $ETESYNC_APP_FOLDER/etesync-server.ini.example $ETESYNC_APP_FOLDER/etesync-server.ini;
    sed -i 's/allowed_host1 = example.com/allowed_host1 = */g' $ETESYNC_APP_FOLDER/etesync-server.ini;
    sed -i "s@secret_file = secret.txt@secret_file = $ETESYNC_APP_FOLDER/secret.txt@g" $ETESYNC_APP_FOLDER/etesync-server.ini;
    python3 $ETESYNC_APP_FOLDER/manage.py migrate;
EOF

# Setup static files
sudo mkdir -p /srv/http/etesync_server/static;
sudo chown -R etesync /srv/http/etesync_server;
sudo -u etesync bash << EOF
    ln -s /srv/http/etesync_server/static $ETESYNC_APP_FOLDER/static;
    python3 $ETESYNC_APP_FOLDER/manage.py collectstatic;
EOF

# Create the Systemd Service
sudo tee /lib/systemd/system/etesync.service > /dev/null << EOF
[Unit]
Description=Etesync gunicorn server

[Service]
User=etesync
Group=etesync
WorkingDirectory=$ETESYNC_APP_FOLDER
ExecStart=$ETESYNC_APP_FOLDER/venv/bin/gunicorn -b 0.0.0.0 etesync_server.wsgi:application
#Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target

EOF

# Enable and Start the service
sudo systemctl daemon-reload;
sudo systemctl enable etesync.service;
sudo systemctl start etesync.service;

TODO: Create a pull request to merge it into the repo

TODO: Create an update script to complement this one as well

BeatLink commented 4 years ago

An improved version of the script. It will be able to install a production ready version of etesync on debian using nginx and uwsgi.

# Set needed variables ------------------------------------------------------------------
export ETESYNC_HOME=/opt/etesync;
export ETESYNC_GITHUB=https://github.com/etesync/server.git;
export ETESYNC_APP_FOLDER=$ETESYNC_HOME/server;
export STATIC_ROOT=/srv/http/etesync_server/static;

# Setup etesync user --------------------------------------------------------------------
sudo useradd -M etesync -s /bin/bash;
sudo mkdir -p $ETESYNC_HOME;
sudo chown -R etesync $ETESYNC_HOME;
sudo usermod -d $ETESYNC_HOME etesync;

# Install Etesync -----------------------------------------------------------------------
sudo apt install -y python3-venv git;
sudo -u etesync bash << EOF
cd $ETESYNC_HOME
git clone $ETESYNC_GITHUB $ETESYNC_APP_FOLDER;
python3 -m venv $ETESYNC_APP_FOLDER/venv;
source $ETESYNC_APP_FOLDER/venv/bin/activate;
python3 -m pip install wheel;
python3 -m pip install -r $ETESYNC_APP_FOLDER/requirements.txt;
EOF

# Configure .ini File -------------------------------------------------------------------
sudo -u etesync tee $ETESYNC_APP_FOLDER/etesync-server.ini > /dev/null << EOF
[global]
secret_file = $ETESYNC_APP_FOLDER/secret.txt
debug = false
static_root = $STATIC_ROOT
static_url = /static/
language_code = en-us
time_zone = America/Jamaica

[allowed_hosts]
allowed_host1 = *

[database]
engine = django.db.backends.sqlite3
name = db.sqlite3
EOF

# Perform migration ---------------------------------------------------------------------
sudo -u etesync bash << EOF
cd $ETESYNC_APP_FOLDER
source $ETESYNC_APP_FOLDER/venv/bin/activate;
python3 $ETESYNC_APP_FOLDER/manage.py migrate
EOF

# Setup static files --------------------------------------------------------------------
sudo mkdir -p $STATIC_ROOT
sudo chown -R etesync $STATIC_ROOT
sudo -u etesync bash << EOF
cd $ETESYNC_APP_FOLDER
source $ETESYNC_APP_FOLDER/venv/bin/activate;
python3 $ETESYNC_APP_FOLDER/manage.py collectstatic
EOF

# Setup uwsgi ---------------------------------------------------------------------------
sudo apt install -y uwsgi uwsgi-plugin-python3;
sudo mkdir -p /etc/uwsgi/sites/;
sudo tee /etc/uwsgi/sites/etesync.ini > /dev/null << EOF
[uwsgi]
socket = $ETESYNC_APP_FOLDER.sock
chown-socket = etesync:www-data
chmod-socket = 660
vacuum = true
plugins = python3

uid = etesync
chdir = $ETESYNC_APP_FOLDER
home = %(chdir)/venv
module = etesync_server.wsgi
master = true
EOF

# Setup ssl keys ------------------------------------------------------------------------
sudo -u etesync bash << EOF
mkdir $ETESYNC_HOME/ssl_keys;
openssl req -x509 -newkey rsa:4096 -keyout $ETESYNC_HOME/ssl_keys/key.pem -out $ETESYNC_HOME/ssl_keys/cert.pem -days 365 -nodes -batch;
EOF

# Setup nginx ---------------------------------------------------------------------------
sudo apt install -y nginx;
sudo tee /etc/nginx/sites-available/etesync-server.conf > /dev/null << EOF
server {
    server_name heimdall;

    listen 8000 ssl http2;
    listen [::]:8000 ssl http2;

    root /srv/http/etesync_server;

    client_max_body_size 5M;

    location /static {
        expires 1y;
        root /srv/http/etesync_server;
    }

    location / {
        uwsgi_pass      unix:$ETESYNC_APP_FOLDER.sock;
        include         uwsgi_params;
    }

    ssl_certificate $ETESYNC_HOME/ssl_keys/cert.pem;
    ssl_certificate_key $ETESYNC_HOME/ssl_keys/key.pem;
}
EOF
sudo ln -s /etc/nginx/sites-available/etesync-server.conf /etc/nginx/sites-enabled/etesync-server.conf
sudo systemctl enable nginx;
sudo systemctl start nginx;

# Setup systemd services ----------------------------------------------------------------
sudo tee /etc/systemd/system/uwsgi.service > /dev/null << EOF
[Unit]
Description=uWSGI Emperor service

[Service]
ExecStart=uwsgi --emperor /etc/uwsgi/sites
Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable uwsgi.service
sudo systemctl start uwsgi.service
tasn commented 4 years ago

Cool, nicely done! I guess maybe it belongs to the wiki or something. What do you think @daftaupe?

Also, I think it's better to have the service file and configuration in separate files (that can be copied by the script) rather than all in the script. I get the allure of having a stand-alone script, but there's also quite a bit of benefit with not having to maintain a few versions of the same script in the repo, one for each distro/example/etc.

daftaupe commented 4 years ago

Thanks for the script @BeatLink !

I think that the uwsgi .service file is not needed (distributions should ship them already, I've check on Arch and that's the case : https://git.archlinux.org/svntogit/community.git/tree/trunk/emperor.uwsgi.service?h=packages/uwsgi, emperor mode is pretty popular from what I've seen). Especially because it's not EteSync specific.

This script would fit in the Wiki as en example of an automated deployment for Debian-like environments maybe ? In my opinion, the repo shouldn't contain config files or other resources related to pieces of software that are not part of the project, so the directory called example-configs/nginx-uwsgi should be moved to the Wiki somehow (if not already there). What do you guys think ?

tasn commented 4 years ago

I'm on the fence, though leaning more towards having things in repo to be honest. The wiki tends to get stale, especially when it comes to code snippets, when people have to remember to update it. While if things are in the repo, it's more in your face. Also, I consider these config files like code, so having version tracking (better than wiki) for them is a nice plus too.

I'm really on the fence though, not sure one way or the other.

lucollab commented 4 years ago

the script from BeatLink works great but gives me a little error in the end:

● uwsgi.service - uWSGI Emperor service
   Loaded: error (Reason: Invalid argument)
   Active: active (exited) since Fri 2020-05-01 18:40:04 CEST; 2min 12s ago
   CGroup: /system.slice/uwsgi.service

Mai 01 18:40:04 meeting uwsgi[2772]: Starting app server(s): uwsgi (omitted; missing conffile(s) in /etc/uwsgi/apps-enabled).
Mai 01 18:40:04 meeting systemd[1]: Started LSB: Start/stop uWSGI server instance(s).
Mai 01 18:42:04 meeting systemd[1]: [/etc/systemd/system/uwsgi.service:5] Executable path is not absolute, ignoring: uwsgi --
Mai 01 18:42:04 meeting systemd[1]: uwsgi.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.

could this help? https://stackoverflow.com/questions/55444108/python-flask-service-file-is-not-starting

daftaupe commented 4 years ago

[...] Also, I consider these config files like code, so having version tracking (better than wiki) for them is a nice plus too.

I just noticed that on the wiki there are revisions for each page that we can compare, so not exactly version tracking, but pretty much sufficient in my opinion :)

lucollab commented 4 years ago

could this help? https://stackoverflow.com/questions/55444108/python-flask-service-file-is-not-starting

when i execute the following comand: uwsgi --emperor --ini /etc/uwsgi/sites/etesync.ini

Than the server runs and is reachable. But there is a problem with the uwsgi service file. With the comand above which works manually, there is still an error when running the service:

● uwsgi.service - uWSGI Emperor service
   Loaded: error (Reason: Invalid argument)
   Active: failed (Result: exit-code) since Fri 2020-05-01 19:17:41 CEST; 2min 3s ago
 Main PID: 4277 (code=exited, status=203/EXEC)

Mai 01 19:17:41 meeting systemd[1]: uwsgi.service: Unit entered failed state.
Mai 01 19:17:41 meeting systemd[1]: uwsgi.service: Failed with result 'exit-code'.
Mai 01 19:19:33 meeting systemd[1]: [/etc/systemd/system/uwsgi.service:8] Executable path is not absolute, ignoring: uwsgi --
Mai 01 19:19:33 meeting systemd[1]: uwsgi.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.

This line in the /etc/systemd/system/uwsgi.service worked for me:

ExecStart=/usr/bin/uwsgi --emperor --ini /etc/uwsgi/sites/etesync.ini

Now the EteSync server is running.

Little notice: I left out the certificate creation section and put the paths of my Lets Encrypt certificates into nginx conf.

tasn commented 4 years ago

[...] Also, I consider these config files like code, so having version tracking (better than wiki) for them is a nice plus too.

I just noticed that on the wiki there are revisions for each page that we can compare, so not exactly version tracking, but pretty much sufficient in my opinion :)

Yeah, I was referring to that in (better than wiki), as in the revision tracking is better than the wiki one, though I agree it wasn't really clear.

tasn commented 4 years ago

This line in the /etc/systemd/system/uwsgi.service worked for me:

ExecStart=/usr/bin/uwsgi --emperor --ini /etc/uwsgi/sites/etesync.ini

Makes sense. To be honest though, I don't know if this service file is even needed at all. I'm on Arch Linux and the uwsgi package ships with a /usr/lib/systemd/system/emperor.uwsgi.service file that just lets me add uwsgi files to a specific directory to have them start automatically. Or alternatively, there's also /usr/lib/systemd/system/uwsgi@.service that lets me start service files standalone by name...

theAkito commented 4 years ago

Nice to see this script. I was thinking of creating a Dockerfile for the server.

tasn commented 4 years ago

@theAkito, there are already a few community Docker images. Take a look at dockerhub. :)

tomkel commented 4 years ago

This was a great script and very helpful, thank you @BeatLink !

BeatLink commented 4 years ago

NP :)

tasn commented 3 years ago

I'm closing this in favour of #40