search

CharlBest / nean-stack-starter

neo4j, express, angular, node
https://nean.io
30 stars 6 forks source link
angular angular-material2 express internalization jwt neo4j nodejs pwa stripe typescript webpack

readme

NEAN (Neo4j, Express, Angular, Node) Stack Starter

A basic app that has all the features that all platforms have like authentication, validation, error handling and more.

DEMO

CI/CD

Stack

Similar to the MEAN stack just with Neo4j rather than Mongo.

Client pages

Coming Soon

Running locally for development (watch/hot reload)

Make sure you have Node.js and Docker installed.

node -v
docker -v
git clone git@github.com:CharlBest/nean-stack-starter.git
cd nean-stack-starter
npm install
npm start

Your app should have automatically opened the browser and be running on localhost:4200

Only do this once at the start to complete setup

Make it your own (after clone)

  1. Rename all occurances of the word "nean" to your chosen name
  2. Replace logo in client assets folder
  3. Create accounts for services
    1. Firebase account (Images)
    2. Stripe (Payments)
    3. Google Cloud Platform
      1. Create VM instance in Compute Engine (f1-micro)
      2. Neo4j Instance (Database Hosting)
      3. RabbitMQ Instance (Message queue)
      4. Nginx reverse proxy
      5. PM2 Node manager
    4. Cloudflare (DNS)
    5. Namecheap Domain

Environment variables

Change to yours.

Client

In the environments folder there are 2 files. one for dev and one for production. The angular CLI switches them at build time.

Server

In the environments folder there is 1 file. In development default values are used. In production your hosting service should set process.env with the appropriate variables

Note: firebase storage has authentication on their buckets. Whitelist your url or something

3rd Party Cloud Solutions

Server setup

Google Cloud (VM), Ubuntu server, Xfce4, VNC server, Putty SSH, Git, NGINX, Angular CLI, NodeJS, PM2, Docker, Neo4j, RabbitMQ, Cloudflare, Namecheap

File destinations

Ports:

nean.io

dev.nean.io

Setup Ubuntu Server

Source: https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-18-04

  1. Google Cloud Platfrom
  2. Compute Engine
  3. VM Instance - Create instance
    1. Machine type - micro (1 shared vCPU)
    2. Ubuntu 18.04 LTS Minimal - 20GB SSD
    3. Allow HTTPS traffic
    4. Advanced
      1. Management: Enable deletion protection
      2. Networking: Static external IP (NB: can't assign afterwards)

Why Google CLoud VM:

Heroku charges $7 per instance that doesn't go to sleep after 30 minutes. That means if I want 3 Node instances, 2 for load balancing and zero downtime updates and 1 for the background worker process I have to pay ~$21. The VM is ~$25 and +- 7 instances can be run on it.

Why not use the f1-micro or g1-small:

When building the client project with the Angular CLI the server freezes and a possible cause is the shared CPU because it doesn't happen with the standard machine types or it could be an out of memory problem.

Ubuntu Desktop Environment

Source: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-18-04

Update

sudo apt update

Breakdown:

sudo apt upgrade

Breakdown:

Install

Packages repo site: https://packages.ubuntu.com/

sudo apt install xfce4 xfce4-goodies tightvncserver nano firefox

Breakdown:

Optional: autocopysel (clipboard copy paste library but consumes lots of memory and scared of multiple running instances)

VNC server setup

  1. Start
vncserver

  1. Enter password and set view-only password to NO

  2. 1 represents the number of vnc server running

vncserver -kill :1
  1. Copy of your existing/default VNC configuration
cp ~/.vnc/xstartup ~/.vnc/xstartup_backup
  1. Edit default config
nano ~/.vnc/xstartup
  1. Edit file
#!/bin/bash
xsetroot -solid grey
/etc/X11/Xsession
export XKL_XMODMAP_DISABLE=1
startxfce4 &

  1. Then hit Ctrl + X, and then Y to save and exit Nano

  2. Start server again (higher resolution and color depth can degrade performance)

vncserver -geometry 1280x720 -depth 24

SSH Tunnel for VNC on Windows Client

  1. Go to https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

  2. Download and install PuTTY and puttygen.exe

  3. Open puttygen

    • Click "Generate"
    • Key comment = eg. bob
    • Click "Save private key" and use in PuTTY connection utility
    • Copy public key and add to Google Compute allowed SSH keys for your instance via the web console

  4. Open PuTTY

    1. Session
    2. Connection > SSH > Auth
      • Private key file for authentication
      • Click "Browse" and choose previously generated private key
    3. Connection > SSH > Tunnels
      • Source port = 5901
      • Destination = localhost:5901
      • Click "Add"
    4. Session
      • Saved sessions = vnc
      • Click "Save"
    5. Click "Open" at the bottom
    6. Enter password for instance in console

VNC viewer

  1. Download and install RealVNC from https://www.realvnc.com/en/connect/download/vnc/windows/
  2. VNC server address = localhost:5901

Setup domain

Namecheap

Buy a domain name and set the name servers to cloudflare

Cloudflare

Setup Nginx Web Server

Source: https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04

Complete source: https://medium.com/@jgefroh/a-guide-to-using-nginx-for-static-websites-d96a9d034940

Example: https://github.com/FranciscoKnebel/nginx-reverseproxy

Proxy summary: https://www.digitalocean.com/community/tutorials/understanding-nginx-http-proxying-load-balancing-buffering-and-caching

  1. Install
sudo apt install nginx
  1. Test
sudo nginx -v
  1. Create /etc/nginx/sites-available/nean.io
server {
    listen 443 http2;
    # IPv6 addresses
    listen [::]:443 http2;

    ssl on;
    ssl_certificate /etc/letsencrypt/live/nean.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/nean.io/privkey.pem;

    server_name nean.io www.nean.io;

    root /var/www/nean.io/dist/client;

    access_log /var/www/nean.io/nginx/nginx.access.log;
    error_log  /var/www/nean.io/nginx/nginx.access.log;

    include snippets/static-files.conf;

    location /api/ {
        proxy_pass http://localhost:3010;
        include snippets/api-params.conf;
    }

    location /analytics/ {
        proxy_pass http://localhost:32768/;
    }
}
  1. Create /etc/nginx/sites-available/dev.nean.io
server {
    listen 443 http2;
    # IPv6 addresses
    listen [::]:443 http2;

    ssl on;
    ssl_certificate /etc/letsencrypt/live/dev.nean.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/dev.nean.io/privkey.pem;

    server_name dev.nean.io;

    root /var/www/nean.io/dev/dist/client;

    access_log /var/log/dev.nean.io/nginx.access.log;
    error_log  /var/log/dev.nean.io/nginx.error.log;

    include snippets/static-files.conf;

    location /api/ {
        proxy_pass http://localhost:3020;
        include snippets/api-params.conf;
    }
}
  1. Create snippets/static-files.conf
index index.html;

location / {
    # First attempt to serve request as file, then as directory, then fall back to displaying the index.html
    try_files $uri $uri/ /index.html;

    # Cacheing
    # expires 1d;
    # add_header Cache-Control "public, no-cache";
}
  1. Create snippets/api-params.conf
proxy_http_version 1.1;

# Set host header
proxy_set_header Host $host;

# List of IP addresses
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

# Log IP on nginx proxy server
proxy_set_header X-Real-IP $remote_addr;

# HTTP or HTTPS?
proxy_set_header X-Forwarded-Proto $scheme;

# CORS https://www.digitalocean.com/community/questions/allow-cors-origin-for-node-angular-api-on-nginx

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Authorization $http_authorization;
# proxy_set_header X-NginX-Proxy true;
# proxy_max_temp_file_size 0;
# proxy_redirect off;
# proxy_read_timeout 240s;
# TODO: proxy_cache_bypass $http_upgrade;
  1. Create symbolic links in /etc/nginx/sites-enabled
ln -s /etc/nginx/sites-available/nean.io /etc/nginx/sites-enabled/nean.io

ln -s /etc/nginx/sites-available/dev.nean.io /etc/nginx/sites-enabled/dev.nean.io

Breakdown:

ln: make link command -s: symbolic link flag

  1. Test config
sudo nginx -t
  1. Restart
sudo service nginx restart
  1. Set permissions
sudo chown -R $USER:$USER /var/www/nean.io

sudo chmod -R 755 /var/www/nean.io

# If the above doesn't work use this
# sudo chown -R www-data:www-data /var/www/nean.io

Note:

4 different cache location exist:

  1. Browser (Cache-Control header)
  2. Service Worker (ng-config.json)
  3. CloudFlare (browser UI)
  4. Nginx (.conf)

Setup Node.js

Source: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for-production-on-ubuntu-18-04

Install & Update NVM (Node version manager: https://github.com/creationix/nvm)

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash

Test NVM

nvm --version

List Node versions

nvm ls-remote

Install Node

nvm install v10.13.0

Important: when installing a new Node version you need to reinstall the global packages like pm2

Test Node

node --version

NPM packages to work (those that require compiling code from source, for example)

sudo apt install build-essential

Notes:

Setup PM2

Install PM2

npm install pm2 -g

Test PM2

pm2 -v

Ecosystem File (configuration)

  1. Create ecosystem.config.js in /var/www/nean.io/

  2. Replace ecosystem.config.js text

// Options reference: https://pm2.io/doc/en/runtime/reference/ecosystem-file/

const environmentVariables = {
  APP_HOST: "",
  AUTHENTICATION_KEY: "37LvDSm4XvjYOh9Y",
  NODE_ENV: "production",
  SENDGRID_API_KEY: "***",
  SENDGRID_TEMPLATE_FEEDBACK: "***",
  SENDGRID_TEMPLATE_FORGOT_PASSWORD: "***",
  SENDGRID_TEMPLATE_INVITE: "***",
  SENDGRID_TEMPLATE_NOTIFICATION: "***",
  SENDGRID_TEMPLATE_PASSWORD_UPDATED: "***",
  SENDGRID_TEMPLATE_PAYMENT_SUCCESSFUL: "***",
  SENDGRID_TEMPLATE_RESEND_EMAIL_VERIFICATION_LINK: "***",
  SENDGRID_TEMPLATE_SYSTEM: "***",
  SENDGRID_TEMPLATE_WELCOME: "***",
  STRIPE_KEY: "***",
  VAPID_PRIVATE_KEY: "***",
  VAPID_PUBLIC_KEY: "***",
};

function generate(environment, server, env, port = 0, instances = 1) {
  return {
    name: `${environment}_${server}`,
    script: `${server}.bundle.js`,
    cwd: `${environment}/dist/server/${server}`,
    instances: instances,
    env: { ...environmentVariables, PORT: port, ...env },
    // Time in ms to wait before restarting a crashing app
    restart_delay: 10000,
    // Number of times a script is restarted when it exits in less than min_uptime
    max_restarts: 10,
    // Minimum uptime of the app to be considered started
    min_uptime: 1000,
  };
}

module.exports = {
  apps: [
    generate(
      "prod",
      "web",
      {
        AMQP_URL: "amqp://server_api:<PASSWORD>@localhost:5672",
        DATABASE_PASSWORD: "<PASSWORD>",
        DATABASE_URI: "bolt://localhost:7687",
        DATABASE_USERNAME: "server_api",
      },
      3010,
      2
    ),
    generate("prod", "worker", {
      AMQP_URL: "amqp://server_worker:<PASSWORD>@localhost:5672",
      DATABASE_PASSWORD: "<PASSWORD>",
      DATABASE_URI: "bolt://localhost:7687",
      DATABASE_USERNAME: "server_worker",
    }),
    generate(
      "dev",
      "web",
      {
        AMQP_URL: "amqp://server_api:<PASSWORD>@localhost:5673",
        DATABASE_PASSWORD: "<PASSWORD>",
        DATABASE_URI: "bolt://localhost:7688",
        DATABASE_USERNAME: "server_api",
      },
      3020,
      2
    ),
    generate("dev", "worker", {
      AMQP_URL: "amqp://server_worker:<PASSWORD>@localhost:5673",
      DATABASE_PASSWORD: "<PASSWORD>",
      DATABASE_URI: "bolt://localhost:7688",
      DATABASE_USERNAME: "server_worker",
    }),
  ],
  deploy: {
    dev: {
      host: "localhost",
      ref: "origin/dev",
      repo: "https://github.com/heroku/node-js-getting-started.git",
      path: "var/www/nean.io/deploy/dev",
      "post-deploy":
        "npm install && npm run build && pm2 reload ecosystem.config.js",
    },
    staging: {
      host: "localhost",
      ref: "origin/staging",
      repo: "https://github.com/heroku/node-js-getting-started.git",
      path: "var/www/nean.io/deploy/staging",
      "post-deploy":
        "npm install && npm run build && pm2 reload ecosystem.config.js",
    },
  },
};

Auto start at boot

  1. Generate script to run
pm2 startup

  1. Copy paste command to setup

  2. Take snapshot of what processes should start on startup

pm2 save
  1. Test
sudo systemctl status pm2-<USERNAME>

Link/connect PM2 to web interface

pm2 link <key> <key> MACHINE_NAME

Detect github commits

https://github.com/adnanh/webhook

Automatic build and deploy if pushed to DEV

Source: https://medium.com/@riyadhalnur/managing-and-deploying-nodejs-apps-with-pm2-173fbc7d3f95

Install Git

Source: https://www.digitalocean.com/community/tutorials/how-to-install-git-on-ubuntu-18-04

  1. Install
sudo apt install git
  1. Set global git configuration
git config --global user.name "Your Name"

git config --global user.email "youremail@domain.com"

Setup Nginx with Let's Encrypt SSL (HTTPS certificate)

Source: https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04

  1. Acquire an SSL cert
sudo apt-get install software-properties-common

sudo add-apt-repository ppa:certbot/certbot

sudo apt-get update

sudo apt-get install python-certbot-nginx

sudo certbot --nginx certonly

  1. Automatic cron job will be created to renew certificates

  2. Remember to set disable HTTP access to server in Google Cloud firewall settings

  3. Remember to set Cloudflare > Crypto > SSL to "Full (strict)"

Setup Nginx with HTTP/2

Source: https://www.digitalocean.com/community/tutorials/how-to-set-up-nginx-with-http-2-support-on-ubuntu-18-04

Add http2 in NGINX conf after port numbers

listen 443 http2;

Setup Neo4j & RabbitMQ with Docker containers (multiple instances)

Install docker

Source: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04

  1. Get package
sudo apt update

sudo apt install apt-transport-https ca-certificates curl software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"

sudo apt update
  1. Install
sudo apt install docker-ce
  1. Test
sudo systemctl status docker
  1. Executing the Docker Command Without Sudo by adding the current user to the docker group
sudo usermod -aG docker ${USER}
  1. Log out and back in for changes to take effect

Docker compose

  1. Create docker-compose.yml
version: "3"

services:
  countly:
    container_name: countly
    image: countly/countly-server
    ports:
      - "32768:80"
    volumes:
      - ./countly:/data/db
    restart: unless-stopped

  neo4j.nean.io:
    container_name: neo4j.nean.io
    image: neo4j/3.5-enterprise
    ports:
      - "7474:7474" ## Browser
      - "7687:7687" ## Bolt connection
    volumes:
      - $HOME/neo4j/nean.io/data:/data
      - $HOME/neo4j/nean.io/backup:/backup
      - $HOME/neo4j/nean.io/logs:/logs
      - $HOME/neo4j/nean.io/plugins:/plugins
    environment:
      NEO4J_dbms_memory_heap_max__size: "512M"
      NEO4J_dbms_memory_pagecache_size: "512M"
      NEO4J_AUTH: "neo4j/password"
      NEO4J_ACCEPT_LICENSE_AGREEMENT: "yes"
    restart: unless-stopped

  rabbitmq.nean.io:
    container_name: rabbitmq.nean.io
    image: rabbitmq:3.7
    hostname: rabbitmq.nean.io
    ports:
      - "5672:5672"
    restart: unless-stopped

  neo4j.dev.nean.io:
    container_name: neo4j.dev.nean.io
    image: neo4j/3.5-enterprise
    ports:
      - "7475:7474" ## Browser
      - "7688:7687" ## Bolt connection
    volumes:
      - $HOME/neo4j/dev.nean.io/data:/data
      - $HOME/neo4j/dev.nean.io/backup:/backup
      - $HOME/neo4j/dev.nean.io/logs:/logs
      - $HOME/neo4j/dev.nean.io/plugins:/plugins
    environment:
      NEO4J_dbms_memory_heap_max__size: "512M"
      NEO4J_dbms_memory_pagecache_size: "512M"
      NEO4J_AUTH: "neo4j/password"
      NEO4J_ACCEPT_LICENSE_AGREEMENT: "yes"
    restart: unless-stopped

  rabbitmq.dev.nean.io:
    container_name: rabbitmq.dev.nean.io
    image: rabbitmq:3.7
    hostname: rabbitmq.dev.nean.io
    ports:
      - "5673:5672"
    restart: unless-stopped

Breakdown:

  1. Run docker compose
docker-compose up -d

Breakdown:

Optional

# terminal within container
docker exec -it <container_name> bash
# Execute cyper in container
bin/cypher-shell -u server_api -p password
# process monitoring for containers
docker stats
# process monitoring for specific container
docker top <container_name>
  1. Create new users 3.2 RabbitMQ 
    # RabbitMQ
## Create virtual host
docker exec -it rabbitmq.nean.io rabbitmqctl add_vhost nean
## Create user
docker exec -it rabbitmq.nean.io rabbitmqctl add_user server_api password
docker exec -it rabbitmq.nean.io rabbitmqctl add_user server_worker password
## Set permissions
docker exec -it rabbitmq.nean.io rabbitmqctl set_permissions -p nean server_api ".*" ".*" ".*"
docker exec -it rabbitmq.nean.io rabbitmqctl set_permissions -p nean server_worker ".*" ".*" ".*"
    • Do the same for other instances

Setup periodic backups via cron

  1. Create crontab for user
export VISUAL=nano; crontab -e
  1. Set backup commands every 1 hour

Short codes

@hourly docker exec neo4j.nean.io bin/neo4j-admin backup --from=localhost:6362 --backup-dir=/backup --name=graph.db-backup --fallback-to-full=true --check-consistency=true --pagecache=2G

  1. Save file

  2. Repeat for neo4j.dev.nean.io instance

  3. Optionally send emails after cron job. https://cloud.google.com/compute/docs/tutorials/sending-mail/using-sendgrid

Setup Neo4j without Docker (single instance)

  1. Install Java
sudo apt install openjdk-8-jre
  1. Signing Key
wget -O - https://debian.neo4j.org/neotechnology.gpg.key | sudo apt-key add -
  1. Source List File
echo 'deb https://debian.neo4j.org/repo stable/' | sudo tee -a /etc/apt/sources.list.d/neo4j.list
  1. Install Neo4j
sudo apt update

sudo apt-get install neo4j=1:3.4.9
  1. Test
sudo service neo4j status
  1. Start automatically on boot
sudo systemctl enable neo4j

Navigate to localhost:7474 in your browser

Set different user

  1. Enter cypher shell
/usr/bin/cypher-shell -u neo4j -p neo4j
  1. Create user and disable password set on first login
CALL dbms.security.createUser('nean_dev', 'nean_dev', false)

Setup RabbitMQ without Docker (single instance)

  1. Signing Key
wget -O - "https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc" | sudo apt-key add -
  1. Source List File (debian bionic main + erlang)
echo "deb https://dl.bintray.com/rabbitmq/debian bionic main erlang" | sudo tee /etc/apt/sources.list.d/bintray.rabbitmq.list
  1. Install packages
sudo apt update

sudo apt install rabbitmq-server
  1. Test
sudo service rabbitmq-server status
  1. Install management plugin
sudo rabbitmq-plugins enable rabbitmq_management
  1. Test

Navigate to localhost:15672 in your browser

Setup Reddis

Logs retention

Copy file to VM

WinSCP

https://winscp.net/eng/download.php

OR

GCloud CLI

gcloud compute scp --recurse <FROM_DIR> <USERNAME>@<VM_INSTANCE_NAME>:/var/www/nean.io/

Breakdown:

Top things to check

Source: https://hashnode.com/post/10-things-you-shouldnt-do-while-running-nodejs-in-production-cisab2fyu0s9oth5341faywcw

Steps to creating a new project/instance

Create new project

Update project

Copy to server

cd /var/www/nean.io/
docker-compose up --detach

Cloudflare

NGINX

PM2

Edit ecosystem.config.js

cd /var/www/nean.io/
pm2 reload ecosystem.config.js --update-env
# save the process list
pm2 save

Steps to update code on server

Local

# Get latest code from base
git fetch upstream
git merge upstream/master
npm run build

Server