search

MarineSensitivity / server

server setup for R Shiny apps, RStudio IDE, R Plumber API, PostGIS database, pg_tileserv
MIT License
0 stars 0 forks source link

setup server on AWS with Docker config #2

Closed bbest closed 9 months ago

bbest commented 9 months ago

Need to reinstate account under ben@ecoquants.com (Account ID: 814665782451), which is still blocked despite adding billing method and not having any payments due.

bbest commented 9 months ago

Had to setup multi-factor authentication (MFA) using Duo Mobile.

bbest commented 9 months ago

Added A records to DNS SquareSpace for subdomain services:

image

bbest commented 9 months ago

With commit https://github.com/ecoquants/server/commit/b12ee92bddba338b85b238c6491a039d037bb007 added to Caddyfile#L68-L87:

tile.marinesensitivies.org {
  reverse_proxy tilecache:6081
}

shiny.marinesensitivies.org {
  reverse_proxy rstudio:3838
}

rstudio.marinesensitivies.org {
  reverse_proxy rstudio:8787
}

file.marinesensitivies.org {
  root * /share/public
    file_server
}

api.marinesensitivies.org {
  reverse_proxy rstudio:8888
}
bbest commented 9 months ago

On Contabo server (ssh root@154.53.57.44):

# change directory to server Github repo
cd server

# get latest from repo with new Caddyfile subdomain reverse proxies
git pull

# view docker container services
docker ps
CONTAINER ID   IMAGE                    COMMAND                  CREATED         STATUS                  PORTS                                                                    NAMES
69578ae75aff   postgis/postgis:latest   "docker-entrypoint.s…"   13 months ago   Up 9 months (healthy)   0.0.0.0:5432->5432/tcp                                                   postgis
f4afbd86f465   server_rstudio           "/init"                  13 months ago   Up 5 months             0.0.0.0:3838->3838/tcp, 0.0.0.0:8787->8787/tcp, 0.0.0.0:8888->8888/tcp   rstudio
0b923ae62ccb   caddy:latest             "caddy run --config …"   16 months ago   Up 7 months             0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 2019/tcp                       caddy
docker 
# restart caddy to get latest Caddyfile update
docker restart caddy

# check logs to attempts to get SSL certificates
docker logs caddy
{"level":"info","ts":1698151684.110244,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["file.marinesensitivies.org"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"error","ts":1698151684.193386,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"file.marinesensitivies.org","issuer":"acme.zerossl.com-v2-DV90","error":"[file.marinesensitivies.org] creating new order: attempt 1: https://acme.zerossl.com/v2/DV90/newOrder: HTTP 429: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1698151684.193444,"logger":"tls.obtain","msg":"will retry","error":"[file.marinesensitivies.org] Obtain: [file.marinesensitivies.org] creating new order: attempt 1: https://acme.zerossl.com/v2/DV90/newOrder: HTTP 429: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":10.532520385,"max_duration":2592000}
{"level":"info","ts":1698151685.5697374,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"tile.marinesensitivies.org","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"error","ts":1698151692.1000445,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"tile.marinesensitivies.org","challenge_type":"http-01","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]}}
{"level":"error","ts":1698151692.1001267,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"tile.marinesensitivies.org","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]},"order":"https://acme.zerossl.com/v2/DV90/order/BPyjYRmlgRKuh0zeuNqDZw","attempt":1,"max_attempts":3}
{"level":"error","ts":1698151692.100178,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"tile.marinesensitivies.org","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 0  - "}
{"level":"error","ts":1698151692.100197,"logger":"tls.obtain","msg":"will retry","error":"[tile.marinesensitivies.org] Obtain: [tile.marinesensitivies.org] solving challenge: tile.marinesensitivies.org: [tile.marinesensitivies.org] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":19.178928585,"max_duration":2592000}
bbest commented 9 months ago

Compare DNS information with new (rstudio.marinesensitivities.org) not yet up at https://rstudio.marinesensitivies.org and working (rstudio.ecoquants.com) up at https://rstudio.ecoquants.com:

dig rstudio.marinesensitivities.org
dig rstudio.ecoquants.com
root@vmi906982:~/server# dig rstudio.marinesensitivities.org

; <<>> DiG 9.16.1-Ubuntu <<>> rstudio.marinesensitivities.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36225
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;rstudio.marinesensitivities.org. IN    A

;; ANSWER SECTION:
rstudio.marinesensitivities.org. 14400 IN A 154.53.57.44

;; Query time: 27 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Tue Oct 24 05:52:17 PDT 2023
;; MSG SIZE  rcvd: 76

root@vmi906982:~/server# dig rstudio.ecoquants.com

; <<>> DiG 9.16.1-Ubuntu <<>> rstudio.ecoquants.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49896
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;rstudio.ecoquants.com.     IN  A

;; ANSWER SECTION:
rstudio.ecoquants.com.  3600    IN  A   154.53.57.44

;; Query time: 167 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Tue Oct 24 05:52:36 PDT 2023
;; MSG SIZE  rcvd: 66
bbest commented 9 months ago

Confusing options from SquareSpace (taking over Google Domains). Will have to wait 72 hrs to see which works:

  1. A record: shiny|tile.marinesensitivities.org TO 154.53.57.44
  2. Domain forwarding: rstudio.marinesensitivities.org TO 154.53.57.44
  3. Nameserver Registration: file.marinesensitivities.org TO 154.53.57.44

References

bbest commented 9 months ago

Domain Fixing

curl "https://cloudflare-dns.com/dns-query?name=rstudiomarinesensitivities.org&type=A" \
  -H "accept: application/dns-json"| json_pp
{
   "AD" : false,
   "Answer" : [
      {
         "TTL" : 14400,
         "data" : "185.199.109.153",
         "name" : "marinesensitivities.org",
         "type" : 1
      },
      {
         "TTL" : 14400,
         "data" : "185.199.108.153",
         "name" : "marinesensitivities.org",
         "type" : 1
      },
      {
         "TTL" : 14400,
         "data" : "185.199.110.153",
         "name" : "marinesensitivities.org",
         "type" : 1
      },
      {
         "TTL" : 14400,
         "data" : "185.199.111.153",
         "name" : "marinesensitivities.org",
         "type" : 1
      }
   ],
   "CD" : false,
   "Question" : [
      {
         "name" : "marinesensitivities.org",
         "type" : 1
      }
   ],
   "RA" : true,
   "RD" : true,
   "Status" : 0,
   "TC" : false
}
curl "https://cloudflare-dns.com/dns-query?name=rstudio.marinesensitivities.org&type=A" \
  -H "accept: application/dns-json"| json_pp
{
   "AD" : false,
   "Answer" : [
      {
         "TTL" : 14400,
         "data" : "154.53.57.44",
         "name" : "rstudio.marinesensitivities.org",
         "type" : 1
      }
   ],
   "CD" : false,
   "Question" : [
      {
         "name" : "rstudio.marinesensitivities.org",
         "type" : 1
      }
   ],
   "RA" : true,
   "RD" : true,
   "Status" : 0,
   "TC" : false
}
bbest commented 9 months ago

All working on new Amazon Web Service (AWS) EC2 instance with 60 GB volume attached:

bbest commented 9 months ago

server

The server software is for setting up web services outside those of Github (e.g. serving website, docs and R package) using Docker (see the docker-compose.yml; with reverse proxying from subdomains to ports by Caddy):

Connect

# ssh
pem='/Users/bbest/My Drive/private/msens_key_pair.pem'
ssh -i $pem ubuntu@msens1.marinesensitivities.org

# $PASSWORD
cat '/Users/bbest/My Drive/private/msens_server_env-password.txt'

Restart

cd ~/server
git pull

# restart with any new configs
sudo docker restart

# update software
sudo docker compose up -d

Reference

bbest commented 9 months ago

SquareSpace Domain Settings - Custom Records

For subdomain direction to server IP address.

Custom Records

Host Type Priority Data
api A N/A 100.25.173.0
file A N/A 100.25.173.0
@ A N/A 185.199.108.153
@ A N/A 185.199.109.153
@ A N/A 185.199.110.153
@ A N/A 185.199.111.153
msens1 A N/A 100.25.173.0
pgadmin A N/A 100.25.173.0
rest A N/A 100.25.173.0
rstudio A N/A 100.25.173.0
shiny A N/A 100.25.173.0
tile A N/A 100.25.173.0
www CNAME N/A marinesensitivities.org
swagger A N/A 100.25.173.0