search

BenioffOceanInitiative / ws-docker

Ships for Whales Docker server software setup
MIT License
0 stars 0 forks source link

server down after json credential hack #1

Closed bbest closed 3 years ago

bbest commented 3 years ago

Documenting process of spinning server back up here.

bbest commented 3 years ago

Although the virtual machine instance-1 was up, I could not access the following:

So I connected to the instance via web browser SSH authenticated as ben@ecoquants.com who has permission in the Benioff Ocean Initiative project:

Then within the terminal window of the instance using bash commands for Ubuntu and some Docker commands:

# inspect all docker processes (ie containers)
docker ps -a
CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS                          PORTS                                                                    NAMES
5c35e6cacdc5        kartoza/pg-backup:11.0                   "/start.sh"              6 months ago        Up 19 hours                     5432/tcp                                                                 ws-postgis-backup
060db126c804        kartoza/geoserver:2.15.2                 "/scripts/entrypoint…"   6 months ago        Up 19 hours (healthy)           0.0.0.0:8080->8080/tcp                                                   ws-geoserver
94290e2e2c18        kartoza/postgis:11.0-2.5                 "/bin/sh -c /docker-…"   6 months ago        Up 19 hours (healthy)           0.0.0.0:5432->5432/tcp                                                   ws-postgis
b8372794d3ba        bdbest/rstudio-shiny:ws                  "/init"                  6 months ago        Up 19 hours                     0.0.0.0:3838->3838/tcp, 0.0.0.0:8787->8787/tcp, 0.0.0.0:8888->8888/tcp   ws-rstudio-shiny
0ef33808a9fc        mysql:8.0.18                             "docker-entrypoint.s…"   6 months ago        Up 19 hours                     0.0.0.0:3306->3306/tcp, 33060/tcp                                        ws-mysql
fdc6a08f5abb        wordpress:php7.4-apache                  "docker-entrypoint.s…"   6 months ago        Up 19 hours                     0.0.0.0:8000->80/tcp                                                     ws-wordpress
9923100ed9c5        jrcs/letsencrypt-nginx-proxy-companion   "/bin/bash /app/entr…"   6 months ago        Restarting (1) 16 seconds ago                                                                            letsencrypt-nginx-proxy-companion
60b3895a9ab3        jwilder/nginx-proxy                      "/app/docker-entrypo…"   6 months ago        Exited (2) 7 days ago                                                                                    ws-proxy
# change directory to where I previously ran git clone https://github.com/BenioffOceanInitiative/ws-docker.git
cd ws-docker/

# try restarting docker containers
docker-compose restart
Restarting ws-postgis-backup                 ... done
Restarting ws-geoserver                      ... done
Restarting ws-postgis                        ... done
Restarting ws-rstudio-shiny                  ... done
Restarting ws-mysql                          ... done
Restarting ws-wordpress                      ... done
Restarting letsencrypt-nginx-proxy-companion ... done
Restarting ws-proxy                          ... error
ERROR: for ws-proxy  Cannot restart container 60b3895a9ab355c0a6177c0dbe094135a3f3419e965d389b271d9dc957b744b7: driver failed programming external connectivity on endpoint ws-proxy (ac99d791a6722fc36b1ccf7f579b21ef0251ff917394354fa3446f05d1d47287
): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use

So port 80 (the default port for non-secure web browser traffic via http://) is in use. So I visited the IP address of instance-1 per the Google Console http://34.71.113.143/:

image

Ok, it looks like the nginx web server is already running on the host machine. Let's turn that off and restart Docker containers:

# turn off nginx
sudo systemctl stop nginx

# disable autostart
sudo update-rc.d -f nginx disable

# try restarting docker containers
docker-compose stop
docker-compose start
Starting nginx-proxy                       ... done
Starting letsencrypt-nginx-proxy-companion ... done
Starting postgis                           ... done
Starting postgis-backup                    ... done
Starting geoserver                         ... done
Starting mysql                             ... done
Starting wordpress                         ... done
Starting rstudio-shiny                     ... done

Yay, no error messages! But these websites are still not accessible:

Let's "dig" into this a bit more and find out to which machine those URLs are pointing:

dig rstudio.whalesafe.com
; <<>> DiG 9.16.1-Ubuntu <<>> rstudio.whalesafe.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9589
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;rstudio.whalesafe.com.         IN      A
;; ANSWER SECTION:
rstudio.whalesafe.com.  1799    IN      A       34.67.27.49
;; Query time: 72 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Dec 23 19:03:59 UTC 2020
;; MSG SIZE  rcvd: 66

So the problem is that rstudio.whalesafe.com points to 34.67.27.49, but needs to point to the latest IP which is 34.71.113.143. The future workaround is to reserve an IP address for the instance. Meanwhile let's get the domain pointing to the correct servers:

  1. Log into buydomainscentral.com, which migrated to https://www.enomcentral.com
    • Login ID: whalesafecom
    • Password: *#**
  2. Navigate to Domains (1) Manage > whalesafe.com > Host Records Edit
  3. Note A the records (api, rstudio, shiny, wp) updated to 34.71.113.143: image

Ok, success! Both of these sites are now responsive (after the DNS servers updated, which could take an hour):

Now to deal with this error message "502 Bad Gateway" from the nginx proxy web server. That just means we need to start the API service as described in ws-api: run_api.R:

Log into https://rstudio.whalesafe.com, and use the Terminal to run:

sudo Rscript /share/github/ws-api/run_api.R &
# loading packages...
Starting server to listen on port 8888
Running the swagger UI at http://127.0.0.1:8888/__swagger__/

Finally, revisiting https://api.whalesafe.com yields the desired website:

image

bbest commented 3 years ago

Getting the JSON credentials updated will be a separate issue. At least the API is back up.