search

marineenergy / server

server software install using Docker
0 stars 0 forks source link

auto-restart API service: + monitoring service, pm2 not restarting #5

Closed bbest closed 1 year ago

bbest commented 2 years ago

if get 502 Bad Gateway at https://api.marineenergy.com, docker restart rstudio. Check out monit. This probably has to do with a memory leak with the server software.

Also needed to run:

# recreate API service
sudo -u shiny pm2 start --interpreter="Rscript" /share/github/api/run-api.R
# save API service
sudo -u shiny pm2 save

Upgrade to latest software, including Caddy and PostGIS.

bbest commented 1 year ago

Upgraded npm from 6.14.4 to 9.6.7:

npm install -g npm

Upgraded pm2 from 5.1.2 to 5.3.0:

npm install -g pm2
pm2 start run-api

[PM2] Applying action restartProcessId on app [run-api](ids: 0) [PM2] [run-api](0) ✓ [PM2] Process successfully started

id name namespace version mode pid uptime ↺ status cpu mem user watching
0 run-api default N/A fork 16533 0s 1 online 0% 25.5mb root disabled
bbest commented 1 year ago 
pm2 startup
[PM2] Init System found: systemd
Platform systemd
Template
[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=root
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/opt/pyenv/plugins/pyenv-virtualenv/shims:/root/.pyenv/shims:/opt/pyenv/bin:/root/.local/bin:/usr/lib/rstudio-server/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/texlive/bin/x86_64-linux:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/root/.pm2
PIDFile=/root/.pm2/pm2.pid
Restart=on-failure

ExecStart=/usr/local/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/local/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/local/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

Target path
/etc/systemd/system/pm2-root.service
Command list
[ 'systemctl enable pm2-root' ]
[PM2] Writing init configuration in /etc/systemd/system/pm2-root.service
[PM2] Making script booting at startup...
[PM2] [-] Executing: systemctl enable pm2-root...
Synchronizing state of pm2-root.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable pm2-root
[PM2] [v] Command successfully executed.
+---------------------------------------+
[PM2] Freeze a process list on reboot via:
$ pm2 save

[PM2] Remove init script via:
$ pm2 unstartup systemd
pm2 save
[PM2] Saving current process list...
[PM2] Successfully saved in /root/.pm2/dump.pm2
bbest commented 1 year ago 
uptime
 19:38:31 up 4 days,  2:26,  0 users,  load average: 0.20, 0.15, 0.17
bbest commented 1 year ago

Added the following free monitor via Better Uptime:

bbest commented 1 year ago

CC: Grace, Zach, Will

Hi Caleb,

It looks like the ME server is set up to restart every week on Sunday. Do you know why this might be?

I continue to have trouble getting the api.marineenergy.app service to automatically restart using pm2 with R plumber. Normally pm2 is used to setup nodeJS backends, which dominate the web, so not sure why this isn't working. I upgraded the npm and pm2 software on the server, ran the setup and saved again as root. Fingers crossed that does the trick.

The /report endpoint of api.marineenergy.app is used to generate reports (via the Reports tab of the app shiny.marineenergy.app/report-v2). The /ferc_docs endpoint will serve FERC document excerpts to PRIMRE via its Guidelines, and a similar /ba_docs endpoint will be set up with the latest BioAssessments. Here's an example of the recently reformatted landing page for an individual FERC document excerpt:

For the BA, checkboxes will get dropped. These pages are generated from this repo:

More details and attempts can be found here:

Thanks, Ben

geocoug commented 1 year ago

Hey @bbest,

Thanks for the details. Sorry this has been such a headache. Our IT group runs updates and snapshots of the server every Sunday, which is why it keeps restarting. It sounds like we may need to have them add additional startup commands to their procedures to make sure the Plumber API gets restarted.

In going through your comments above, does this cover the restart process? 

# Enter the shiny container
sudo docker exec -it rstudio bash

# Recreate API service
sudo -u shiny pm2 start --interpreter="Rscript" /share/github/api/run-api.R

# Save API service
sudo -u shiny pm2 save

# Start the API
sudo -u shiny pm2 start run-api

We can also have them ping the API to make sure it is available once everything is restarted. Currently, they are just checking to make sure that all of the correct docker containers are running.

bbest commented 1 year ago

Hi Caleb,

Ooh, yes please! That should do the trick (although the last command is redundant with initial pm2 start), or as a one liner from the host machine:

sudo docker exec rstudio sudo -u shiny pm2 start --interpreter="Rscript" /share/github/api/run-api.R

Thanks a bunch, Ben

On Fri, May 26, 2023 at 12:11 PM @geocoug wrote: Hey Ben,

Thanks for the details. Sorry this has been such a headache. Our IT group runs updates and snapshots of the server every Sunday, which is why it keeps restarting. It sounds like we may need to have them add additional startup commands to their procedures to make sure the Plumber API gets restarted.

In going through your comments on GitHub, does this cover the restart process?

# Enter the shiny container
sudo docker exec -it rstudio bash

# Recreate API service
sudo -u shiny pm2 start --interpreter="Rscript" /share/github/api/run-api.R

# Save API service
sudo -u shiny pm2 save

# Start the API
sudo -u shiny pm2 start run-api

We can also have them ping the API to make sure it is available once everything is restarted. Currently, they are just checking to make sure that all the correct docker containers are running.

geocoug commented 1 year ago

Should be set now. Let's see how things go on Sunday.

We set a cron job to trigger at boot as root, wait two minutes then run the command.

@reboot /home/cforrest/docker_api_start.sh

Once started, the endpoint api.marineenergy.app/ferc_docs is pinged and expects a 200 response.

bbest commented 1 year ago

Seems to be working finally, but with a 2 hour lag. Closing for now