Closed bbest closed 1 year ago
Upgraded npm
from 6.14.4 to 9.6.7:
npm install -g npm
Upgraded pm2
from 5.1.2 to 5.3.0:
npm install -g pm2
pm2 start run-api
[PM2] Applying action restartProcessId on app [run-api](ids: 0) [PM2] [run-api](0) ✓ [PM2] Process successfully started
id | name | namespace | version | mode | pid | uptime | ↺ | status | cpu | mem | user | watching |
---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | run-api | default | N/A | fork | 16533 | 0s | 1 | online | 0% | 25.5mb | root | disabled |
pm2 startup
[PM2] Init System found: systemd
Platform systemd
Template
[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target
[Service]
Type=forking
User=root
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/opt/pyenv/plugins/pyenv-virtualenv/shims:/root/.pyenv/shims:/opt/pyenv/bin:/root/.local/bin:/usr/lib/rstudio-server/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/texlive/bin/x86_64-linux:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/root/.pm2
PIDFile=/root/.pm2/pm2.pid
Restart=on-failure
ExecStart=/usr/local/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/local/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/local/lib/node_modules/pm2/bin/pm2 kill
[Install]
WantedBy=multi-user.target
Target path
/etc/systemd/system/pm2-root.service
Command list
[ 'systemctl enable pm2-root' ]
[PM2] Writing init configuration in /etc/systemd/system/pm2-root.service
[PM2] Making script booting at startup...
[PM2] [-] Executing: systemctl enable pm2-root...
Synchronizing state of pm2-root.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable pm2-root
[PM2] [v] Command successfully executed.
+---------------------------------------+
[PM2] Freeze a process list on reboot via:
$ pm2 save
[PM2] Remove init script via:
$ pm2 unstartup systemd
pm2 save
[PM2] Saving current process list...
[PM2] Successfully saved in /root/.pm2/dump.pm2
uptime
19:38:31 up 4 days, 2:26, 0 users, load average: 0.20, 0.15, 0.17
Added the following free monitor via Better Uptime:
CC: Grace, Zach, Will
Hi Caleb,
It looks like the ME server is set up to restart every week on Sunday. Do you know why this might be?
I continue to have trouble getting the api.marineenergy.app service to automatically restart using pm2 with R plumber. Normally pm2 is used to setup nodeJS backends, which dominate the web, so not sure why this isn't working. I upgraded the npm and pm2 software on the server, ran the setup and saved again as root. Fingers crossed that does the trick.
The /report
endpoint of api.marineenergy.app is used to generate reports (via the Reports tab of the app shiny.marineenergy.app/report-v2). The /ferc_docs
endpoint will serve FERC document excerpts to PRIMRE via its Guidelines, and a similar /ba_docs
endpoint will be set up with the latest BioAssessments. Here's an example of the recently reformatted landing page for an individual FERC document excerpt:
For the BA, checkboxes will get dropped. These pages are generated from this repo:
More details and attempts can be found here:
Thanks, Ben
Hey @bbest,
Thanks for the details. Sorry this has been such a headache. Our IT group runs updates and snapshots of the server every Sunday, which is why it keeps restarting. It sounds like we may need to have them add additional startup commands to their procedures to make sure the Plumber API gets restarted.
In going through your comments above, does this cover the restart process?
# Enter the shiny container
sudo docker exec -it rstudio bash
# Recreate API service
sudo -u shiny pm2 start --interpreter="Rscript" /share/github/api/run-api.R
# Save API service
sudo -u shiny pm2 save
# Start the API
sudo -u shiny pm2 start run-api
We can also have them ping the API to make sure it is available once everything is restarted. Currently, they are just checking to make sure that all of the correct docker containers are running.
Hi Caleb,
Ooh, yes please! That should do the trick (although the last command is redundant with initial pm2 start), or as a one liner from the host machine:
sudo docker exec rstudio sudo -u shiny pm2 start --interpreter="Rscript" /share/github/api/run-api.R
Thanks a bunch, Ben
On Fri, May 26, 2023 at 12:11 PM @geocoug wrote: Hey Ben,
Thanks for the details. Sorry this has been such a headache. Our IT group runs updates and snapshots of the server every Sunday, which is why it keeps restarting. It sounds like we may need to have them add additional startup commands to their procedures to make sure the Plumber API gets restarted.
In going through your comments on GitHub, does this cover the restart process?
# Enter the shiny container
sudo docker exec -it rstudio bash
# Recreate API service
sudo -u shiny pm2 start --interpreter="Rscript" /share/github/api/run-api.R
# Save API service
sudo -u shiny pm2 save
# Start the API
sudo -u shiny pm2 start run-api
We can also have them ping the API to make sure it is available once everything is restarted. Currently, they are just checking to make sure that all the correct docker containers are running.
Should be set now. Let's see how things go on Sunday.
We set a cron job to trigger at boot as root, wait two minutes then run the command.
@reboot /home/cforrest/docker_api_start.sh
Once started, the endpoint api.marineenergy.app/ferc_docs is pinged and expects a 200 response.
Seems to be working finally, but with a 2 hour lag. Closing for now
if get 502 Bad Gateway at https://api.marineenergy.com,
docker restart rstudio
. Check out monit. This probably has to do with a memory leak with the server software.Also needed to run:
Upgrade to latest software, including Caddy and PostGIS.