Use something (recommended by Ops) for monitoring

[kelynch]

Acceptance Criteria

• [ ] Install Ops recommended Monitoring for PDC Describe (in princeton_ansible - https://github.com/pulibrary/princeton_ansible/issues/3021)
• [ ] Install Ops recommended Monitoring for Globus boxes
• [ ] Install Ops recommended Monitoring for PDC Discovery
• [ ] We monitor these things as a starting point:
  • [ ] ping check for system being alive
  • [ ] application responds as expected on port 443
  • [ ] disk usage
  • [ ] database connectivity
  • [ ] sshd
  • [ ] S3 connectivity
  • [ ] ssl cert expiration date
  • [ ] CPU usage
  • [ ] memory usage
  • [ ] sidekiq
• [ ] There is a link to the dashboard in either the RDSS-handbook (applications page?) or the pdc-describe readme https://lib-monitor.princeton.edu/c/~/n/default/events?q=event.check.state%20!=%20passing Asks for two factor without telling you! We need to scope the above url to our project and passing or not.

[carolyncole]

Hey team! Please add your planning poker estimate with Zenhub @bess @hectorcorrea @jrgriffiniii @kelynch

[kelynch]

Parts of this completed, documented in https://github.com/pulibrary/rdss-handbook/blob/main/monitoring.md.

Closing this and will regroup with Operations on remaining tasks, to be separately ticketed.

[bess]

Reopening because we don't have another ticket that captures our need for sensu monitoring. I added sidekiq to the list here.

[bess]

We need monitoring for sidekiq because it can crash silently:

[jrgriffiniii]

The following agents have been created for pdc-describe-prod using princeton_ansible:

[jrgriffiniii]

There exists no accompanying proxy for these, I shall evaluate and look to create this now.

[jrgriffiniii]

From what I could best determine, proxy entities are created automatically from status checks invoked from the backend:

[jrgriffiniii]

There seem to be authorization issues when attempting to edit ServiceComponents which I have already created (for a Service which I have also created):

[jrgriffiniii]

I am now currently able to successfully create ServiceComponents which subscribe to alerts from Agents: