Prometheus - Log Strategy

[raftmsohani]

Description: We need to specify how we would like to logs backend logs. We also need to add frontend, server level health monitoring, and Postgres logs to Prometheus.

All modifications in this issue are limited solely to the LOCAL ENVIRONMENT but should be environment agnostic, ensuring that we can fulfill all requirements and acceptance criteria in this issue on cloud.gov in next ticket.

Acceptance Criteria:

As an engineer I want to create and improve log strategy for sending logs to Prometheus

• [ ] Database (Postgres) logs are available in Grafana
• [ ] Logs from backend are available in Grafana
• [ ] Logs from frontend (Nginx) are available in Grafan
• [ ] Backend and Frontend servers are monitored by Grafana (availability, cpu usage, HDD usage)
• [ ] Log retention is defined
• [ ] README is updated, if necessary

Tasks:

• [ ] Define backend logs to be added to Prometheus similar to what is defined here
• [ ] Create an API endpoint that can be securely called by Prometheus
• [ ] Add Nginx logs to Prometheus and verify they are available in Grafana
• [ ] Add server monitoring for both backend and frontend with cpu usage, HDD usage and availability information
• [ ] Run Testing Checklist and confirm all tests pass

Supporting Documentation: Please include any relevant log snippets/files/screen shots

• An example for logging Django logs using Prometheus
• Monitoring Django app with Prometheus

Notes:

• Sentry logging can be added using similar logic addition as the following to settings file:

  import sentry_sdk
  from sentry_sdk.integrations.django import DjangoIntegration
  from sentry_sdk.integrations.logging import LoggingIntegration
  import django
  import logging
  sentry_sdk.init(
      dsn="http://84db0b2b74fea94d8d127ccbcadb94c8@10.0.0.235:9000/2",
      # Set traces_sample_rate to 1.0 to capture 100%
      # of transactions for performance monitoring.
      traces_sample_rate=1.0,
      integrations=[DjangoIntegration(), LoggingIntegration(level=logging.DEBUG, event_level=logging.DEBUG)],
  )

Open Questions:

• What other metrics do we want to add to Grafana?