Enable Service Monitoring

[fbaig]

Problem There is no mechanism to detect whether any of the I-GUIDE services is down, not accessible or having any issues. All current detection is done manually which is not optimal and needs to be automated to mitigate failures and increase efficiency.

Potential Solution

• Enable OpenStack monitoring on JetStream instances
• Integrate Prometheus for further monitoring

Pull Requests ToDo ...

[rkalyanapurdue]

ToDo:

1. Create a detailed system deployment diagram, identifying services that need monitoring.
2. Deploy Prometheus in the Kubernetes cluster on Jetstream2 and configure it for Jupyter & CVMFS monitoring.
3. Identify a solution for OpenStack monitoring.
4. Demonstrate Grafana views showing services & CI Platform usage metrics.

[yirugi]

MEMO for 2/27 hacking session

Prometheus

JupyterHub metrics

• https://jupyterhub.readthedocs.io/en/stable/reference/monitoring.html

Ansible for Prometheus

• https://github.com/prometheus-community/ansible

Prometheus config for external K8s

Not sure if it works. It seems we need to config account, and may have SSL issue https://stackoverflow.com/questions/50587087/prometheus-outside-kubernetes-cluster

kubernetes_sd_configs:
- api_server: https://<ip>:6443
  role: node
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true
  basic_auth:
      username: kube
      password: Superkube01
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

• https://www.linkedin.com/pulse/monitoring-kubernetes-prometheus-outside-cluster-steven-acreman/
• complicated one:
  • https://gist.github.com/sacreman/b61266d2ec52cf3a1af7c278d9d93450

Grafana

Grafana Installation on Debian/Ubuntu

• https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/

OpenStack monitor

• In Prometheus
  • https://prometheus.io/docs/prometheus/latest/configuration/configuration/#openstack_sd_config
  • https://medium.com/@pasquier.simon/monitoring-your-openstack-instances-with-prometheus-a7ff4324db6c

    
    global:
    scrape_interval:     15s
    evaluation_interval: 15s
    rule_files: []
    scrape_configs:
    # Scrape Prometheus itself

  • job_name: 'prometheus' static_configs:
    • targets: ['localhost:9090']

      Scrape OpenStack instances

  • job_name: 'openstack' openstack_sd_configs:
    • identity_endpoint: https://openstack.example.com:5000/v2.0 username: simon project_name: prometheus-lab password: supersecret role: instance relabel_configs:

      Keep only active instances

    • source_labels: [__meta_openstack_instance_status] action: keep regex: ACTIVE

      Keep only instances which are flagged for scraping

    • source_labels: [__meta_openstack_tag_prometheus_io_scrape] action: keep regex: 'true'

      Update the scraping port if required

    • source_labels:
    • address
    • meta_openstack_tag_prometheus_io_port action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: address__

      Replace the default instance by the OpenStack instance name

    • source_labels: [__meta_openstack_instance_name] target_label: instance

• In Grafana
  • Install Graphite service in OpenStack, add it as a datasource in Grafana
  • https://achchusnulchikam.medium.com/monitoring-instances-on-openstack-using-collectd-graphite-and-grafana-38c757355457

CVMFS

CVMFS monitoring tool

• https://github.com/cvmfs-contrib/cvmfs-servermon

Helm charts

• Not going to use K8s since it is difficult config external IPs
• https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md
• https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/README.md

[yirugi]

2/27 Hacking Report

Requirements:

• Monitor the health and performance of the following services:
  • OpenStack instance in JetStream
  • Kubernetes cluster
  • JupyterHub deployment
  • CVMFS

What we have:

• Generic Website Monitoring System: https://i-guide.github.io/Status-Monitoring-with-Upptime/

Progress:

• Monitoring Stack:

  • Prometheus: A monitoring service to collect metrics from the target services.
  • Supported Services: Kubernetes, OpenStack.
  • Grafana: A visualization tool to display the collected metrics in dashboards.
  • Get metric data from Prometheus
  • JupyterHub metric

• Initial Deployment:

  • We attempted to deploy Prometheus and Grafana within the Kubernetes cluster.
  • However, configuring external IPs for these services proved challenging.

• Alternative Deployment:

  • Deployment:
  • Deploy Prometheus and Grafana as separate virtual machines (VMs) within an OpenStack instance.
  • Challenges:
  • Connecting Kubernetes to the Prometheus service resulted in certificate and authentication issues. Resolving these issues might require further investigation.

• Alternative Approach:

  • Utilize logs from OpenStack instances and integrate them with Grafana using Logstash for visualization.

Next Steps:

• Investigate and address the certificate/authentication issues for connecting Kubernetes to the Prometheus service deployed in OpenStack.
• Evaluate the feasibility and implementation timeline for the log-based monitoring approach using Logstash and Grafana. Prioritize the most critical services for initial monitoring based on business needs and potential impact.
• Investigate CVMFS monitoring tool: https://github.com/cvmfs-contrib/cvmfs-servermon

[yirugi]

Remote k8s connection to Prometheus can be done if we set "Service Account Tokens": https://iximiuz.com/en/posts/kubernetes-api-call-simple-http-client/

[rkalyanapurdue]

Consider using Zabbix and integrate with OpenStack + Kubernetes