Integration of prometheus alarms into SIP demo.

[david-terrett]

Description:

Integration of Prometheus alarms demo into SIP.

To achieve this a set of additional services providing a Prometheus alarms backend are started and the EC Master Controller has been updated to set the value of an SDP state alarm (Prometheus) Gauge whenever the SDP state is in the 'alarm' state.

For more details see sip/platform/alarms/prometheus/README.md.

• A new docker-compose.yml demo file as been added to the deploy/ folder. This starts:
  • A Prometheus container
  • Globally replicated push gateway containers
  • An alarm receiver container
  • An alert manager instance container
  • zookeeper and kafka containers
• The following platform modules have been added:
  • sip/platform/alarms/prometheus/alarm_receiver: Application which listens to HTML PUT requests from Prometheus and inserts the JSON describing the alert into a Kafka queue with topic 'SIP-alarms'.
  • sip/platform/alarms/prometheus/alert_manager: Prometheus alert manager configure to forward alarms to the SIP alarm receiver (see above)
  • sip/platform/alarms/prometheus: Prometheus configuration files for receiving alerts from all push gateways. Currently configured with a single alert rule which raises an alert called SipStateAlarm if the metric sip_state has value 1 .

This completes [TSK-2707]

Testing instructions:

See sip/platform/alarms/prometheus/README.md

Types of changes

• [ ] Bug fix (non-breaking change that fixes an issue)
• [x] New feature (non-breaking change that adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

[codecov[bot]]

Codecov Report

Merging #76 into master will not change coverage. The diff coverage is 100%.

@@           Coverage Diff           @@
##           master      #76   +/-   ##
=======================================
  Coverage   86.07%   86.07%           
=======================================
  Files          59       59           
  Lines        2671     2671           
  Branches      221      221           
=======================================
  Hits         2299     2299           
  Misses        321      321           
  Partials       51       51

Impacted Files	Coverage Δ
...control/processing_controller/scheduler/release.py	100% <100%> (ø)	:arrow_up:
...rol/processing_block_controller/sip_pbc/release.py	100% <100%> (ø)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update b29b2db...3987c4b. Read the comment docs.

[david-terrett]

Ben,

You need to give me (davidterrett) permission to write to the skasip docker repository (or something – I get “access to the resource denied” when I try to push an image).

David

From: Benjamin Mort notifications@github.com Sent: 01 February 2019 01:01 To: SKA-ScienceDataProcessor/integration-prototype integration-prototype@noreply.github.com Cc: Terrett, David (STFC,RAL,RALSP) david.terrett@stfc.ac.uk; Author author@noreply.github.com Subject: Re: [SKA-ScienceDataProcessor/integration-prototype] Integration of prometheus alarms into SIP demo. (#76)

@bmort commented on this pull request.

Overall this looks excellent. Before approving however a couple of minor things:

1. I've taken the liberty of updating the PR description a bit to improve traceability in the git history. Could you check this and update accordingly if you think i've misunderstood or missed anything important.
2. Could the new skasip/<***> images be pushed to Docker Hubhttps://cloud.docker.com/u/skasip - I'm happy to help with this if you like or need to get a Docker Hub account sorted out.
3. Add some testing instructions to the PR description or a README.md file somewhere in the code with a pointer to that in the PR testing section.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/SKA-ScienceDataProcessor/integration-prototype/pull/76#pullrequestreview-198888974, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGuBMDh_q8vCxMdAB6p8WRABq3Ts9nBxks5vI5HTgaJpZM4aVvgK.

[david-terrett]

Ben,

What should I do with the version number of the master controller? It is currently 1.2.1 in master. Should I bump it to 1.2.2 or 1.3.0 (or even 2.0.0)?

David From: Benjamin Mort notifications@github.com Sent: 01 February 2019 01:01 To: SKA-ScienceDataProcessor/integration-prototype integration-prototype@noreply.github.com Cc: Terrett, David (STFC,RAL,RALSP) david.terrett@stfc.ac.uk; Author author@noreply.github.com Subject: Re: [SKA-ScienceDataProcessor/integration-prototype] Integration of prometheus alarms into SIP demo. (#76)

@bmort commented on this pull request.

Overall this looks excellent. Before approving however a couple of minor things:

1. I've taken the liberty of updating the PR description a bit to improve traceability in the git history. Could you check this and update accordingly if you think i've misunderstood or missed anything important.
2. Could the new skasip/<***> images be pushed to Docker Hubhttps://cloud.docker.com/u/skasip - I'm happy to help with this if you like or need to get a Docker Hub account sorted out.
3. Add some testing instructions to the PR description or a README.md file somewhere in the code with a pointer to that in the PR testing section.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/SKA-ScienceDataProcessor/integration-prototype/pull/76#pullrequestreview-198888974, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGuBMDh_q8vCxMdAB6p8WRABq3Ts9nBxks5vI5HTgaJpZM4aVvgK.

[david-terrett]

All done…

David

From: Benjamin Mort notifications@github.com Sent: 01 February 2019 01:01 To: SKA-ScienceDataProcessor/integration-prototype integration-prototype@noreply.github.com Cc: Terrett, David (STFC,RAL,RALSP) david.terrett@stfc.ac.uk; Author author@noreply.github.com Subject: Re: [SKA-ScienceDataProcessor/integration-prototype] Integration of prometheus alarms into SIP demo. (#76)

@bmort commented on this pull request.

Overall this looks excellent. Before approving however a couple of minor things:

1. I've taken the liberty of updating the PR description a bit to improve traceability in the git history. Could you check this and update accordingly if you think i've misunderstood or missed anything important.
2. Could the new skasip/<***> images be pushed to Docker Hubhttps://cloud.docker.com/u/skasip - I'm happy to help with this if you like or need to get a Docker Hub account sorted out.
3. Add some testing instructions to the PR description or a README.md file somewhere in the code with a pointer to that in the PR testing section.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/SKA-ScienceDataProcessor/integration-prototype/pull/76#pullrequestreview-198888974, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGuBMDh_q8vCxMdAB6p8WRABq3Ts9nBxks5vI5HTgaJpZM4aVvgK.