search

hasadna / knesset-data

APIs and documentation to allow getting data from the Israeli Parliament (Knesset)
MIT License
10 stars 12 forks source link

should have a knesset dataservice health monitor #4

Closed OriHoch closed 7 years ago

OriHoch commented 8 years ago

User Story

Open Knesset periodically (every night) gets votes data form The Knesset using the Knesset Dataservice.

On March. 2 there was a problem on Knesset side which caused votes api to stop working

Expected

should have a process that periodically (daily) checks the knesset dataservice services and automatically open new issues if problems are encountered

it should check:

OriHoch commented 8 years ago

this could be a python command that will run periodically via cron

this command should:

daonb commented 8 years ago

What's the base? py.test?

OriHoch commented 8 years ago

whatever you wish.. open to suggestions

bennybauer commented 8 years ago

@OriHoch Hi, I'm hasadna newbie. This issue should be pretty straightforward and i'd like to help.

Where the health check can be deployed? Does Hasadna have any budget for compute resources (AWS, Heroku, etc...)?

I've noticed that there's already code for the check. IS anything missing there?

Thanks, Benny

OriHoch commented 8 years ago

thanks, the metadata check code you linked to is a service that checks the metadata about the knesset services. not the data itself

this issue is about something similar that will check the data itself for any inconsistencies. at the most basic level it will check if the service returns http status 200 or not, more advanced stuff will be to open the data itself and use some kind of heuristics to determine if it's valid or not.

regarding deployment - we have everything needed to deploy this service. This issues should just be a python command which we will run somehow / somewhere

bennybauer commented 8 years ago

For fast solution, checking only the http status code, we can use a service like StatusCake, the free tier will be sufficient for this. Basically this is it: http://uptime.statuscake.com/?TestID=IEdmFQLe1V (I've created it on my free account, we should create an account for hasadna).

It has nice integrations, so when one of the endpoints is down you can trigger email, Slack, public tweet (could be nice!). If needed, it's possible to plug the email or slack to create a github issue. BTW, slack should be free for non profit orgs.

WDYT?

OriHoch commented 8 years ago

Sounds good

On Sun, Jun 5, 2016, 11:27 Benny Bauer notifications@github.com wrote:

For fast solution, checking only the http status code, we can use a service like StatusCake http://statuscake.com, the free tier will be sufficient for this. Basically this is it: http://uptime.statuscake.com/?TestID=IEdmFQLe1V (I've created it on my free account, we should create an account for hasadna).

It has nice integrations, so when one of the endpoints is down you can trigger email, Slack, public tweet (could be nice!). If needed, it's possible to plug the email or slack to create a github issue. BTW, slack should be free https://slack.com/pricing#slack_for_good for non profit orgs.

WDYT?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hasadna/knesset-data/issues/4#issuecomment-223799853, or mute the thread https://github.com/notifications/unsubscribe/ABJLBgIu2ckjdczQg8hin3duIrz9A6ggks5qIohzgaJpZM4HRTpS .

bennybauer commented 8 years ago

So how do we take it from here?

  1. What should be the strategy for endpoint downtime handling? Open a github issue, or just sending a link to the dashboard? What should be the distribution list?
  2. Is there a slack team? do we want one?
OriHoch commented 8 years ago

Yes we have slack , I'll invite you

On Sun, Jun 5, 2016, 11:36 Benny Bauer notifications@github.com wrote:

So how do we take it from here?

  1. What should be the strategy for endpoint downtime handling? Open a github issue, or just sending a link to the dashboard? What should be the distribution list?
  2. Is there a slack team? do we want one?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hasadna/knesset-data/issues/4#issuecomment-223800171, or mute the thread https://github.com/notifications/unsubscribe/ABJLBnoVNRChbKBbgUyQXwLUbIjPsTRqks5qIoqWgaJpZM4HRTpS .

OriHoch commented 8 years ago

@bennybauer - please send me you email, you can send me in twitter

uda commented 8 years ago

@OriHoch and @bennybauer

  1. I should be able to get from slack the NPO upgrade after I fill in the form
  2. I created an account with status cake, but they don't seem to have NPO discounts.

So we can either save the tests for status cake in a repo and update via api https://www.statuscake.com/api/ Or, we keep the enterprise solutions for infrastructure monitoring, and provide an organization wide instance like Zabbix http://www.zabbix.com/license.php

OriHoch commented 8 years ago

personally I like the option of calling statuscake api from our server

bennybauer commented 8 years ago

@uda ,

  1. maybe @morchickit is already on the slack NPO, better check with her
  2. I prefer statuscake (or any other hosted solution) on maintaining Zabbix :)

If we're going on with statuscake please send me the credentials over slack and I'll handle this issue.

OriHoch commented 7 years ago

too much changed since this issue was opened, it's less relevant now that we use the datapackages