Get alerts when any something unexpected happens

[zelima]

Originally coming from https://github.com/datahq/datahub-qa/issues/158

As a project manager, I want to be notified when something goes wrong Eg: service is down, service not responding as expected, processing data takes too long so that I'm aware problem exists and quickly make actions to fix

Acceptance Criteria

• [ ] For now run daily and get notified via Email

Tasks

• [x] List all the services and expected responses for each
• [x] create repo datahub-health
• [ ] create python module with functions testing each service
  • [ ] collect errors/success
  • [ ] Save as CSV(s)
  • service,endpoint,method,success,errors,date
• [x] setup travis to run script
• [ ] setup travis to push csv as datasets

Analysis

Services:

• Specstore https://github.com/datahq/specstore
  • GET /source/{owner}/{dataset-id}/{revision-number}
  • GET /source/{owner}/{dataset-id}/latest
  • GET /source/{owner}/{dataset-id}/successful
  • POST /source/upload
  • POST /source/update
• auth
  • GET /auth/check
  • GET /auth/authorize
  • GET /auth/update
  • GET /auth/public-key
  • GET /auth/callback (???)
  • GET /auth/resolve
  • GET /auth/profile
• filemanager https://github.com/datahq/filemanager
  • GET /storage/info//
  • GET storage/owner/ storage/dataset_id/ storage/flow_id/
• Bitstore https://github.com/datahq/bitstore
  • POST /authorize
  • GET /info
  • GET /presign
• Plans https://github.com/datahq/plans
  • /plans (Needs login)
• Resolver https://github.com/datahq/resolver
  • GET /resolver/resolve
• Metastore https://github.com/datahq/metastore
  • GET /metastore/search
  • GET /metastore/search/events
• Frontend: https://github.com/datahq/frontend
  • GET /
  • GET /{owner}/dataset
  • GET /Search

Also Things to check: Speed of the push and success

How should we monitor

We have to options:

1. Create a module and run scripts on Travis daily and save the outcome as a dataset on the gihub. Travis will take care of sending Emails if build fails
   • We could even not save anything and just look at the travis build history. But think saving data with error messages would be useful
2. Create a module as a service and serve an endpoint: api.datahub.io/health/check with GET and POST methods. One would trigger and run the checks and other just display history.

While 2nd options looks better and cleaner (and maybe useful in future) it will require additional day or 2 work while other should not take more then 5-6 hours. To setup server, setup continuous deploy etc..

So I think better and faster way would be option 1. That will one once in a day and send Emails of buid will fail. This way we can react on the problem within max 24 hours.

What should we check

We should create (if not already exists) the test users with several test datasets pushed on datahub. + give limitations

• owner: tester
• datasets from test data:
  • 5kb
  • 1mb
  • 10mb

Specstore

• GET Latest revision returns status 200

• Push the dataset

  • POST /source/upload without content
  • Check status code
  • Check success is False
  • Check error message is correct
  • POST /source/upload without owner in content
  • DITTO
  • POST /source/upload wrong owner in content
  • DITTO
  • POST /source/upload without JWT
  • DITTO
  • POST /source/upload with new dataset to exceed limit
  • DITTO
  • POST /source/upload with Invalid content
  • DITTO
  • POST /source/upload with invalid schedule
  • DITTO
  • POST /source/upload with valid content and start timer
  • Wait reasonable time
    • If exceeds exit with error
    • If fits in time proceed

• GET /source/{owner}/{dataset-id}/{revision-number}

  • GET non-existing revision number or/and word
  • Expect 404
  • GET non-existing owner or/and dataset
  • Expect 404
  • GET latest revision with number
  • Expect 200
  • GET /successful
  • Expect 200 and revision numbers match
  • GET /latest
  • Expect 200 and revision numbers match

• POST /source/update This endpoit not exists remove from README

Auth

• Check authenticated
  • GET without JWT or wrong JWT
  • Expect 200 and authenticated = False
  • Get with correct valid JWT
  • Expect 200 and user info in response
• Authorize for service
  • GET without JWT, without service or wrong JWT
  • Expect 200 and permission {}
  • Get permission with valid JWT and services
  • rawstore specstore:
    • expect 200, permissions filled and Token to be there
• Check user update
  • POST without JWT, invalid JWT or username
  • expect 200 and success False
  • POST with existing username or unknown user (wrong userid)
  • expect 200 and success False
• Check public key public-key
  • Get /public-key
  • expect 200 and public key in pem format
• Check resolver
  • Get /resolve with invalid username
  • expect 200 and user = null
  • Get /resolve with valid username
  • expect 200 and user = userID
• Check get profiel
  • Get /profiel with invalid username
  • expect 200 and found = false
  • Get /profiel with valid username
  • expect 200 and found = True

Filemanager

• Check file info
  • Get pkgstore.datahub.io /path to the file
  • Expect 200 and found True valid file info
  • Get non-existing file
  • Expect 200 and found False
• Check owner storage
  • Get Valid owner
  • Expect 200 totalbytes > 0
  • Get invalid owner
  • Expect 200 and totalnbyte = 0
• Check dataset storage
  • DITTO
• Check flow storage
  • DITTO

Bitstore

• Check info
  • Get /rawstore/info
  • Expect 400
  • Get /rawstore/info valid token (from auth/authorize)
  • Expect 200 and valid prefixes
• Check authorize
  • POST /rawstore/authorize invalid token or no user
  • Expect 400
  • POST /rawstore/authorize invalid username
  • Expect 401
  • POST /rawstore/authorize with valid body
  • Epxect 200 and file info
• Check presign
  • GET /rawstore/presign invalid jwt or missing owner
  • Expect 400
  • GET /rawstore/presign incorrect user
  • Expect 401
  • Get /rawstore/presign valid headers and URL not needing presign
  • Expect 200 and same url Get /rawstore/presign valid headers and URL needing presign
  • Expect 200 and signed url

Resolver

• Check resolves
  • GET /resolver/resolve with invalid user ID
  • Expect 200 and userID null
  • GET /resolver/resolve with valid user ID
  • Expect 200 and userID = username

Metastore

• Check datasets
  • GET /metastore/search
  • Expect 200 and len results > 0
  • GET /metastore/search authorized
  • Expect 200 and len results > 0
• Check events GET /metastore/search/events
  • Expect 200 and len results >2 GET /metastore/search/events with authorized id
  • Expect 200 and len results >2

Frontend:

• Check home page
  • Get datahub.io
  • Expect 200
• Check Search
  • Get datahub.io/search
  • Expect 200
• Check showcase
  • GEt datahub.io/tester/dataset
  • Expect 200

plans

• Check not allowed
  • Get /plans
  • Expect 401

[zelima]

FIXED for now just getting Emails - no CSV