Centralized logging

[max-zilla]

Previously, we had:

• InfluxDB for pipeline files & extraction statuses
• ELK stack at one point for python logging centralization
• Pipeline from Arizona just puts logs in text format

2018:

• centralized logging for extractors (for analysis of issues, troubleshooting, tracking)
• logging for pipeline
• queryable database of status with lifecycle tracking for a given file. for example, take a stereo RGB bin file and track it through Clowder, extractors, Level_1 outputs, etc.
• InfluxDB for extractor monitoring and visualization (consider redundancy)
• CheckMK OMD stuff for instances, CPU, disk monitoring.

Acceptance criteria:

• design meeting around how to structure
• design document https://docs.google.com/document/d/12R548_3ORN5mSGD4xrzNyfKYDnTcsD1hzR9AboyDpDo/edit
• design review
• creation of subsequent implementaton tasks

https://github.com/terraref/computing-pipeline/issues/380

[dlebauer]

For requirements I am going to add:

• number of times each data portal is accessed,
• how much data is retrieved
• something unique about the user
  • username where access is restricted
  • IP address otherwise

If a single sign / unified identify management across interfaces would help we should also discuss the time required to implement such a service

[max-zilla]

The linked google doc above has been updated. Look for "PROPOSED DESIGN - JAN 24, 2018" halfway down.

[craig-willis]

Removing from current sprint since we've deprioritized.