Access to logs

[lottspot]

Docker container logs are being shipped into an elasticsearch backend by default. Project owners need to be able to view and search these logs.

[lottspot]

I've been looking into this one extensively, and it's actually a little bit tricky because the obvious solution is not suitable for multitenant access.

• The de facto solution to this problem is to use Kibana; however, Kibana has no built in access controls without a commercial plugin. We would need document-level security within Kibana to ensure project owners only have access to their own logs.
• There are free alternatives to the commercially licensed plugin, but they only provide index-level security.
  • In order to create a separate index for each project, we would need to add a new fluentd configuration entry by hand for each pod deployed by a project; most projects will probably need to deploy multiple pods. Cumbersome, but possible.
  • The free alternatives may only be compatible with older Kibana versions
• It seems that the only viable Kibana alternative out there is Grafana. Grafana may be suitable to our needs however, as it provides fairly robust role-based access control out of the box. What will be unclear until I'm able to experiment more is whether or not it provides suitable capability to search and analyze log entries to end users.

Right now, I think Grafana is the best hope of putting together per-project log collections from the elasticsearch database where container logs are being shipped. Won't know more until I've had a chance to mess around with it though.

[lottspot]

So after additional research and experimentation, I've more or less concluded that this problem is going to be more time consuming to solve than I wanted it to be. In order to set up quality multitenant access to container logs, I think we basically need to tear down the logging infrastructure which is deployed with the cluster addons and stand up something slightly different in its place. I wrote up a bit on how I would like this to be architected in the alpha cluster deployment wiki page.

[MooseQuest]

Defining the stanzas need to ingest logs and then allowing them to be seen per tenant can be achieved many ways. Setting up a consistent logging structure that is either:

1. Collected on the mounted file system which can be read by the ingest daemon
2. Forwarded to the Ingest Daemon with HTTP Event Collection (JSON)

With these two, especially the HEC method, we can tokenize the logs allowing someone to view logs that way.

For example:

Amazon uses SNS and SQS to allow the data to be logged and reviewed. While the pipelines need to be made, this still allows for you to use Object Storage for logging. Therefore setting up the consumers might be an easier method for logging structured data.

[lottspot]

This one turned out to be a doozy. I had to trash the entire architecture I can up with, for so many reasons, the most important of which were:

1. Too much wasted time trying to work around stupid difficulties
2. The solution I had originally thought up quite frankly sucks for multitenancy

Instead, I opted to deploy a graylog virtual appliance, and all logs are being shipped to it. We can easily give people access to graylog by creating accounts for them, and creating and granting access to data streams which contain their logs. Good stuff.

[lottspot]

The fluentd part of this pipeline was deployed in ce335c4. Unfortunately, nothing for the graylog instance is in git yet.