Support multiple Logsene tokens

[otisg]

If I have 2 containers, one running Apache and the other one running Elasticsearch, just for example, I may actually want to send their logs to 2 different Logsene apps.

[megastef]

Current Status

• One Logsene App supports multiple types
• SPM agent identifies type of logs and extracts application specific fields, in your case the Websever logs get the type "access_log" and Elasticsearch logs get the type "elasticsearch". If the application type is unkown, the container id is used as type identifier. This means multiple Applications can log into a single Logsene App. For Kibana4 Dashboards in Logsene a query containing the type could be used. Or for global charts like total log activity numbers across all images/containers could be visualized ...

Design for the new feature "Support Multiple Logsene Tokens"

We should discuss three options:

• A - Static setup before starting SPM Agent. Disadvantage: would require restarts of the agent. The configuration would need settings for container name and Logsene token. I see a problem for the users to maitain it in dynamic environments for micro services (frequently changed setups / elastic scaling).
• B - Introduce an HTTP API to set Container "Log Routing" (like in Logspout). But I would add security to it (https, authentication) because logs could contain sensitive data, and changing the routing with unprotected API calls (like in Logspout) sounds not got to me.

• C - We could use the Docker API to scan the environment variables of started container for LOGSENE_TOKEN. Users simple add -e LOGSENE_TOKEN=token to the Containers run command (Nginx, Elasticsearch, ...) and SPM Agent will recognize that the Logs should be routed to the specified Logsene Applications. This could be extended to additional config settings like filters (e.g. to log only errors). Example:

  docker run -e LOGSENE_TOKEN=YOUR_ES_APP_TOKEN      
  LOGSENE_FILTER="error|exception|warning" -p 9200:9200 elasticsearch
  
  docker run -e LOGSENE_TOKEN=YOUR_WEB_APP_TOKEN -p 80:80 nginx

  Logs of Containers without LOGSENE_TOKEN will be routed to the default Logsene App specified during startup of sematext/spm-agent-docker.

I strongly recommend solution C. The Configuration is made per App container. Nobody needs to maintain configuration files for it, or write API interfaces. There is no network service involved, which means no API related security issues to deal with. Plus the auto discovery functions might be helpful for other features in the future.