Document the application logging facilities provided by the fluent service

gene1wood commented 9 years ago

Can you link to or write up a description of

how an application in nubis would use fluent to push it's logs out so that they could be accessed by Opsec?
how the fluent aggregation works (pointer to IAM role it uses, pointer to s3 buckets it uses, pointer to security group rules that allow clients to talk to aggregator, etc)

In our conversation a few weeks back you described this (which I took notes about), but I'd like to get the official story (what you've actually implemented)

A Fluentd collector will be deployed in both the staging and production VPC for each product ( http://www.fluentd.org/ )
    This collector will have an IAM role granting it write access to an S3 bucket
    It will have a security group granting Shared Services systems to connect inbound to it
    It will announce its logging service to consul
Fluentd agents will be deployed on all instances
    The agents read on disk local log files
    Parse them to turn them into structured data
    Send the structured data to Fluentd collector over TCP in JSON

tinnightcap commented 8 years ago

@gene1wood is there an entry on the RRA for this issue that I can link against? I need to know the level of risk. - Thanks

tinnightcap commented 8 years ago

@gozer will write more docs

gene1wood commented 8 years ago

Yes, this issue is linked from the #applogging RRA risk.

nubisproject / nubis-fluent-collector

Document the application logging facilities provided by the fluent service #22