Unable to run plugin for multiple AWS accounts

[joschi]

Due to the changes in c85cbbe079eef1075e088a20b95efa9046140393 it is now impossible to run this plugin multiple times for different AWS accounts on the same Graylog cluster.

[jhaar]

Indeed. We have a "prod" AWS account and a "dev". Complete separate AWS accounts.

I'd love to be able to use one AWS username over many AWS accounts - but I don't think that's supported? So we have to have separate usernames (and therefore tokens) for each monitored AWS account

(I hope I've got my terminology correct - I'm the security guy - not the AWS guy! ;-)

[brucegivens]

We also use multiple AWS accounts, 3 at the moment - would be great to get a fix for this!

[wrsuarez]

+1 for needing multiple account inputs. We have over 20 AWS accounts for various purposes and my only recourse now is to set up Graylog forwarders on each account to access the local data using IAM users specific to that account and forward them to GELF inputs on our main production Graylog server. I've set this up with a CloudFormation script that builds it all out and provisions the instance up to and including sending API calls to the new Graylog instance to configure it but a single server with multiple account access would be magic.

[Ashex]

I'm surprised this has been an issue for so long as a multi-account setup is rather normal these days. I'm evaluating Graylog for consuming CloudTrail and this is a major blocker.

[hezor]

Anything new with this issue?

[Ashex]

I came up with a workaround of having multiple SQS queues in one account that are subscribed to the Cloudtrail SNS topic in each account.

http://docs.aws.amazon.com/sns/latest/dg/SendMessageToSQS.cross.account.html

[wrsuarez]

That works for Cloud trail but not VPC Flowlogs

On Mon, Jan 2, 2017 at 2:46 AM -0800, "Ahmed" notifications@github.com<mailto:notifications@github.com> wrote:

I came up with a workaround of having multiple SQS queues in one account that are subscribed to the Cloudtrail SNS topic in each account.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Graylog2/graylog-plugin-aws/issues/13#issuecomment-269956624, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFKDLTE1m4JUlxfqmCWFqMlKn6P4IN6Fks5rONV9gaJpZM4Jac7M.

[Ashex]

You can do the same with kinesis streams.

http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CrossAccountSubscriptions.html

The best way to implement this would be to have graylog consume from a dedicated aws account where the queues and streams live.

[wrsuarez]

Sure if I set up cross account access on every account. But we have over 40 we manage. I'd prefer to run a cloud formation script on each account that turns on all the logging I want then makes an API call to our main Graylog server to add each accounts access keys.

On Mon, Jan 2, 2017 at 7:54 AM -0800, "Ahmed" notifications@github.com<mailto:notifications@github.com> wrote:

You can do the same with kinesis streams.

http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CrossAccountSubscriptions.html

The best way to implement this would be to have a dedicated aws account where the queues and streams live which graylog consumes from.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Graylog2/graylog-plugin-aws/issues/13#issuecomment-269989307, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFKDLe-YK4wixzm9d_cqN5JPn0U-RVXmks5rOR2cgaJpZM4Jac7M.

[Ashex]

It's a really crappy workaround for Flow Logs as VPCs can be thrown away very easily (requiring a lot of maintenance to add/remove), I can live with it for CloudTrail since it's a one time setup.

That said, #28 has the right approach in moving away from Credentials when graylog is running AWS. Ideally this would be reimplemented with the ability to assume a role or use provided keys.

[radykal-com]

Maybe the best way to allow multiple credentials and also use the authentication chain to use the assume role (in ec2 instances) would be some kind of auth hierarchy. Like, having cluster wide authentication (set through the default aws sdk chain like env vars, file system credentials file, assume role, etc...) and then a per input credentials override like it was before. This way, if you don't specify any credentials in the input it will go through the default auth chain until it can auth (allowing the AssumeRole auth, that actually doesn't work), otherwise if you set the access key and token in the input itself, these values will be the only ones used for this input.

[radykal-com]

Im about to develop the feature as described in my previous post. Want to make a PR back when done? This will allow authenticate each input in this order: 1º Input specific credentials (aws, key and secret), like it was before the breaking commit commented by @joschi 2º Global Plugin config credentials (the one working in current builds) 3º Default client auth chain that will allow to use the environment vars, instance profile, etc (all the default ways the sdk uses by default)

[radykal-com]

Got a PR to review #36

[bernd]

This got fixed in #36.