Configure AMI to forward logs to Elasticsearch host

[nickclyde]

Fixes #132

This should configure all instances to forward app logs from svlogd to rsyslogd, and configure rsyslog to forward logs to an Elasticsearch host. Further work needs to be done to set up an AWS Elasticsearch Domain for each application. I have a few questions before I can continue:

1. By default svlogd forwards to port 514, but I'm not exactly sure if rsyslog listens on that port by default (netstat -anp | grep 514 returns nothing on an instance), how do I set it to listen on that port for UDP traffic?
2. It's unclear to me where the best place to put the terraform config for the ES domain instance. Under ops/aws/terraform should it go in the deployment folder? Also, should we only have on ES instance per app, or one per env?
3. Once the config for provisioning an ES domain is set up, how do we define {{ elastic_search_ip }} for the Ansible template? I'm still pretty new to Ansible.

Resources used: http://serverascode.com/2016/11/11/ryslog-to-elasticsearch.html https://stackoverflow.com/questions/34822533/how-to-install-rsyslog-v8-15-on-amazon-linux-ami https://unix.stackexchange.com/questions/183423/combine-svlogd-logs-from-different-runit-services

[nickclyde]

I uncommented $ModLoad imudp and $UDPServerRun 514 in /etc/rsyslog.conf and restarted rsyslog, that seemed to do the trick, so I will add that to the playbook.

@paulsmith Still would like some input on the other points.

[paulsmith]

I would like to keep things as simple as possible here for now. I'm thinking that we should just set up a single ES instance in the Ad Hoc AWS account, with a security group that permits 514 from any of our VPCs' external addresses, and then just hardcode the ES instance's hostname into the rsyslog config for now. There's a lot of moving parts introduced by this, and it also complicates and makes longer initial app creation if we do it per app (I think per env is too much), that I would like to tackle that piece of things in a separate issue.

[nickclyde]

Okay, I've set up an ES instance at https://search-soapbox-logs-e4c42dccry6rjee2zqj5fh54cm.us-east-1.es.amazonaws.com and configured it to receive traffic from any resource in our AWS account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::968246069280:root"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:968246069280:domain/soapbox-logs/*"
    }
  ]
}

Looks like AWS ES instances receive regular web traffic instead of the default 9200 ES port, so I configured rsyslog to send over port 443.

I think this should be good to go now, we should do a test deployment on this branch. I'll give it a shot a little bit later.

[nickclyde]

Okay, finally got some insight to this. When I ssh'ed into my lordemoji instance, none of the files that should have been introduced via my Ansible changes were present, even though I created the app while on the elasticsearch-logs branch. When I installed the packages/files via ssh, the logs were sent to the ES instance successfully! But I'm a little lost as to why my playbook changes weren't successful.

[nickclyde]

Aha, figured out the issue. Forgot to run packer build soapbox-app.json. Will run that and then try a deployment with the new AMI.

[nickclyde]

Okay, with the new AMI it works! Last thing we need is to configure access to the ES instance, right now I have it fully open to test. To limit access to our own resources, we need to attach AmazonElasticsearchServiceRolePolicy to the soapbox-app role in IAM here, but for some reason I'm getting an error. I think it may be a permissions thing, so @paulsmith may need to add the policy.

[paulsmith]

There is some kind of AWS API error happening at the moment when I try to attach the managed policy to the IAM role: So we'll have to revisit this later and try again.

[nickclyde]

Yep, same error I'm getting. @paulsmith can we just merge this in and leave the ES instance open for now?

[nickclyde]

@paulsmith Ended up using the AWS CLI to attach a different policy to the role:

aws iam attach-role-policy --role-name soapbox-app --policy-arn arn:aws:iam::aws:policy/AmazonESFullAccess

That seemed to do the trick: after updating the access policy for the ES instance, logs are still being received! :+1: I also allowed access to the Kibana page for the exit IP of the pritunl VPN, so you should be able to view the logs if you are connected to the VPN: https://search-soapbox-logs-e4c42dccry6rjee2zqj5fh54cm.us-east-1.es.amazonaws.com/_plugin/kibana/app/kibana

So I think this should be totally ready to go now!

[paulsmith]

@ClydeDroid thanks for seeing this one through!