RFC: "Untrusted" Layer Addition

naftulikay commented 6 years ago

RFC for @lordnynex @asciifaceman @smaslennikov @catherinetcai @taup1n

Proposal

I propose adding another layer to the layers that we have for third party services which are trusted less. I am not set on the name, but "untrusted" is a candidate, I hope we can come up with something better.

Titan has the following layers presently:

DMZ (WAN-facing resources)
Routing (Internal layer 5 and 7 routing)
Services (Internal backend services and asynchronous workers)
Data (Internal data stores)
Admin (Internal administrative services)

I recently witnessed a contractor creating an entirely separate VPC for a Nominatim cluster, which is a third party location database. I am not entirely sure of the usefulness, hence the RFC. Running a service like this in an "untrusted" layer would allow tighter security rules for this area of the network in NACLs and security groups.

Pros

Network confinement for third-party untrusted services.

Cons

Is it really needed?
Does it really benefit and add to Titan?

slavaaaaaaaaaa commented 6 years ago

I'm not sure this is needed, as all daemon software we run is potentially untrusted and may or may not communicate with outside services. If I understand correctly, Nominatim is no different.

Another question is, how would this layer be different technically?

lordnynex commented 6 years ago

For a lot of reasons not worth describing, the network ACL's are extremely permissive. By design, 'titan' delegates a lot of security to actual security groups. I don't see a lot of value to an entire untrusted layer because there is still the overhead of wrapping any app or database in a security group that must be configured to allow the traffic.

naftulikay commented 6 years ago

@lordnynex yes this makes sense to me too. Everything is going to be confined by security groups anyway.

The only thing I see would be ading NACL rules to disallow all SSH access coming out of this new layer.

I will probably close and reject this, but I'll also wait for @taup1n @catherinetcai and @asciifaceman.

naftulikay commented 6 years ago

Okay, done waiting. Closing this because there is no good justification for it.

If you feel that you have a use case for this, simply tightly couple security groups and possibly add TLS if your security use case needs internal TLS.

For example, consider an ELK stack composed of static EC2 Elasticsearch instances, an Elasticsearch ELB, a Kibana instance, a Logstash auto-scaling group, and a Logstash ELB.

Here's a first go at a Kibana security group using CIDR-based loose coupling:

kibana/r.security_groups.tf

resource "aws_security_group" "kibana" {
  name = "..."
  description = "..."
  vpc_id = "${data.terraform_remote_state.network.vpc_id}"
  ingress {
    protocol = "tcp"
    from_port = 80
    to_port = 80
    cidr_blocks = ["${data.terraform_remote_state.network.routing_cidr_blocks"]
    ipv6_cidr_blocks = ["${data.terraform_remote_state.network.routing_ipv6_cidr_blocks"]
  }
}

This will allow all traffic from the routing layer to Kibana. The design of the routing layer is to provide layer 5 and 7 routing, so it makes sense that it will need to route into Kibana to be able to serve the UI over WAN if desired. (Put it behind authentication, you've been warned.)

Here's an example of tight coupling:

kibana/r.security_groups.tf

resource "aws_security_group" "kibana" {
  name = "..."
  description = "..."
  vpc_id = "${data.terraform_remote_state.network.vpc_id}"
  ingress {
    protocol = "tcp"
    from_port = 80
    to_port = 80
    self = true
    security_groups = ["${data.terraform_remote_state.routing.instance_security_group_id}"]
  }
}

With this tightly-coupled security group, we can only permit traffic directly from routing instances to the Kibana instance. No other traffic will be allowed. Via self = true, Kibana will be able to remotely access itself without the loopback interface.

This same example can be generalized to all of the ELK services. Tightly coupled security groups would look like this:

Elasticsearch Instances: Only allow TCP:9200 and TCP:9300 from the Elasticsearch load balancer and from Elasticsearch instances themselves. This means that all traffic to Elasticsearch must go through the load balancer except for inter-node communication.
Elasticsearch Load Balancer: Only allow TCP:9200 and TCP:9300 from Kibana and Logstash instances. This prevents an adversary in the network from accessing Elasticsearch from non-Kibana and non-Logstash nodes.
Logstash Instances: Only allow TCP:5140 from the Logstash load balancer.
Logstash Load Balancer: Allow syslog incoming traffic over TCP:514 (forward to instances on TCP:5140) from anywhere in the network, or in the case of a hub-spoke, any of the child networks via a list of IPv4 and IPv6 CIDR blocks. This is probably what you want if you're forwarding syslog to Logstash.
Kibana: Only allow TCP:80 from the routing layer CIDR blocks or specifically from your routing service's security group.

The above would have the following implications:

Nobody can access Kibana except for your routing service(s).
Nobody can access Logstash nodes except via the ELB.
Nobody can access Elasticsearch except via the ELB.
Nobody can access the Elasticsearch ELB except Logstash and Kibana instances.
The entire network can forward syslog traffic to the Logstash ELB.

The only way to make this more secure IMHO is to add TLS at all the hops, but that complicates the design tremendously and would mean TLS deployment for most internal services.

Phew. I'm closing this.

naftulikay / titan