What is kops missing to support a HIPAA compliant cluster

[chrislovecnm]

Users we need feedback!!

[hollowimage]

disclaimer: the below is mostly in regrads to AWS.

1. tenancy = dedicated when creating launch configurations
2. encrypted storage for everything (this isnt super hard, but anything that PHI touches needs to be encrypted)
3. ideally default deny=all on SGs opening ONLY whats needed
4. ability to use a custom egress thats not a natGW

basically, anything that does NOT need to be open/public/unrestricted needs to be closed/private/restircted

some other thoughts:

private topology is pretty much a hard must-have.

https://aws.amazon.com/blogs/security/frequently-asked-questions-about-hipaa-compliance-in-the-aws-cloud-part-two/

the above services are presently hipaa-compliant in AWS:

Amazon DynamoDB Amazon Elastic Block Store (Amazon EBS) Amazon Elastic Compute Cloud (Amazon EC2) Amazon Elastic MapReduce (Amazon EMR) Amazon Glacier Amazon Redshift Amazon Relational Database Service (Amazon RDS) using only MySQL and Oracle engines Amazon Simple Storage Service (Amazon S3) excluding use of Amazon S3 Transfer Acceleration Elastic Load Balancing (ELB)

tenancy condition is a required one. in general you can create a tenanted VPC, but then you break other aws stuff that only works in shared (like aws lambda).

also bastion mode is a very good thing to have

[montyz]

HIPAA checklist

Generally speaking HIPAA compliance is all about protecting electronic Personal Health Information ePHI and Personally Identifiable Information PII. ePHI data must be secured and only accessible to appropriate personel. Many of these issues are best addressed by policy instead of software, but I would love to hear where kubernetes or kops or third-party software can help. Here's my shortlist:

1. ePHI/PII over the wire must be encrypted
2. ePHI/PII sitting on disk is encrypted
3. audit controls: records of actions taken to alter configuration of servers
   • Keeping configuration in version control desirable
4. establish a method for periodic security updates
5. guard against malicious software
6. monitor login attempts and track any discrepancies
7. password management: creation, safeguarding, changing
8. data backup plan
9. unique user identification (don't share logins)
10. emergency access procedure
11. dedicated instances are required
12. separate PII (personally identifiable info), PHI (health info that’s protected) and other info
    • don’t comingle PII and PHI if possible
13. have processes about off-boarding - making sure you revoke access for users in a timely fashion
14. log (separately) any access to PII / PHI data
15. data backup - separate PII and other if possible, make sure it’s encrypted; make sure the backups are tested (restored) once in a while
16. granular access control if possible - some users can only see PII, others only PHI - very few both
17. auto-logoff (require re-auth frequently enough)
18. security incidents detected and reported

Specifically relevant to kops:

• etcd encrypted storage and communication
• howto upgrade a base image to incorporate security patches (hopefully rare, mainly for docker issues?)
• pointers to third party network encryption addons
• auditing--what is logged by kops & kubectl
• dedicated instances possible by specifying kops to use a VPC with tenancy set to 'dedicated'

[wmbutler]

Many of these issues could be mitigated by allowing the user to specify:

1. ami (since the AMI could be prebuilt to support encrypted volumes and current patches to OS)
2. tenancy - dedicated

[andrewl3wis]

Actually as of Mid-May dedicated Tenancy is no longer a requirement. It was never an actual HIPAA requirement, but something that AWS required to keep in compliance with the BAA you had to sign with them. Now that this is gone it's made using just about everything easier.

Two big gotcha's now: At Rest Encryption - AWS is somewhat opaque about this but most people interpret this to mean using AWS encryption on all EBS volumes, which means that the root AMI needs to be encrypted.

In transit - All PHI/PII needs to be encrypted in transit. Some folks manage this via using SSL to secure all connections at the service level, but a more brute force method is to encrypt all network traffic. I use opportunistic strongswan connections and PKI, but weave also solves this issue(and that is what I am going to test using in kops). Only downside with weave is shared password that needs to be preloaded.

[cordoval]

@meatlewis how is your test coming alone? Have you tested calico?

@wmbutler i think no, because with ami you make it too specific the settings and will reduce the options for having most systems supported. There is for instance CoreOS tools etc.

I found a way to mitigate the logging of kops and kubectl by using perhaps http://gravitational.com/teleport/ but that is paid. There is a script command that records and saves to a file but not to a database service. Wish there was an open source with an agent to do this open sourcedly.

@montyz great list compiled, however i think you in a sentence confuse PHI with PII, i think PII can include PHI when you connect the clinical data with the names for example. But then I don't have as much experience so can you clarify please?

[abhiyerra]

I've been trying to secure the base images provided by kops with a few things for HIPAA and PCI compliance for my customers. One is the encrypted boot volume and second is adding a file system audit check using OSSEC

https://github.com/opszero/auditkube

The next components that I am looking into are 2FA Login and Authentication and potential Logging Solutions.

[cordoval]

related https://github.com/kubernetes/kops/issues/1368#issuecomment-278235552

[chrislovecnm]

What is the status of this? Can we get some documentation started?

[andrewl3wis]

@cordoval Weave works great, I've looked at calico as well but it needs IPsec support which you'd have to roll on your own. It's not impossible but it most likely outside the scope of kops. For folks looking to get into that strongswan past 5.3.0 has support for opportunistic encryption. You could use a pre-loaded shared key and the "admin networks" underlying IP ranges to create opportunistic encryption. Since Calico uses VXlan it should just work.

@abhiyerra Teleport has an open source version. We're approaching this by moving controls upstream and uses Terraform/Jenkins to apply infrastructure as code. All our audits are going to be on that, and direct access will be heavily restricted.

[cordoval]

@meatlewis do you have more pointers on this? i just getting ready to start upgrading my network, i currently use kubenet

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta. /lifecycle stale

[cordoval]

/lifecycle frozen

[gankit]

Question - do people prefer Weave or Calico for network encryption?

[gankit]

/remove-lifecycle stale

[ReillyTevera]

@gankit Perhaps I'm not seeing it but I'm not seeing that Calico supports network encryption directly at the moment. Weave does so we would prefer to have that as an option.

Flannel also appears to be close to having an encrypted overlay network: link

[chrislovecnm]

/lifecycle frozen /remove-lifecycle stale

[iamonkara]

any progress on this ?

[haz-mat]

About a year ago I did an in-depth PoC with Kops for a healthcare provider on AWS. Nothing about Kops inherently prevented us from having a fully HIPAA+SOC2 compliant cluster. However, we ultimately decided to build the cluster with kubeadm and terraform on Container Linux. This was mostly because we found it easier to integrate into our existing VPC, which is something we very much wanted to do because of the way our edge appliances worked. More generally we wanted a higher degree of flexibility than Kops offered despite it being more work.