search

giantswarm / roadmap

Giant Swarm Product Roadmap
https://github.com/orgs/giantswarm/projects/273
Apache License 2.0
3 stars 0 forks source link

Proposal for bastion hosts SSH session logging #857

Closed alex-dabija closed 1 month ago

alex-dabija commented 2 years ago

Propose alternatives for bastion hosts SSH session logging in order to have security compliant clusters. Any proposal needs to be agreed with our customers.

Background

Currently, SSH port forwarding is being used in order to access Vault. Customers have policies forbidding the use of SSH port forwarding and want every command executed on any host, including bastion hosts, logged.

Questions

whites11 commented 2 years ago

For context: the reason why we are using port forward is because opsctl needs to talk to the installation vault and that is not publicly reachable outside the installation LAN. We use the bastion host as a bridge and expose the vault service to the host machine through port forwarding. The actual SSH session than happens without any forwarding.

whites11 commented 2 years ago

Given we cannot avoid using vault for authentication the way we did so far (https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-certificates) I see the following options:

I really don't see any other option

whites11 commented 2 years ago

Regarding command logging, we could use auditctl:

auditctl -a exit,always -F arch=b64 -S execve
auditctl -a exit,always -F arch=b32 -S execve

this logs to syslog all calls to execve and thus all commands executed by anybody in the system. Example:

exit=0 a0=7f0e4a21e987 a1=7f0e4a21e6b0 a2=7f0e4a21e808 a3=8 items=2 ppid=906 pid=928 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="tail" exe="/usr/bin/tail" subj=kernel key=(null)
type=EXECVE msg=audit(1543671660.203:64): argc=2 a0="tail" a1="/var/log/audit/audit.log"
type=CWD msg=audit(1543671660.203:64):  cwd="/home/sweh"
type=PATH msg=audit(1543671660.203:64): item=0 name="/usr/bin/tail" inode=266003 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1543671660.203:64): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=273793 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0

this is ultra verbose and potentially extremely io intensive. I can't recommend running this on the k8s nodes themselves.

whites11 commented 2 years ago

@alex-dabija not sure how far you want me to go with the spike

VF-mbrauer commented 2 years ago

Hi @whites11

  • expose the vault service through the bastion host using a TCP proxy. Bastion hosts are only reachable via VPN (they don't even have a public IP) so this is NOT a security threat. Also would require a non-gigantic change to opsctl and keep working.

I don't agree here, as it is not about having a Private Network around and avoid people from outside accessing the system. The Problem is, that Security wants to know who did things on the system in case of any issue. Even GS staff or trusted people needs to be monitored in case of problems. There irrelevant of public- or internal available.

Cheers Maik

@alex-dabija cc

whites11 commented 2 years ago

Hi @whites11

  • expose the vault service through the bastion host using a TCP proxy. Bastion hosts are only reachable via VPN (they don't even have a public IP) so this is NOT a security threat. Also would require a non-gigantic change to opsctl and keep working.

I don't agree here, as it is not about having a Private Network around and avoid people from outside accessing the system. The Problem is, that Security wants to know who did things on the system in case of any issue. Even GS staff or trusted people needs to be monitored in case of problems. There irrelevant of public- or internal available.

Cheers Maik

@alex-dabija cc

What exactly do you disagree with? We've been asked two different things here: remove the usage of ssh tunnels AND log SSH activities. This sentence you quoted is about removing the SSH tunnel but your comment is about activity logging so I am confused.

VF-mbrauer commented 2 years ago

Hi @whites11

no worries, let me try to clarify more in detail.

First, I wanted to say regarding "disagreeing", that just because this is an internal flow and not a public one, it does not mean that it is less important. So, it is a security threat for us even it is just internal, as also internal flows can bypass the box with tunneling, and therefore, they might be out of logging.

The other thing was you said that you talked about Tunneling and I was talking about Logging. So, these topics are somehow also close to each other. If tunneling is there it would mean that on this box Logging is not possible as commands are not written as part of the session. So, it is a combination that plays together.

Hope I got things right now and sorry if I was not clear enough before. Also, happy to have a quick 1-2-1 if needed. Thanks a lot.

@alex-dabija cc

alex-dabija commented 2 years ago

We are currently evaluating Teleport as a possible solution to SSH to hosts. Teleport does have session logging capabilities.

T-Kukawka commented 1 year ago

@alex-dabija please move this issue to appropriate team - don't think it is Phoenix anymore

alex-dabija commented 1 year ago

@alex-dabija please move this issue to appropriate team - don't think it is Phoenix anymore

Yes, it's not Phoenix anymore. I moved it to team BigMac.

gawertm commented 1 year ago

we assume it will be included in teleport. however, keeping this task in the backlog for a while until we know teleport is working as expected

gawertm commented 1 month ago

works with teleport