search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.66k stars 1.77k forks source link

Locks do not get removed in trusted clusters #33619

Open myoung34 opened 1 year ago

myoung34 commented 1 year ago

Expected behavior:

Locks can be managed across leaf clusters

Current behavior:

Locks are created but not deletable in leaf clusters

Bug details:

Cluster setup:

➜  ~ tsh clusters
Cluster Name                   Status Cluster Type Labels Selected
------------------------------ ------ ------------ ------ --------
teleport.test.ue1.redact.net online root                *
teleport.test.uw1.redact.net online leaf
teleport.test.uw2.redact.net online leaf

Root node 

sh-4.2$ hostname -f
teleport.test.ue1.redact.net

sh-4.2$ sudo /usr/local/bin/tctl get locks

Run tctl lock --user=myoung34-testing --message="Test" --ttl=1h

Check locks on root node: 

sh-4.2$ sudo /usr/local/bin/tctl get locks
kind: lock
metadata:
  id: 1697593029690519733
  name: 697cb761-7ce9-43b9-babb-aa113cb94e42
spec:
  created_at: "2023-10-18T01:37:09.690173065Z"
  created_by: api-admin
  expires: "2023-10-18T02:37:09.615955401Z"
  message: Test
  target:
    user: myoung34-testing
version: v2

Check locks on leaf node:

[myoung34@teleport ~]$ hostname -f
teleport.test.uw1.redact.net

[myoung34@teleport ~]$ sudo /usr/local/bin/tctl get locks
kind: lock
metadata:
  id: 1697593030218273816
  name: teleport.test.ue1.redact.net/697cb761-7ce9-43b9-babb-aa113cb94e42
spec:
  created_at: "2023-10-18T01:37:09.690173065Z"
  created_by: api-admin
  expires: "2023-10-18T02:37:09.615955401Z"
  message: Test
  target:
    user: myoung34-testing
version: v2

At this point everything is fine and expected

Delete the lock tctl rm lock/697cb761-7ce9-43b9-babb-aa113cb94e42

Check the root node:

sh-4.2$ sudo /usr/local/bin/tctl get locks
sh-4.2$

The logs on the root node

Oct 18 01:37:09 teleport teleport[17093]: 2023-10-18T01:37:09Z INFO [AUDIT]     lock.created cluster_name:teleport.test.ue1.redact.net code:TLK00I ei:0 event:lock.created expires:0001-01-01T00:00:00Z name:697cb761-7ce9-43b9-babb-aa113cb94e42 user:myoung34-testing time:2023-10-18T01:37:09.698Z uid:eb322eea-4f6f-44f5-80a7-4899bef0e15c updated_by:api-admin user:api-admin events/emitter.go:271
Oct 18 01:37:55 teleport teleport[17093]: 2023-10-18T01:37:55Z INFO [AUDIT]     lock.deleted cluster_name:teleport.test.ue1.redact.net code:TLK01I ei:0 event:lock.deleted expires:0001-01-01T00:00:00Z name:697cb761-7ce9-43b9-babb-aa113cb94e42 time:2023-10-18T01:37:55.596Z uid:eda72e3f-006a-4540-aea7-de7a9b7c8113 user:api-admin events/emitter.go:271

Everything is expected, except that the lock persists and is undeletable on all leaf nodes:

[myoung34@teleport ~]$ hostname -f
teleport.test.uw1.redact.net

[myoung34@teleport ~]$ sudo /usr/local/bin/tctl get locks
kind: lock
metadata:
  id: 1697593030218273816
  name: teleport.test.ue1.redact.net/697cb761-7ce9-43b9-babb-aa113cb94e42
spec:
  created_at: "2023-10-18T01:37:09.690173065Z"
  created_by: api-admin
  expires: "2023-10-18T02:37:09.615955401Z"
  message: Test
  target:
    user: myoung34-testing
version: v2

It seems undeletable as well 

[myoung34@teleport ~]$ sudo /usr/local/bin/tctl rm lock/teleport.test.ue1.redact.net/697cb761-7ce9-43b9-babb-aa113cb94e42
ERROR: lock "teleport.test.ue1.redact.net/697cb761-7ce9-43b9-babb-aa113cb94e42" is not found

Im unable to remove this lock in any capacity from the leaf nodes

The locks are valid too. I can log into the root node (shows no lock) but am still locked out of leafs (that show locks)

➜  ~ tsh login --proxy=teleport.test.ue1.redact.net teleport.test.ue1.redact.net --auth=github
> Profile URL:        https://teleport.test.ue1.redact.net:443
  Logged in as:       myoung34-testing
  Cluster:            teleport.test.ue1.redact.net

➜  ~ ssh teleport.test.ue1.redact.net "sudo /usr/local/bin/tctl get locks"
➜  ~
➜  ~ tsh login --proxy=teleport.test.ue1.redact.net teleport.test.uw2.redact.net --auth=github
> Profile URL:        https://teleport.test.ue1.redact.net:443
  Logged in as:       myoung34-testing
  Cluster:            teleport.test.uw2.redact.net

➜  ~ ssh teleport.test.uw2.redact.net "sudo /usr/local/bin/tctl get locks"
channel 0: open failed: administratively prohibited: lock targeting User:"myoung34-testing" is in force: Test
stevenGravy commented 1 year ago

A workaround in the leaf cluster is to export the locks to a YAML file. You can then add a expires date in the past. Apply the locks update (tctl create -f locks.yaml) and the lock will no longer be effective.

...
spec:
  target:
    user: testlock
  expires: "2023-03-28T15:10:03.629419018Z"
myoung34 commented 1 year ago

I'd have to do this anywhere the locks exist, correct? (n-1 proxy servers)

stevenGravy commented 1 year ago

I'd have to do this anywhere the locks exist, correct? (n-1 proxy servers)

Yes, you would.

coltonparsons-vgs commented 1 month ago

The locks appear to be un-deletable because of the slash / in their names. This applies to a few other situations too, so I've opened https://github.com/gravitational/teleport/issues/47838 to describe this issue in the general case.