Open mattirantakomi opened 1 year ago
cc @maisem @andrew-d
https://github.com/tailscale/tailscale/blob/ffaa6be8a4d84f3c5595328a44f3df2a8cf92e7f/ssh/tailssh/incubator.go#L792-L794 This will ignore the error if tailscaled is running as non-root and the groups were already correct. On Google Cloud Run I believe it will be running as root. GCR's container runtime is unique and not a regular Linux container, but it does allow its processes to run with UID zero (the container makes sure this doesn't let the contained processes do anything harmful).
On Google Cloud Run container will run with that UID which is defined in Dockerfile
. I can confirm that Tailscale SSH is working fine on Cloud Run when container is running with root privileges.
I think that groupsMatchCurrent(groupIDs)
on incubator.go line 792 is not working as it should as tailscaled is trying to set groups even gids are already matching.
As you can see from the opening post, the failing syscall is:
setgroups(1, [1000]) = -1 EPERM (Operation not permitted)
And that is the reason why command exits with code 1.
@mattirantakomi, the code is:
func setGroups(groupIDs []int) error {
//...
err := syscall.Setgroups(groupIDs)
if err != nil && os.Geteuid() != 0 && groupsMatchCurrent(groupIDs) {
// If we're not root, ignore a Setgroups failure if all groups are the same.
return nil
}
return err
}
func groupsMatchCurrent(groupIDs []int) bool {
existing, err := syscall.Getgroups()
if err != nil {
return false
}
if len(existing) != len(groupIDs) {
return false
}
groupIDs = slices.Clone(groupIDs)
sort.Ints(groupIDs)
sort.Ints(existing)
return slices.Equal(groupIDs, existing)
}
So it should first fail to set, but then see you're non-root, but then see they match (groupsMatchCurrent
) and thus succeed.
What are your current groups in a Google Cloud Run container? Is syscall.Getgroups
also returning permission denied?
Full strace log attached: tailscale_strace.txt
Test user "www" with uid 1000 belongs to group "www" with gid 1000.
www@localhost:~$ id
uid=1000(www) gid=1000(www) groups=1000(www)
What is the issue?
I tried to open ssh connection from my own computer (Ubuntu 22.04 LTS) to a container running Tailscale on Google Cloud Run service but connection failed with "operation not permitted" error message.
From Google Cloud Run logs I can see that the following command exited with code 1:
I tried to debug with strace and found out that the failing syscall is setgroups.
As far as I know this bug should have been already fixed on https://github.com/tailscale/tailscale/pull/6904 but I'm not sure about it.
Steps to reproduce
Get tailscale auth key from Tailscale Admin Web UI -> Settings -> Keys -> Auth keys -> Generate auth key (https://login.tailscale.com/admin/settings/keys). Use generated key as Cloud Run environment variable
TAILSCALE_AUTHKEY
.Run test container (https://github.com/mattirantakomi/cloudrun-nginx-test) on Google Cloud Run. Nginx is included and responding on port 8080 just to make Cloud Run health check happy.
I tried both gen1 and gen2 execution environments, neither doesn't work.
ssh www@100.11.22.33
.Are there any recent changes that introduced the issue?
-
OS
Linux
OS version
Ubuntu 22.04 LTS
Tailscale version
1.42.0
Other software
No response
Bug report
BUG-0937c81c9c29b37dc74221ea0bec337231bfcf36999b0a0ff752edde9aa2d8f2-20230621061253Z-1f619bfeaef05cc7