Session can't create. "shell command failed: fork/exec : no such file or directory"

[newjones]

Hi, I was trying to test Teleport on a host with public IP. I run all teleport services binded to all interfaces and when I try by web launching the shell doesn't work and the logs throw following message:

ERRO[3616] shell command failed: fork/exec : no such file or directory  component=node fields=map[remote:<IPADDRESS>:33094 login:<username>teleportUser:<username>id:9 local:127.0.0.1:3022] file=srv/sess.go:497 func=srv.(*session).start
ERRO[3616] fork/exec : no such file or directory         file=srv/sshserver.go:849 func=srv.(*Server).dispatch
fork/exec : no such file or directoryERRO[3616] ssh: could not start shell                    file=web/terminal.go:159 func=web.(*terminalHandler).Run.func1

Any idea?

[russjones]

Hi @newjones, two questions:

1. Does user you are running Teleport as have the ability to create a shell for another user? For example, are you running Teleport as root or are you running Teleport under your normal account and trying to create a shell for some other user?
2. Are you running Teleport within a container?

[newjones]

Hi @russjones, I run teleport with root. Then I create an account and this user session appears on web after login by two factor (google). The problem is when I try to launch the web console when appears the errors. Not, on baremetal.

[kontsevoy]

@newjones looks like the login you're using for the web session does as a shell set to something which can't be found. If you SSH into that box (as the same user you're using in Web UI), and execute these commands:

$ ps -Fp $$
$ env

Can you please send the output for both?

[newjones]

Hi @kontsevoy :

$ ps -Fp $$

UID PID PPID C SZ RSS PSR STIME TTY TIME CMD

27001 26999 0 6567 4524 3 19:20 pts/1 00:00:00 -sh #$ env LC_PAPER=es_ES.UTF-8 XDG_SESSION_ID=17042 LC_ADDRESS=es_ES.UTF-8 LC_MONETARY=es_ES.UTF-8 TERM=xterm-256color SHELL=/bin/sh SSH_CLIENT= LC_NUMERIC=es_ES.UTF-8 SSH_TTY=/dev/pts/1 LC_ALL=es_ES.UTF-8 USER= LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36: LC_TELEPHONE=es_ES.UTF-8 MAIL=/var/mail/ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games LC_IDENTIFICATION=es_ES.UTF-8 PWD=/home/ LANG=es_ES.UTF-8 NODE_PATH=/usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript LC_MEASUREMENT=es_ES.UTF-8 SHLVL=1 HOME=/home/ ## the same LOGNAME= ##the same as web SSH_CONNECTION= LESSOPEN=| /usr/bin/lesspipe %s XDG_RUNTIME_DIR=/run/user/3002 LESSCLOSE=/usr/bin/lesspipe %s %s LC_TIME=es_ES.UTF-8 LC_NAME=es_ES.UTF-8 _=/usr/bin/env

[kontsevoy]

@newjones ok, that didn't help, but I have a better idea now. I have prepared a special build of teleport and uploaded a 64-bit Linux binary for you to try. Do you mind replacing teleport executable with it and producing this error? It will output more debugging info:

$ curl http://s3.gravitational.io/ev/teleport.gz | gunzip > teleport
$ sha1sum teleport 
b8493c6d1205505725c58d08a63bfddb934d7d38  teleport
$ chmod +x teleport

It will dump something that starts with ----->>>> &exec.Cmd into the log, would you please try and post it back here? Thanks.

[newjones]

Hi @kontsevoy

ERRO[0111] ----->>>> &exec.Cmd{Path:"", Args:[]string{"-."}, Env:[]string{"LANG=en_US.UTF-8", "PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games", "HOME=/home/", "USER=", "SHELL=", "SSH_TELEPORT_USER=", "SSH_SESSION_WEBPROXY_ADDR=host.domain.com:3080", "SSH_TELEPORT_HOST_UUID=0ffb1f8d-90e9-47b1-a004-64dbc1f8c2dd", "SSH_TELEPORT_CLUSTER_NAME=0ffb1f8d-90e9-47b1-a004-64dbc1f8c2dd", "LANG=es_ES.UTF-8", "TELEPORT_SESSION=32349085-4f54-11e7-931d-002590d688ba", "TERM=xterm", "SSH_CLIENT= 58830 3022", "SSH_CONNECTION= 58830 127.0.0.1 3022", "SSH_TTY=/dev/pts/11", "SSH_SESSION_ID=32349085-4f54-11e7-931d-002590d688ba"}, Dir:"/home/", Stdin:io.Reader(nil), Stdout:io.Writer(nil), Stderr:io.Writer(nil), ExtraFiles:[]os.File(nil), SysProcAttr:(syscall.SysProcAttr)(0xc4203ec3f0), Process:(os.Process)(nil), ProcessState:(os.ProcessState)(nil), ctx:context.Context(nil), lookPathErr:error(nil), finished:false, childFiles:[]os.File(nil), closeAfterStart:[]io.Closer(nil), closeAfterWait:[]io.Closer(nil), goroutine:[]func() error(nil), errch:(chan error)(nil), waitDone:(chan struct {})(nil)} component=node fields=map[local:127.0.0.1:3022 remote::58830 login: teleportUser: id:3] file=srv/sess.go:535 func=srv.(session).start ERRO[0111] shell command failed: fork/exec : no such file or directory component=node fields=map[teleportUser: id:3 local:127.0.0.1:3022 remote: login:] file=srv/sess.go:537 func=srv.(session).start ERRO[0111] fork/exec : no such file or directory file=srv/sshserver.go:901 func=srv.(Server).dispatch fork/exec : no such file or directoryERRO[0111] ssh: could not start shell file=web/terminal.go:159 func=web.(terminalHandler).Run.func1 WARN[0112] open /var/lib/teleport/log/sessions/default/32349085-4f54-11e7-931d-002590d688ba.session.log: no such file or directory file=events/auditlog.go:361 func=events.(AuditLog).GetSessionEvents

[kontsevoy]

so, based on &exec.Cmd{Path:"", Args:[]string{"-."} the shell seems to be set to an empty string for the OS user you're logging in as (you can probably look in your /etc/passwd to confirm). It is unclear how is this possible... are you doing something funky like SSH into a container or something? we should probably respond with "shell is not set for user xxxx" or something.

[newjones]

Hi @kontsevoy yeahhh!! We use salt to deploy the skeleton of OS and seems the receipe generates on /etc/passwd empty last field (shell) although we never got problems on local and remote login shell sessions. Many thanks!!

[kontsevoy]

hehe, software is hard, glad you were able to sort this out.