Support for interactive `deis run bash` session

[deis-admin]

From @gabrtv on August 20, 2013 21:11

While #70 provided us with the ability to run one-off admin commands, we really need an interactive shell into the ephemeral docker containers.

This is quite tricky, as it requires a full duplex channel (HTTP Chunked Encoding? WebSockets?) as well as a select loop on the API server to handle routing stdin, stdout and stderr.

https://github.com/paramiko/paramiko/blob/master/demos/interactive.py#L39 provides a good example of a terminal-enabled select loop.

Copied from original issue: deis/deis#117

[deis-admin]

From @tombh on January 16, 2014 16:2

Reminded of this from #465. Just to add another example alongside the paramiko one; I've implemented socket communication from the CLI client in Danabox using Python's Twisted framework. I called the socket server 'rendevous' after Heroku's implementation, though it's nowhere near as functional. All it does at the moment is stream the output from a server-side deploy command and receive heartbeats from active docker containers.

The main thing I'd like to improve is authentication. Ideally I think you'd want to reuse the cookie from the HTTP session, I haven't looked into that, so I'm not even sure if it's possible.

[deis-admin]

From @mboersma on January 16, 2014 16:8

Cool! We had done websockets / socket.io with Twisted in a previous product at OpDemand, and I've done plenty of Twisted in the past, but I think we want to avoid that this time around because the code is more difficult to read (and write), and a priority for Deis is keeping things approachable for developers. Although your socket server here is nice and clean...hmm.

There's a websockets package using django, gevent, and redis, all of which are already in our toolbox. (Redis will be very soon, see the containerize branch.) I'm going to give that a go today and see how it works, but I'm open to other approaches. I want the implementation to work for deis run bash as well as deis logs -f.

[deis-admin]

From @tombh on January 16, 2014 16:18

Yeah Twisted code is very hard to read :/ If you can come up with a simpler way I'll definitely use it Danabox as well :)

[deis-admin]

From @bacongobbler on February 4, 2014 20:16

Why not leverage docker-py for the CLI? It already supports attaching to containers and log streaming, so it seems like the problem has already been solved for us. The only issue I see is that we have to figure out how an API call to the controller would let us grab this information to run and attach to the container.

[deis-admin]

From @andyshinn on May 29, 2014 3:54

I wonder if something like https://github.com/progrium/termshare would work for this? You can run server in a container and proxy it through the router. Then you could run this interactively on a instance, get the URL ID, and either present the user with a URL for the browser or wrap up the termshare client somehow.

[deis-admin]

From @bacongobbler on August 25, 2014 4:44

Update on this bug: I'm currently researching different ways on how we can tackle this. The problem lies with the fact that Django was never initially designed for long-running requests such as websockets. However, there are Django plugins such as https://github.com/stephenmcd/django-socketio which help solve this problem. Need to do a bit more deep-sea diving as I haven't delved much into the area of websockets before :)

Also to note that there's a very informative article on using Django with socketio and gevent. I'll experiment with this as well.

[deis-admin]

From @evadne on December 7, 2014 14:56

Is it possible to just use SSH?

1. Query the latest version of the application thru etcdctl / Deis Controller
2. Query location of the private registry
3. Assemble image name
4. $ fleetctl ssh machine_id docker run -ti $IMAGE_NAME /bin/bash -c "cd /app && $COMMAND"

[deis-admin]

From @bacongobbler on December 7, 2014 21:47

@evadne it's possible but we wouldn't want users to be able to access the cluster through fleet. If it's in a public PaaS setting (such as someone OEM'ing Deis or something), it's a security concern to give users host-level access.

[deis-admin]

From @bacongobbler on April 11, 2015 18:5

for the meantime, this has been the workaround for most users: https://github.com/deis/deis/issues/3449#issuecomment-91845128

[deis-admin]

From @jwaldrip on April 22, 2015 3:56

@bacongobbler I will say that the workaround I have been doing is to just use docker exec to get into the running container. Seems to work fine for me (as much as it pains me to do it)

[deis-admin]

From @fbjork on October 21, 2015 18:32

Any update on this? This is as pretty big limitation of Deis compared to Heroku.

[deis-admin]

From @gabrtv on October 21, 2015 19:15

@fbjork I would have liked to ship this earlier, but as we've discussed in the open planning meetings, this feature is on hold pending Deis v2. Reason being the plumbing we need for this already exists in kubectl exec, versus having to write and maintain a bi-directional channel from scratch -- only to throw it away for v2.

We expect to have an RC for Deis v2 available by the end of the year.

[deis-admin]

From @shidenkai0 on November 17, 2015 11:2

Hi everyone, I wrote a python script that opens a rails console directly from your deis app folder, using TLS to authenticate to the docker remote API on your cluster nodes, to use it, you have to enable Docker remote API with TLS, see: https://coreos.com/os/docs/latest/customizing-docker.html I personally put it in my PATH directory (/usr/local/bin). You may find it in my repo: https://github.com/hajimenomomo/deis_rails_console.git Note that you may use the framework of this script to execute any kind of interactive command in a deis cluster. I hope this helps.

[deis-admin]

From @rstacruz on January 7, 2016 11:56

The workaround is to SSH into one of your CoreOS nodes:

# try the one in DEISCTL_TUNNEL
ssh core@1.2.3.4

then check out the docker containers running:

docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS              PORTS                                                                                      NAMES
bc11d8da5e52        1.2.3.4:5000/staging:v10              "/runner/init start w"   2 minutes ago       Up 2 minutes        0.0.0.0:32869->5000/tcp                                                                    staging_v10.web.1

Then use that container ID for docker exec:

docker exec -i -t bc11d8da5e52 bash

[deis-admin]

From @nathansamson on January 7, 2016 12:0

If you have a more recent deis (v1.11+ for sure) version you can just do

deisctl dock your_app /runner/init yourcmd"

you can even pick a specific container with deisctl dock your_app_v10.web.1 /runner/init yourcmd"

I am not sure if this is intended to work (it originally was supposed to be for deis components itself) but I find it very useful for application containers as well

[deis-admin]

From @rstacruz on January 7, 2016 12:9

Can confirm... that works brilliantly. Be sure to have your deis key in ssh-add and be sure you have DEISCTL_TUNNEL set.

[deis-admin]

From @krancour on January 7, 2016 15:32

The one potential problem with that workaround, however, is that deisctl is an admin tool whilst deis is in many cases a developer tool. In an org with many developers and good separation of roles, the sysadmins / devops guys or gals likely would not have widely distributed the key that would allow a developer to use deisctl.

[deis-admin]

From @rstacruz on January 7, 2016 15:36

however, is that deisctl is an admin tool whilst deis is in many cases a developer tool

sadly so! hope a proper implementation of interactive deis run would land soon exactly because of this.

[deis-admin]

From @nathansamson on January 7, 2016 15:56

@krancour While thats certainly true, ssh-ing into the server has the exact same problem (only the admins should be able to do so).

So while we all are eagerly waiting for deis run to work interactively natively, deisctl is a very handy trick to work with...

[deis-admin]

From @bacongobbler on February 24, 2016 23:9

FYI this won't be immmediately implemented in Deis v2 as we are targeting for v1 compatibility while rebasing on top of kubernetes, so the workaround in v2.0.0 will be to run kubectl exec --namespace=myapp -it <pod name> bash. This will achieve the same behavior as running docker exec -it <my container> bash in v1.

[deis-admin]

From @sheerun on January 18, 2017 20:1

My ugly workaround:

export NAMESPACE=xxx PROCESS=web && kubectl exec -it --namespace $NAMESPACE "$(kubectl get pods --namespace $NAMESPACE -o name | grep $NAMESPACE-$PROCESS | cut -d '/' -f 2)" -- bash -c '. /app/.profile.d/ruby.sh && PATH=/app/.heroku/node/bin:$PATH /app/bin/rails c'

It's a shame deis doesn't support interactive output..

[deis-admin]

From @bacongobbler on January 18, 2017 20:6

@sheerun you can simplify that bash incantation with -- /runner/init rails c if you're using Heroku buildpacks.

[deis-admin]

From @bacongobbler on January 18, 2017 20:7

also see https://github.com/deis/controller/pull/1162.

[deis-admin]

From @sheerun on January 18, 2017 21:4

@bacongobbler thanks! here's my new deis-run command:

deis-run() { export NAMESPACE="$1" && PROCESS="$2" && shift && shift && kubectl exec -it --namespace $NAMESPACE "$(kubectl get pods --namespace $NAMESPACE -o name | grep $NAMESPACE-$PROCESS | cut -d '/' -f 2)" -- /runner/init "$@"; }

Usage:

# deis-run <application> <process> <command>
deis-run my-app web rails c

[scottrobertson]

It's messy, but i modified this a bit so that you can run it without specifying the namespace when you are in your apps git repo. This will need to be changed to allow a namespace to be passed if you are not in a repo.

function deis-run() {
  export NAMESPACE=$(deis info | grep "id:" | tail -1 | cut -d ':' -f 2 | xargs)
  export PROCESS="$1" && shift
  POD=$(kubectl --namespace=$NAMESPACE get pods -o name | grep -m 1 $PROCESS | cut -d '/' -f 2)
  kubectl exec --namespace=$NAMESPACE -it $POD /runner/init "$@"
}

deis-run web rails c

[phuongnd08]

I take ideas of these bash and package to a gem: https://github.com/phuongnd08/deis-interactive Hopefully it will be useful. Pull requests are welcome.