Che on AWS EC2 to allow remote browser clients through SSH tunnel

[GuSuku]

I have set up Che on EC2 and can fully work on it through a firefox running locally on that host (using VNC for remote viewing). But I am not able to do the same if I access it on my laptop's browser:

The security settings for EC2 are as follows:
Type            Protocol        Port Range      Source
Custom TCP Rule     TCP         32768 - 65535       73.XXX.XXX.XX/32
SSH         TCP         22          73.XXX.XXX.XX/32

When setting up the SSH tunnel, I forward the following way: 8080/localhost/8080

When starting Che on the remote, I issue the following: ~/Applications/eclipse-che-4.2.0/bin/che.sh run --remote:localhost

From my laptop browser's localhost:8080, I can view the dashboard and stop running workspaces. I can also start workspaces, even though I see the following log: https://gist.github.com/GuSuku/8e44a3c467a3414ec9be31940cd17f83 To wit, in spite of the warnings at the end (Client has aborted connection. Response writing omitted.), the workspace does get started.

But, from my laptop browser client, when opening a running workspace in IDE, I get the following popup:

window.IDE.eventHandlers.initializationFailed@http://localhost:8080/ide/wksp-65qm?uid=612032:39:27
Wdb@_app-0.js:1782:21
ieb@_app-0.js:1787:380
_.Un@_app-0.js:4290:317
gS/<@_app-0.js:1316:276 

And the IDE is stuck at LOADING: Starting workspace agent

I am sure it has got something to do with ports, but not sure which port and what I should do.

= More info: =

• When I do docker ps, I get the following:

CONTAINER ID        IMAGE                                                        COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                                         NAMES
9ab79d815907        eclipse-che/che_workspaceu3p1r5ppkt1xb4ba_ws-machine_w5z4n   "/bin/sh -c 'sudo /us"   9 minutes ago       Up 9 minutes        0.0.0.0:32795->22/tcp, 0.0.0.0:32794->4401/tcp, 0.0.0.0:32793->4403/tcp, 0.0.0.0:32792->4411/tcp, 0.0.0.0:32791->8000/tcp, 0.0.0.0:32790->8080/tcp, 0.0.0.0:32789->9876/tcp   che_workspaceu3p1r5ppkt1xb4ba_ws-machine_w5z4n

• For the command - docker inspect --format='{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq) - I get the following: /che_workspaceu3p1r5ppkt1xb4ba_ws-machine_w5z4n - 172.17.0.2

[ddementieva]

@GuSuku --remote requires a public IP. Have you tried it?

[GuSuku]

@ddementieva With the same security settings as above, when I issue my AWS EC2 Public IP in --remote, I can no longer start my workspace even on my host's own browser - keeps saying STARTING on both host's and client's browser. Log file: https://gist.github.com/GuSuku/c1fa762b0566a80ebab93fa764c0365a

docker ps shows no running containers/images.

[james10174]

@GuSuku So you run the server on the EC2 and port forward the 8080 via ssh to your localhost/local computer. I am surprised you can start a workspace from your local web browser though since you are not forwarding ports 32768-65535 which is required for browser to connect to workspace agent. Make sure to use the EC2 servers external ip address not the internal one. Try the following

bin/che.sh -r:$(curl -s https://4.ifcfg.me/) start

[GuSuku]

@james10174 Thanks for the response.

I am surprised you can start a workspace from your local web browser though since you are not forwarding ports 32768-65535

I do forward those ports. Under the security settings I posted in my first post above, you can see it under my "custom TCP rule" for inbound TCP connections to those ports, but only from my laptop's external IP. The outbound rule on EC2 is to allow everything to everyone.

Make sure to use the EC2 servers external ip address not the internal one. Try the following bin/che.sh -r:$(curl -s https://4.ifcfg.me/) start

It was my public/external IP that I used earlier, and it matches curl -s https://4.ifcfg.me/ I am surprised that I am not able to start a workspace even from the browser on my EC2 when I specify IP explicitly this way but have no problem when I issue --remote:localhost

[james10174]

@GuSuku Are you ssh port forwarding 32768-65535? Not sure you could forward this range through ssh. I know you are forwarding 8080 but i didn't see the rest. I don't think this is your end goal though. So first break the ssh port forwarding just in case it is causing issues. Run bin/che.sh -r:$(curl -s https://4.ifcfg.me/) start. Then in your laptop web browser to navigate to :8080/dashboard/#/ with ipdadress from above $(curl -s https://4.ifcfg.me/). If you get there try to create a workspace. You may also want to create 1gb of swap since EC2 only has 1GB of ram.

[james10174]

@GuSuku OK I decided to take the EC2 dive myself. You can find my guide at https://docs.google.com/document/d/1FrnQZOBg7JxjnYsl--sGtUYcEYVJhB1rKYj5dbA33mo/pub . I just got done with it so let me know if there are any issues. The commands to run in terminal I didn't run sequentially so there might be issues with it.

[ghost]

@james10174 thank for a detailed guide. Indeed, the only thing that may block a workspace from starting is a range of blocked ports

@GuSuku --remote localhost isn't required if the browser is local to the server.

If the external IP does not work, this is a port issue. Can you attach a screenshot with the security rules you have established?

[GuSuku]

@james10174 Thanks for that detailed guide. I am reading through it. Meanwhile...

I was not SSH forwarding 32768-65535 - But only allowing incoming TCP to these ports from my local IP. And yes, only 8080 was forwarded. In the long run, I want to be able to access Che through SSH.

As you asked, I broke the SSH forward and re-started Che with my host's remote IP, say 54.84.XX.XXX (same as $(curl -s https://4.ifcfg.me/)). I also added a security rule to allow incoming TCP to 8080 from my laptop's IP.

• The same problem as before persists on my laptop's browser pointed to http://54.84.XX.XXX:8080/dashboard/#/ - Can access dashboard and stop workspace on laptop, but when starting a workspace, ran into the same issue and eventual timeout. I had also increased Che timeout to 60000 and have a swap of 2GB on host.
• Same problem persists on the host's browser (reminder: I did not face any problem on host browser with --remote:localhost flag) if I access at http://localhost:8080/dashboard/#/. Oddly, if I try accessing at http://54.84.XX.XXX:8080/dashboard/#/ on my host's own browser, not even the dashboard loads. I added a security rule to allow all traffic on all ports from 127.0.0.1/32, but didn't help.

[james10174]

@eivantsov Thanks. Feel free to share the link again. The EC2 questions seem to come up often.

@GuSuku Let me know how the guide works for you. I didn't have swap on guide but for larger projects it is a good idea to have. I have had issues of running out of memory at 1GB and 2GB seems to work better. Depends on your project sizes.

[GuSuku]

@eivantsov Here is the screenshot. Added the 2nd and 3rd incoming rule as per my latest comment above.

[james10174]

@GuSuku I think the All traffic is over ruled by the custom TCP rules. Remove All traffic. Also change source to anywhere which will be 0.0.0.0. This allows external ip address to use it. Should look like the following when you are done.

You might be able to change port 8080 from anywhere to localhost or 127.0.0.1 once you get the above working. This would be a good idea to do actually because your che server wouldn't be exposed to the internet.

[TylerJewell]

We have this docs page. If someone wants to make some recommendations to that page as suggested edits, they'll come through and then we can merge them in.

[james10174]

@GuSuku Use my guide. I got it to work. Do the following in addition to guide.

[GuSuku]

@james10174 So that was it, thanks! Like you said, I changed the source to Anywhere for 8080 and 32768-65535 and it works on my laptop browser pointed to http://54.84.XX.XXX:8080/dashboard/#/ and on my host's browser pointed to http://localhost:8080/dashboard/#/ and http://54.84.XX.XXX:8080/dashboard/#/ And apparently the overruling is the other way around, since the following works for me:

But unfortunately it is not workable as it stands - since my IDE is open to the world. I would need to restrict the incoming IP and/or need SSH tunnel.

edit: I just saw your follow up post as I was typing this - reading through it now

[james10174]

@TylerJewell I tried to do the guide but CentOS7 was not an option as an operating system. Not all of the commands worked for the Amazon Linux AMI on the docs for CentOS7.

[TylerJewell]

@GuSuku - if you want to try out Codenvy - that is a multi-user solution that has many forms of authentication for users on it. It requires more resources, but the workspaces are identical to Eclipse Che. Codenvy is free for certain usage for up to 5 users with a fair source license.

[GuSuku]

@james10174 Did what you suggested (allow all external IP to access 32767-65535 but restrict only the localhost to access 8080) and then SSH-tunneled 8080/localhost/8080 and I can work with Che on my laptop's browser pointed to http://54.84.XX.XXX:8080/dashboard/#/. Only oddity was, host's browser works only when pointed to http://localhost:8080/dashboard/#/ but not to http://54.84.XX.XXX:8080/dashboard/#/, unlike when 8080 was opened to the world, but I can live with this.

Finally, I removed the localhost rule for 8080, and everything works (except for that oddity on my host's browser):

Now, I am only nervous about allowing all external IP to access 32768-65535. Would be glad if there is a fix for this!

@TylerJewell Thanks, will take a look at Codenvy.

[james10174]

@GuSuku 8080 with source 127.0.0.1/32 makes it so only the tunnel can reach your server port 8080. Follow my guide without the changes mentioned above and you should have no problem accessing it from your browser through internet. However anyone can access it too. A much easier approach to all this is to use codenvy servers who developed Eclipse Che. Codenvy allows you to use it anywhere from most browsers without tunneling. Most public and company computers sit behind firewalls and don't have ssh clients installed to tunnel. Much much easier just to use Codenvy as Tyler mentioned.

[GuSuku]

@james10174 Seems like specifying no explicit rule for 8080 restricts that port only to localhost (which then, in my case, is tunneled to laptop's 8080), since I could not access http://54.84.XX.XXX:8080/dashboard/#/ on my laptop without SSH tunnel

On Codenvy: I am also later planning to run Che server on my at-home work desktop, that has a gpu for some number crunching, and be able to access it from any portable browser. So this is why hosting on Codenvy might not work for me.

I am generally a newbie to all this, but is it by design or a bug that Che does not work when ports 32768-65535 are restricted to incoming tcp from the client browser's IP, and instead requires those ports to be opened to incoming tcp from the world?

[TylerJewell]

@GuSuku It is by design. The issue is Docker, not so much Che. What is happening is that you are running Che and each Che workspace is a Docker container. You can run anything you want in that container - it's a runtime. If you start a server in that container on port 8080, Docker uses ephemeral port mapping to map that port to an external port. It takes over the full range of 32768-65535. It starts at the bottom and works its way up. So even though you have "EXPOSED" port 8080, the outside world accesses it in the ephemeral range.

This allows you to deploy any kind of server, or any kind of port - and there is a way for the outside world to get to it. When a client connects at 32768, Docker then knows how to map it to the internal server in the container.

[james10174]

@GuSuku Eclipse Che uses docker as Tyler just said. When opening up a port on docker it assigns a random number in that range. I mentioned it in the other post https://github.com/eclipse/che/issues/1412 use netcat for testing. nc -l 35000 does not assign to port 35000 but a random one in docker. In EC2 try the same thing out side docker and it will connect to 35000. The reason it didn't work for you in the other bug response is I failed to mention to run as root or sudo netstat. You will get a pretty good setup with your computer gpu doing the method your are talking about. I would however try to get as much working on Codenvy if you can first or make it part of your development cycle. I actually use a beaglebone(like raspberry pi) to ssh into che workspace to link both together. You should investigate to see if the same could work for you. This could allow you to not be so dependent on one computer for development. Also if you need a team later this setup may allow more flexibility.

Terminal 1
nc -l 35000
#can also do nc -l <external ipaddress> 35000

Terminal 2
netstat -tulpen | grep nc
#write down port number (ie 0.0.0.0:<port assigned>)
nc localhost <port assigned>
#can also do nc <external ipaddress> <port assigned>

Terminal 1
invalid connection to [127.0.0.1] from (UNKNOWN) [127.0.0.1] <number>
#<ctrl-c> to exit

[GuSuku]

@TylerJewell I get that port mapping part (like in the output from my case pasted below, where docker's internal 22 is mapped to external-facing 32865). But I am only confused at the part where you say "When a client connects at 32768...". This part is outside of docker, and so I am confused as to why this fails if my EC2 firewall restricts this connection only to a client with a specific IP to connect to 32768, while it only works when 32768 is opened to the world.

ubuntu@ip-172-XX-XX-XXX:~$ docker ps
CONTAINER ID        IMAGE                                                        COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                                         NAMES
7bbdb7bfdf3e        eclipse-che/che_workspaceu3p1r5ppkt1xb4ba_ws-machine_obkol   "/bin/sh -c 'sudo /us"   30 minutes ago      Up 30 minutes       0.0.0.0:32865->22/tcp, 0.0.0.0:32864->4401/tcp, 0.0.0.0:32863->4403/tcp, 0.0.0.0:32862->4411/tcp, 0.0.0.0:32861->8000/tcp, 0.0.0.0:32860->8080/tcp, 0.0.0.0:32859->9876/tcp   che_workspaceu3p1r5ppkt1xb4ba_ws-machine_obkol

@james10174 Thanks, I will take a relook at codenvy for my workflow.

I did the following from outside docker and it works as follows. The last time I reported nothing happened was because I was ignorant to not test it by sending test messages. console1: ubuntu@ip-172-XX-XX-XXX:~$ nc -l 35000 console 2:

ubuntu@ip-172-XX-XX-XXX:~$ sudo netstat -tulpen | grep nc
tcp        0      0 0.0.0.0:6001            0.0.0.0:*               LISTEN      1000       337108      10760/Xtightvnc 
tcp        0      0 0.0.0.0:35000           0.0.0.0:*               LISTEN      1000       745227      15389/nc        
tcp        0      0 127.0.0.1:5901          0.0.0.0:*               LISTEN      1000       337110      10760/Xtightvnc 
ubuntu@ip-172-XX-XX-XXX:~$ nc localhost 35000
test
test1

Back at console1:

test
test1

Are you talking about running either of these two consoles within a running docker (eg. docker exec -it <mycontainer> <nc-command>) and the other outside?

[james10174]

@GuSuku You could do that but Che workstations have built in terminals you can run it in but you would need to sudo apt-get install netcat though. This terminal is inside workspace=container. I think port 32768 is just an example port. The client must know which ports to connect which che identifies and keeps track of.

[GuSuku]

@james10174 Oh ok. I wouldn't have been able to test that way in that earlier issue, because I wasn't able to connect to the IDE back then. Now I can test it, but I get the port mapping part. But I dont get that other part.

I think port 32768 is just an example port. The client must know which ports to connect which che identifies and keeps track of

I understand. I was also using 32768 only as an example in place of 32768-65535 range. When I restricted inbound connections from just my laptop's IP, I applied that to all ports in that range.

[james10174]

@GuSuku Sorry nc doesn't work in workspace container like I thought. You can test docker out by itself though if you want to learn more. docker run -t -i ubuntu /bin/bash should let you nc and do other tests. docker is really great once you get used to it. Che is the only IDE I know of that uses it really well. Like docker, code that runs on Che in a docker container should be able to run anywhere docker can be installed. Pretty exciting when docker beta for windows and mac is being tested.

[GuSuku]

@james10174 Yes it is exciting! I am not a lot into developing for multiple environments as I am into running some numbers in fixed environment. So, I would be pretty content with Che's browser-based IDE, without dockers. I looked into Eclipse Orion, but it seemed like features like code completion, rename refactoring, error highlighting etc are not available for the languages that I use (python, java, c/c++). I would be glad to be wrong though!

At my EC2 firewall, I restricted ports 32768-65535 to just my laptop's ID, and tried what @eivantsov suggested in the other thread, and clearly the workspace container's ephemeral ports are accessible from my laptop, even while that workspace doesn't work from within Che on my browser: On remote host:

ubuntu@ip-172-XX-XX-XXX:~$ docker ps
CONTAINER ID        IMAGE                                                        COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                                         NAMES
475eea9b7ea9        eclipse-che/che_workspaceu3p1r5ppkt1xb4ba_ws-machine_w777g   "/bin/sh -c 'sudo /us"   46 minutes ago      Up 46 minutes       0.0.0.0:32872->22/tcp, 0.0.0.0:32871->4401/tcp, 0.0.0.0:32870->4403/tcp, 0.0.0.0:32869->4411/tcp, 0.0.0.0:32868->8000/tcp, 0.0.0.0:32867->8080/tcp, 0.0.0.0:32866->9876/tcp   che_workspaceu3p1r5ppkt1xb4ba_ws-machine_w777g

On local client:

xxxx@vagrant-ubuntu-trusty-64:$ curl -v http://54.XX.XX.XXX:32871/api/ext/
* Hostname was NOT found in DNS cache
*   Trying 54.XX.XX.XXX...
* Connected to 54.XX.XX.XXX (54.XX.XX.XXX) port 32871 (#0)
> GET /api/ext/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 54.XX.XX.XXX:32871
> Accept: */*
> 
< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< Set-Cookie: JSESSIONID=00A98680D25BA855BDC2C60834534862; Path=/ide/; HttpOnly
< Cache-Control: public, no-cache, no-store, no-transform
< Content-Type: application/json
< Transfer-Encoding: chunked
< Vary: Accept-Encoding
< Date: Mon, 06 Jun 2016 22:24:27 GMT
< 
{"rootResources":[{"regex":"/jdt/([^/]+?)/compiler-settings(/.*)?"..........

Is this still not a bug?

[james10174]

@GuSuku Alright finally figured it out :). You need to allow the server access to it too. I think it needs to access port 4401 and 4403 which are assigned a port randomly in that range too.

[GuSuku]

@james10174 Unfortunately that did not work for me. It was slightly better, in the sense I could start a workspace on my laptop's browser, but I still could not open its IDE.

[ghost]

@GuSuku the problem is that making these ports accessible for your laptop only isn't enough.

When you run Che server, it starts a workspace and then tries to reach it. It uses an external IP that you have specified.

So, opening these ports to the world is the only option here. You need to make sure that http://54.84.XX.XXX:${port-range} can be accessed from within the VM. It is both server AND browser that ping ws-agent. SSH tunnel works for the browser, but not server.

Why don't you set up a security rule for this port range for 0.0.0.0/0 and give it a try this way?

All suggestions and instructions that @james10174 shared here are correct.

[james10174]

@GuSuku Start over with new EC2 with my guide with modified security group above. Everything works for me and IDE comes up. Nice thing about the cloud is there are no environmental variables.

[GuSuku]

@eivantsov I did try by opening that port range to the world, and everything works. But doing that on the remote host just to allow the local client browser with a known specific IP (along the remote host itself) is a bit odd. But then I am new to fiddling with networking.

@james10174 Oh ok, will try that!

[james10174]

@GuSuku In the security group, "EC2" in the picture is the external ip(internal ip does not work) of EC2 server and "laptop" is your external IP of your laptop.

--edit make sure you add swap. It creates weird issues when trying to compile larger programs. Refer to this link method 2 on how to do that http://www.thegeekstuff.com/2010/08/how-to-add-swap-space/?utm_source=feedburner

[ghost]

@GuSuku well, this is how Che networking works. Server and client use the same IP to reach a workspace agent that is running in a docker container. If server can use a local (internal IP), the client cannot.

[ghost]

@GuSuku have you finally managed to run Che? Did port opening help?

[jlewi]

I solved a similar problem on GCE. However, I setup dynamic port forwarding with my ssh connection. I then configured chrome to use a SOCKs proxy. So far this is working great. I jotted down some notes here.

[TylerJewell]

@jlewi - this is great!

@ddementieva - we should add a matching google cloud page that is similar to the EC2 page to the Eclipse Che docs. https://eclipse-che.readme.io/docs/configuration-che-on-ec2. With the permission of @jlewi - we could use his document as the template for it.

[jlewi]

@TylerJewell By all means feel free to copy and modify my notes.

[oldirtybasti]

Apart from Amazon AWS, I want to use Eclipse Che on my home server and develop from my notebook via SSH tunnelling. I can only open Port 22 an 80 to the outside world. Is there a way how that can be accomplished? Is there a tutorial available anywhere?

The tips in this thread only helped, if I forwarded all the necessary ports from my home router to the server.

[TylerJewell]

@oldirtybasti - you should consider running Che with Bitnami. They provide a reverse-proxy engine that sits in front of Che that provides some basic authentication and routes traffic through a smaller number of ports.

If you want to run your own server behind only a limited number of ports, this will require one of two scenarios. 1: If you already know your workspaces, launch che on your server, launch those workspaces, and then install an SSH server listening on a particular port in the workspace. Most of our workspaces already have SSHd running inside of them. You can then click the SSH icon in the IDE to discover the port that SSH is running on . You could then set up an SSH tunnel from the outside world to that internal port, through a tunnerler over your allowed port 22 / 80. You'd need to setup a tunneler somewhere inside of your network.

2: You would need to deploy a reverse-proxy that routed all traffic from the outside world to the right location internally. I don't think this would be possible without writing some internal extensions to Che that make it possible to handle some of this routing logic.

I think you want the first scenario.

[oldirtybasti]

Tyler, thank you very much for the information. Unfortunately I have tons of data and need a very strong machine, therefore I want to keep the server in-house.

About solution 1: What about all the others ports that get mapped randomised? I do have to communicate over them too, if I want to use the IDE to develop, stop and start workspaces etc. I read about ssh that one can tunnel more than 1 port within an ssh connection. Would it be viable to tunnel all of the 32k Ports at once? Is that even possible?

[TylerJewell]

@oldirtybasti - yes, that is correct - all of the other ports that are exposed by the workspace would not be routed through that SSH tunnel.

I am not really sure if SSH tunneling all ports is possible or not. I don't think our team has a lot of experience with SSH tunnels.

There is this proposal of a pull request that will allow a reverse proxy to send certain ports to a different system. You could potentially modify this to have the reverse proxy those ports to another proxy that consolidates them down to a single more secure port.

But this sort of tunneling all ports through a single port or SSH tunnel is not something we have tested or thought much about. We have all traffic on a Codenvy system running through its singular router interface to limit the flow of traffic and to apply man-in-the-middle permissions. But we haven't done anything about routing all ports through a single port. It would have an impact on performance, but it would add security.

https://github.com/eclipse/che/pull/2004

[samssann]

I see #2004 is moving forward. Is it likely that the feature you mentioned in your last response will be available once merged?

[TylerJewell]

@samssann - I am sorry, I don't know how I missed this question previously. Usually our maintainers are pretty good about staying on top of discussions. #2004 is a step in the direction but it's not the complete solution. The fuller solution will take through 2017 and it will involve building in a reverse proxy into the server so that requests tunneling in on a single port can be redirected to back-end workspace servers on any port.

2004 is an improvement that allows browsers to directly communicate to workspaces over hostnames, not just IP addresses.

[samssann]

No problem! Thank you for your answer. Is there a PR or an issue to follow on the development of the 'fuller solution'?

[TylerJewell]

No - not yet. Committers are still debating in chat forums the approach and priority of it all.

[vinokurig]

Is this still an issue? If so please let us know otherwise in 1 week we will close this due to inactivity.