Provide ability for U-ADS to use HTTPD with Jupyterhub

[galenatjpl]

@mcduffie , I think what is meant by this ticket, is that JuptyerHub can be:

1. Pointed to by httpd configuration
2. this can be setup at JuptyerHub deploy-time (via the management console / marketplace)
3. httpd is also integrated with cognito to control authentication

Did I get this correct?

If so, I think the estimate will probably be more like a 5. I will assign this ticket to @ramesh-maddegoda as well.

[mike-gangl]

i think we simply want a pass through to the jupyter without authentication- jupyter does its own login/auth as a part of its workflow, it only needs to be routed through httpd.

[GodwinShen]

@ramesh-maddegoda any updates on this one? it's been a couple of weeks since someone last posted a comment/update

[ramesh-maddegoda]

@ramesh-maddegoda any updates on this one? it's been a couple of weeks since someone last posted a comment/update

@GodwinShen, We have deployed an httpd in MCP Dev as a Shared Services HTTPD server. However, it only works with http URLs and https URLs are failing. I am trying to fix this.

[GodwinShen]

@ramesh-maddegoda did you manage to fix the https URLs failing? This ticket is blocking a couple of U-ADS items at the moment.

[ramesh-maddegoda]

@GodwinShen (CC: @galenatjpl ), I was able to fix the issues with proxing https in Shared Services account.

I will share the same comment I posted in a Slack thread below:

I think the httpd proxy was assumed as an already existing service even before implementing and deploying it on shared service account. We had it in design "diagrams", but in fact it was deployed in Unity Dev (shared services) last week. Currently I am troubleshooting some issues related with it, when I try to write URL rewriting rules with it.

Proxying a website behind httpd is as simple as follows:
ProxyPass "/"  "http://www.example.com/"
ProxyPassReverse "/"  "http://www.example.com/"

There is a user guide on that: https://httpd.apache.org/docs/2.4/howto/reverse_proxy.html

However, when we proxy a website, in many cases usually it will have many missing links due to absolute paths used etc. So we have to use URL rewriting. URL rewriting needs some understanding of the syntax and standards. https://httpd.apache.org/docs/2.4/rewrite/intro.html

While the developers are waiting to use Unity Dev httpd server, they can try this locally using httpd docker image located at https://hub.docker.com/_/httpd

Then, they can use the local httpd to write the proxy rules and URL rewrite rules (because the URL re-writing rules can be application specific). When they are ready to deploy the app, we can slightly modify reuse the same rules in Unity Dev httpd.

[mike-gangl]

i'm seeing the https failures as well. in the unity-venue-dev httpd proxy, i added a route to the unity-venue-dev jupyter hub as follows:

RewriteEngine on
ProxyPass /jupyter https://jupyter-venue-dev-alb-****.us-west-2.elb.amazonaws.com:8000/
ProxyPassReverse /jupyter https://jupyter-venue-dev-alb-****.us-west-2.elb.amazonaws.com:8000/

and am seeing the following issue in the httpd logs:

No protocol handler was valid for the URL /jupyter/hub (scheme 'https'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

[mike-gangl]

might need some more modules to enable proxying to SSL sites

mod_ssl ?

[mcduffie]

I still need the ability to modify the HTTPD configuration during Jupyterhub deployment. I would need information on how to do that.

[ramesh-maddegoda]

@mike-gangl , I was able to proxy JupyterHub on Shared Services (Unity Dev) account with the following configs (but we still have to change JupyterHub redirect URL):

SSLProxyEngine On
SSLProxyCheckPeerCN on
SSLProxyCheckPeerExpire on

ProxyPass /hub https://jupyter-venue-dev-alb-**********.us-west-2.elb.amazonaws.com:8000/hub/
ProxyPassReverse /hub https://jupyter-venue-dev-alb-**********.us-west-2.elb.amazonaws.com:8000/hub/

I enabled following modules.

 proxy_module
 proxy_http_module
 rewrite_module
 ssl_module

The full list of modules enabled (including default modules):

 core_module (static)
 so_module (static)
 watchdog_module (static)
 http_module (static)
 log_config_module (static)
 logio_module (static)
 version_module (static)
 unixd_module (static)
 access_compat_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 auth_openidc_module (shared)
 authn_core_module (shared)
 authn_file_module (shared)
 authz_core_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 filter_module (shared)
 headers_module (shared)
 mime_module (shared)
 mpm_event_module (shared)
 negotiation_module (shared)
 proxy_module (shared)
 proxy_http_module (shared)
 proxy_wstunnel_module (shared)
 reqtimeout_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 socache_shmcb_module (shared)
 ssl_module (shared)
 status_module (shared)

[ramesh-maddegoda]

@mike-gangl , I got the same error earlier with httpd docker (No protocol handler was valid for the URL /jupyter/hub (scheme 'https'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule).

Now I am using Apache2 (New Ubuntu version of httpd) in an EC2 instance in Unity Dev account and the error is not there. (We decided to use an Apache2(httpd) server hosted on EC2 in Shared Services account for rapid configuration changes and quick config reloads for troubleshooting).

[galenatjpl]

@jpl-btlunsfo just to sync up everyone here, are you also using the Apache2 (New Ubuntu version of httpd) version? I think you are basically leveraging (in https://github.com/unity-sds/unity-cs/issues/403) the solution that was based on Docker in the unity-proxy repo, but I'm just mentioning the comment by @ramesh-maddegoda above, in case that is critical to make your effort a success as well. Perhaps it's not relevant, but perhaps it is.

[jpl-btlunsfo]

@galenatjpl , @ramesh-maddegoda I didn't change the base-image for unity-proxy's dockerfile, it's stil using FROM ubuntu/apache2

I believe the container is also specifically enabling the following additional modules (I didn't change anything here either):

auth_openidc proxy proxy_http proxy_wstunnel rewrite headers

The only one I think might need adding is the ssl_module, or mod_ssl, but that's only if it's not preincluded in this base image. (And perhaps mod_proxy_html, as I mentioned in a previous meeting)

[galenatjpl]

Let's check back in the week of June 17, with @mike-gangl to see where we are at with the progress of websockets, etc..

[galenatjpl]

reassigning to @mike-gangl per comment above, and team's understanding.