connecting to jupyter

[jcwright77]

I'm using the dev jupyter app. The launch works fine and the process starts on a computer node. The 'connect' button opens:

https://ood.host.edu/node//33203/login where 33203 the the port jupyter is running on on the computer node. But this page does not open anything, I don't see any logic about forwarding the port from the compute node to the ood front end. How is this supposed to work? -john

┆Issue is synchronized with this Asana task by Unito

[johrstrom]

I don't see any logic about forwarding the port from the compute node to the ood front end. How is this supposed to work?

It's the library in /opt/ood/mod_ood_proxy - these files https://github.com/OSC/ondemand/tree/master/mod_ood_proxy

What's the response you are seeing? I mean, who is responding apache or the origin Jupyter? In your web browsers dev tools you should be able to see the response headers. Usually servers set a Server header that'll indicate who it's from. apache's clear, but Juypter will say something about Tornado.

[jcwright77]

Took at step back and am trying to verify reverse proxy which apparently is need for this to work.

In ood_portal.yml, I see host_regex to a single host: host_regex: 'gpu-04.my.domain.edu'

restart apache2

follow documentation to ssh to gpu-04 and run nc -l high_port_no

check hostname -f matched host_regex

after nc -l 8899

try

https://ood.domain.edu/node/gpu-04.domain.edu/8899

no response. Try without

https://ood.domain.edu/node/gpu-04/8899

also doesn't work. apache2 access shows 404 error.

-john

On Thu, July 21, 2022 12:48 pm, Jeff Ohrstrom wrote:

I don't see any logic about forwarding the port from the compute node to the ood front end. How is this supposed to work?

It's the library in /opt/ood/mod_ood_proxy - these files https://github.com/OSC/ondemand/tree/master/mod_ood_proxy

What's the response you are seeing? I mean, who is responding apache or the origin Jupyter? In your web browsers dev tools you should be able to see the response headers. Usually servers set a Server header that'll indicate who it's from. apache's clear, but Juypter will say something about Tornado.

-- Reply to this email directly or view it on GitHub: https://github.com/OSC/ondemand/issues/2152#issuecomment-1191716979 You are receiving this because you authored the thread.

Message ID: @.***>

-- Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright

[johrstrom]

You may also need to enable the proxy altogether by setting rnode_uri and node_uri.

host_regex: '\w+\.my\.domain\.edu'
rnode_uri: '/rnode'
node_uri: '/node'

You can check the resulting ood-portal.conf in /etc/apache2/sites-enabled/ood-portal.conf to see if there's indeed a /node Location.

[jcwright77]

Had those, the issue was that sudo /opt/ood/ood-portal-generator/sbin/update_ood_portal

doesn't actually replace /etc/apache2/sites-available/ood-portal.conf but replaces /etc/apache2/sites-available/ood-portal.conf.new So changes weren't taking effect. I had forgotten that.

After merging changes reverse proxy works, so lets see if jupyter reverse proxy works too. -john

On Thu, July 21, 2022 1:37 pm, Jeff Ohrstrom wrote:

You may also need to enable the proxy altogether by setting rnode_uri and node_uri.

host_regex: '\w+\.my\.domain\.edu'
rnode_uri: '/rnode'
node_uri: '/node'

You can check the resulting ood-portal.conf in /etc/apache2/sites-enabled/ood-portal.conf to see if there's indeed a /node Location.

-- Reply to this email directly or view it on GitHub: https://github.com/OSC/ondemand/issues/2152#issuecomment-1191759938 You are receiving this because you authored the thread.

Message ID: @.***>

-- Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright

[johrstrom]

Yea we have some logic around replacements if you edit /etc/apache2/sites-available/ood-portal.conf by hand directly. It seems to think you've done that. You can supply the -f option to force it. Also note that bouncing apache runs the same command.

[jcwright77]

Ok last problem seems to be the compute host not being set in the jupyter configuration.

I've verified if I edit the host in config.py and manually start jupyter on the compute host, the redirection from the ood host works.

Despite setting host in both cluster.yml and jupyter/submit.yml.erb to:

batch_connect: basic: script_wrapper: | %s set_host: "host=$(hostname -f)"

The value after /node/ in both the 'connect' url and the jupyter config.py setting remain empyty.

any ideas?

-john

On Thu, July 21, 2022 1:53 pm, Jeff Ohrstrom wrote:

Yea we have some logic around replacements if you edit /etc/apache2/sites-available/ood-portal.conf by hand directly. It seems to think you've done that. You can supply the -f option to force it. Also note that bouncing apache runs the same command.

-- Reply to this email directly or view it on GitHub: https://github.com/OSC/ondemand/issues/2152#issuecomment-1191773594 You are receiving this because you authored the thread.

Message ID: @.***>

-- Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright

[johrstrom]

Yea cluster.d updates require a restart to be re-read. You should be able to tell what's being written in job_script_content.sh in the session directory (that link on the card).

Restart your PUN through the help or develop menu. No need to bounce apache or do anything with privilege, this just bounces your nginx.

[jcwright77]

Thanks for your help. got it working. Setting host properly for our system with 'hostname -f' in submit_yml.erb was the last step. (We don't have local hostnames so hostname -A does not return a second hostname for awk to grab.

-john

On Thu, July 21, 2022 3:03 pm, Jeff Ohrstrom wrote:

Yea cluster.d updates require a restart to be re-read. You should be able to tell what's being written in job_script_content.sh in the session directory (that link on the card).

Restart your PUN through the help or develop menu. No need to bounce apache or do anything with privilege, this just bounces your nginx.

-- Reply to this email directly or view it on GitHub: https://github.com/OSC/ondemand/issues/2152#issuecomment-1191833619 You are receiving this because you authored the thread.

Message ID: @.***>

-- Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright

[johrstrom]

Glad to hear it. I'm closing this now, just reach out again if you have issues.

[jcwright77]

Ok so reverse proxy is working but jupyter redirect is not. The launch button opens:

https://ood.domain.edu/node//41115/login That is it does not insert the FDQN of the compute node. If I put that in by hand, it still doesn't work. Both return a 404 -john

On Thu, July 21, 2022 1:53 pm, Jeff Ohrstrom wrote:

Yea we have some logic around replacements if you edit /etc/apache2/sites-available/ood-portal.conf by hand directly. It seems to think you've done that. You can supply the -f option to force it. Also note that bouncing apache runs the same command.

-- Reply to this email directly or view it on GitHub: https://github.com/OSC/ondemand/issues/2152#issuecomment-1191773594 You are receiving this because you authored the thread.

Message ID: @.***>

-- Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright

[johrstrom]

Check out the hostname in the connection.yml file in the working directory of the job. This file holds the information we need to build the URL (that's currently missing the hostname). set_host (from earlier in this ticket) helps populate this file with the hostname.

There's a link to the jobs working directory in the job card that you connect from. It's the session Id link.