Deploying a open on demand demo with ondemand-dex authentication

[AMK13]

We are trying to setup a demo of open on demand with ondemand-dex. We installed all the packages but however we are not sure how to setup the configs. Can you please advise us so we can proceed further.

System details:

- AlmaLinux 8 OS8 Gen2
- Size: Standard_D4s_v3, 16Gib mem, spot
- 30GiB standard SSD(locally redundant)

We logged into the VM and do the following as per documentation.

sudo dnf config-manager --set-enabled powertools sudo dnf install epel-release sudo dnf module enable ruby:3.1 nodejs:18

sudo dnf install https://yum.osc.edu/ondemand/3.1/ondemand-release-web-3.1-1.el8.noarch.rpm sudo dnf install ondemand

sudo systemctl start httpd sudo systemctl enable httpd

yum install ondemand-dex systemctl restart ondemand-dex systemctl enable ondemand-dex.service systemctl start ondemand-dex.service

We are not sure what to change in the config file to proceed further.

Can you please advise on how to set the config files and how to proceed further. At this stage we are only interested in a demo therefor we only need the simplest authentication.

Look forward to hear from you.

Muneeb/Roberto RedOak Consulting

[johrstrom]

As an off the top guess, I'd guess that /etc/ood/dex doesn't have the right permissions to create files like this DB file. Looking at the spec file - I don't see how it changes this directories permissions.

That's the first thing I'd check - if /etc/ood/dex has ondemand-dex:ondemand-dex ownership.

[root@e80a428ddf70 ~]# ls /etc/ood/ -lrta
total 16
drwxr-xr-x 1 root         root         4096 Mar 25 16:50 .
drwxrwxr-x 5 jeff         jeff         4096 Apr 30 18:16 config
drwxr-xr-x 1 root         root         4096 May 15 12:47 ..
drwx------ 1 ondemand-dex ondemand-dex 4096 May 15 12:47 dex
[root@e80a428ddf70 ~]# ls /etc/ood/dex/
config.yaml  config.yaml.20240515T124728  dex.db  ondemand.secret
[root@e80a428ddf70 ~]# ls /etc/ood/dex/ -lrta
total 120
-rw------- 1 ondemand-dex ondemand-dex  4533 Mar 20  2023 config.yaml.20240515T124728
drwxr-xr-x 1 root         root          4096 Mar 25 16:50 ..
-rw------- 1 ondemand-dex ondemand-dex    37 May 15 12:47 ondemand.secret
-rw------- 1 ondemand-dex ondemand-dex   809 May 15 12:47 config.yaml
-rw-r--r-- 1 ondemand-dex ondemand-dex 98304 May 15 12:47 dex.db
drwx------ 1 ondemand-dex ondemand-dex  4096 May 15 12:47 .

[AMK13]

Please see the screen shot attached. I can see you have ondemand.secret and dex.db in your etc/ood/dex/ directory. Are we missing something or any steps.

[johrstrom]

Hmmmmm, OK - Do you have selinux enabled? That's the only thing I could think of.

Are we missing something or any steps.

I don't think so - I'm pretty sure you just install and start the service and everything should just happen.

I'll play around with it a bit on my end and see if there's anything you can do.

[johrstrom]

OK - I just removed the database, restarted the service and everything came back up.

What I'm guessing you need is actual dex configurations. I guess I may have assumed you had configured it, maybe you didn't. That may be the issue - it may not work out of the box. You have to provide some sort of configuration for it, like LDAP. Here's the link for configuring dex to look at your LDAP.

https://osc.github.io/ood-documentation/latest/authentication/dex.html?highlight=dex#configuring-ondemand-dex-for-ldap

[robertosussex]

Hi there, thanks What would you recommend though for a light demo? LDAP is not the easiest things to do from scratch ... Kind regards Roberto

[robertosussex]

For instance a containerized version ....etc?

[johrstrom]

Yea, we have these rake tasks in the root of the project to build a development container. Though you should note that this is for a localhost development - not for deploying to a VM where others can connect to it.

So it's something you can boot up and demo, but not something you should setup for others to connect to.

https://github.com/OSC/ondemand/blob/master/DEVELOPMENT.md#fullstack-container

[robertosussex]

Yes, makes sense thanks. R.

[robertosussex]

bundle config --local path vendor/bundle bundle install rake dev:start

===================

It does not work on my machine...anything more standard ? Regards Roberto

[johrstrom]

I don't see what the errors are to help you with that, but yes, there's an alternative prebuilt containers here: https://github.com/ubccr/hpc-toolset-tutorial/ . Though it's a large docker-compose project with Slurm and 2 compute nodes and LDAP node and coldfront

[robertosussex]

OK I had to reinstall bundler...please hold on ... R.

[robertosussex]

I got the error

Installing rspec-core 3.10.1 Fetching rspec-expectations 3.10.1 Installing rspec-expectations 3.10.1 Fetching rspec-mocks 3.10.2 Installing rspec-mocks 3.10.2 Fetching net-scp 1.2.1 Installing net-scp 1.2.1 Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory: /drives/c/Users/RobertoScipioni/AppData/Roaming/MobaXterm/home/ondemand/src/vendor/bundle/ruby/2.6.0/gems/ed25519-1.3.0/ext/ed25519_ref10

/usr/bin/ruby.exe -I /usr/share/rubygems -r ./siteconf20240517-1062-jyxq1d.rb extconf.rb mkmf.rb can't find header files for ruby at /usr/share/ruby/include/ruby.h

=========================================

Please advice

Kind regards

[johrstrom]

can't find header files for ruby at /usr/share/ruby/include/ruby.h

I think you need ruby-devel (or ruby-dev on some Linux distros). Looks like you're on Windows. I'd suggest WSL. I don't know how to build ruby programs if you're in DOS/Windows directly.

Seems like you'd need a bunch of stuff setup like libraries and so on. I don't know how to do that in Windows.

Still, maybe the containers from https://github.com/ubccr/hpc-toolset-tutorial/ are your best bet as you can pull the images directly from dockerhub without having to build them.

[robertosussex]

Hi,

On another machine ....

roberto_scipioni@Roberto3DOERK5O:~/ondemand/src$ sudo rake dev:start sudo: unable to resolve host Roberto3DOERK5O: Name or service not known rake aborted! LoadError: cannot load such file -- dotenv <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' /home/roberto_scipioni/ondemand/src/lib/tasks/rake_helper.rb:5:in <top (required)>' <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' /home/roberto_scipioni/ondemand/src/Rakefile:13:in<top (required)>'

==========================================

Please advice

Kind regards

[johrstrom]

I would not use sudo here - you don't need any privilege for this. I'm guessing you installed the gems as the non-root user, then you're attempting to use them as root. But again, aside from installing header libraries like ruby-devel you should be able to do all the instructions listed in the MD file here as an unprivileged regular user.

[johrstrom]

Also the command should be bundle exec rake dev:start.

[robertosussex]

============= Using parser 3.0.2.0 Using rainbow 3.0.0 Using regexp_parser 2.1.1 Using rubocop-ast 1.12.0 Using ruby-progressbar 1.11.0 Using unicode-display_width 2.1.0 Using rubocop 1.22.1 Using rubocop-rspec 2.6.0 Using rubyzip 2.3.2 Using selenium-webdriver 4.0.3 Using watir 7.0.0 Bundle complete! 10 Gemfile dependencies, 75 gems now installed. Bundled gems are installed into ./vendor/bundle roberto_scipioni@Roberto3DOERK5O:~/ondemand/src$ rake exec dev:start rake aborted! LoadError: cannot load such file -- dotenv <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' /home/roberto_scipioni/ondemand/src/lib/tasks/rake_helper.rb:5:in <top (required)>' <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:inrequire' <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in require' /home/roberto_scipioni/ondemand/src/Rakefile:13:in<top (required)>' /usr/share/rubygems-integration/all/gems/rake-13.0.6/exe/rake:27:in `<top (required)>' (See full trace by running task with --trace)

======================================

[johrstrom]

roberto_scipioni@Roberto3DOERK5O:~/ondemand/src$ rake exec dev:start

Issuing rake as an argument to bundle is actually important here. This should be bundle exec rake dev:start, so that rake (the command you actually want to run) is run within the bundled environment.

[robertosussex]

Ok working now......hold on :-) R.

[robertosussex]

Once everything is compiled/installed, how do I start the container OpenOn Demand ? Regards Roberto

[johrstrom]

bundle exec rake dev:start starts the container up - it just recognized that it doesn't exist so it's building it.

bundle exec rake dev:stop stops it and there's also a restart to restart it and bash to exec into the container.

[robertosussex]

localhost:8080

is asking for an email and a password I did enter a password however, what is the email?

[robertosussex]

username@localhost worked.

OK, How would I go about adding Jupyter to the container ?

Kind regards

[johrstrom]

OK, How would I go about adding Jupyter to the container ?

Yea I'm not sure how much mileage you're going to get out of that single container. The docker-compose containers in https://github.com/ubccr/hpc-toolset-tutorial/ not only have OnDemand but have a Slurm cluster too. The OnDemand container in that repository is already configured to submit to the Slurm cluster (also a part of that container) with the Jupyter app installed in it.

[robertosussex]

OK thanks. I will try https://github.com/ubccr/hpc-toolset-tutorial/

[robertosussex]

I loaded the containers from https://github.com/ubccr/hpc-toolset-tutorial/ However, upon trying loading https://localhost:3443 it gives

Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request.

..........xdmod and coldfront do seem to load...though ...

Please advice

Roberto

[robertosussex]

I have destroyed the previous containers. Downloaded them again and it now works :-) Thanks Roberto

[robertosussex]

Hi,

I was wondering whether it would e possible to add users to the demo. I suppose I would need to go inside the containers and modify the configs etc? Please advice Kind regards Roberto

[johrstrom]

I was wondering whether it would e possible to add users to the demo. I suppose I would need to go inside the containers and modify the configs etc?

Yes you can see how they get setup in the LDAP container (there's an LDAP container). That said there are several users created, mostly for coldfront demonstration:

https://github.com/ubccr/hpc-toolset-tutorial/blob/172b52cb18c3c42e5da0a44f8855bd7b738eb5e3/ldap/base.config#L2

Though they all have the same password - so you should use precaution if you're setting this up somewhere where many people can access it.

https://github.com/ubccr/hpc-toolset-tutorial/blob/172b52cb18c3c42e5da0a44f8855bd7b738eb5e3/ldap/install.sh#L37

[robertosussex]

Thanks, I will try that. Regards Roberto

[robertosussex]

Right, so where is the "base.config" ? I could find none. Please advice Roberto

[johrstrom]

It appears to be copied into the container when built - then removed during the build process. In any case - you can pull it from github: https://raw.githubusercontent.com/ubccr/hpc-toolset-tutorial/172b52cb18c3c42e5da0a44f8855bd7b738eb5e3/ldap/base.config

That said - I don't know how to add users to LDAP, though a quick google search indicates the command is ldapadd though, again, I don't quite know how to add the user with the right DN and so on.

[robertosussex]

Hi,

The question was more like where should the file go ...in which directory? Kind regards Roberto

[johrstrom]

The question was more like where should the file go ...in which directory?

Rerunning that exact file(s) won't do anything because they've already been ran during the image build process.

If you are trying something like editing those files to add more users - then it doesn't matter where they go. You're just sourcing information (environment variables) that get's written out to files to /container/service/slapd/assets/config/bootstrap/ldif/custom (found here: https://github.com/ubccr/hpc-toolset-tutorial/blob/172b52cb18c3c42e5da0a44f8855bd7b738eb5e3/ldap/install.sh#L44C11-L44C71)

The actual .sh and .config file location doesn't matter because the shell script will behave regardless of the $CWD. Though source /build/base.config in the shell script will have to be modified to point to the correct location.

[robertosussex]

OK thanks