search

ubccr / hpc-toolset-tutorial

Tutorial for installing Open XDMoD, OnDemand, & ColdFront
GNU General Public License v3.0
121 stars 72 forks source link

Open OnDemand Internal Server Error #184

Closed james-s-willis closed 3 months ago

james-s-willis commented 3 months ago

When trying to open the OnDemand URL: https://localhost:3443 in a browser I get the following error:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

The error appears in the ondemand logs (docker compose logs -f ondemand) as:

ondemand  | Completed successfully!
ondemand  | ---> Starting ondemand-dex...
ondemand  | ---> Starting ondemand httpd24...
ondemand  | AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.19.0.12. Set the 'ServerName' directive globally to suppress this message
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="Dex Version: v2.36.0, Go Version: go1.19.2, Go OS/ARCH: linux amd64"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config issuer: https://localhost:5554"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config storage: sqlite3"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config static client: OnDemand"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config connector: ldap"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config skipping approval screen"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config refresh tokens rotation enabled: true"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="keys expired, rotating"
ondemand  | time="2024-07-25T14:22:48Z" level=info msg="keys rotated, next rotation: 2024-07-25 20:22:47.998753652 +0000 UTC"
ondemand  | time="2024-07-25T14:22:48Z" level=info msg="listening (telemetry) on 0.0.0.0:5558"
ondemand  | time="2024-07-25T14:22:48Z" level=info msg="listening (http) on 0.0.0.0:5556"
ondemand  | time="2024-07-25T14:22:48Z" level=info msg="listening (https) on 0.0.0.0:5554"
ondemand  | 2024/07/25 14:23:58 http: TLS handshake error from [::1]:52182: local error: tls: bad record MAC
ondemand  | 2024/07/25 14:39:17 http: TLS handshake error from [::1]:51144: local error: tls: bad record MAC
ondemand  | 2024/07/25 14:42:10 http: TLS handshake error from [::1]:40056: local error: tls: bad record MAC

I'm running the containers in Pop OS 20.04. XDMoD and Coldfront interfaces work correctly. The Open OnDemand interface had previously worked as recent as 2 weeks ago. I have tried running ./hpcts destroy and ./hpcts cleanup as well as cloning a new repository and running the setup again. My colleague can also reproduce the error.

The fact that it had previously worked leads me to believe it could be an expired certificate maybe?

johrstrom commented 3 months ago

The fact that it had previously worked leads me to believe it could be an expired certificate maybe?

It is exactly an expired certificate. IDK if they're publishing new images or not, but if you build the images it should work. That is, what's on Dockerhub is bad, but rebuilding locally with docker compose build should work (on x86 chips at least. If you have a Mac M1 I think there's still a patch that this repo needs).

vanzonr commented 3 months ago

I am seeing the same error in Ubuntu 22.04.4.
I noticed that docker exec -it ondemand openssl x509 -enddate -noout -in /etc/pki/tls/certs/localhost.crt gives an expiration date of 'Jul 13 12:27:14 2024 GMT'.

johrstrom commented 3 months ago

Yea the images need to be rebuilt. You can do so locally and it should work. But again, it'll work for x86 machines. If you're on a Mac M1 there's some issue in building that we haven't patched yet.

james-s-willis commented 3 months ago

Thanks! Rebuilding the images locally worked. However, some functionality fails. When trying to access the Job Composer tab I get:

Exception: OodApp::SetupScriptFailed

Per user setup failed for script at /var/www/ood/apps/sys/myjobs/./bin/setup-production for user hpcadmin with output: /usr/share/gems/gems/bundler-2.2.33/lib/bundler/definition.rb:480:in `materialize': Could not find rdoc-6.3.3 in any of the sources (Bundler::GemNotFound)
    from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/definition.rb:185:in `specs'
    from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/definition.rb:233:in `specs_for'
    from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:18:in `setup'
    from /usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:150:in `setup'
    from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/setup.rb:20:in `block in <top (required)>'
    from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/ui/shell.rb:136:in `with_level'
    from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/ui/shell.rb:88:in `silence'
    from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/setup.rb:20:in `<top (required)>'
    from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `require'
    from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
    from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:149:in `require'
<internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- bundler/setup (LoadError)
    from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
Stack trace:

/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:271:in `block (2 levels) in run_setup_production'
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:257:in `chdir'
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:257:in `block in run_setup_production'
/usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:398:in `block in with_unbundled_env'
/usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:692:in `with_env'
/usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:398:in `with_unbundled_env'
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:254:in `run_setup_production'
/var/www/ood/apps/sys/dashboard/app/controllers/apps_controller.rb:35:in `show'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:228:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:106:in `run_callbacks'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:165:in `process'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:39:in `process'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:190:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:254:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `each'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:842:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/etag.rb:27:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/conditional_get.rb:27:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/head.rb:12:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:266:in `context'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:260:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/cookies.rb:697:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/lograge-0.12.0/lib/lograge/rails_ext/rack/logger.rb:18:in `call_app'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:26:in `block in call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:37:in `tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:99:in `tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/method_override.rb:24:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/runtime.rb:22:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/sendfile.rb:110:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/host_authorization.rb:148:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/engine.rb:539:in `call'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:419:in `block (3 levels) in start_threads'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'

Any ideas what could be causing this?

johrstrom commented 3 months ago

Sorry about that! I see that appear every now and again on some systems for whatever reason.

I found that I could exec into the container and install the gem and then it would work.

docker exec -it ondemand /bin/bash
[root@ondemand /]# source /opt/ood/ondemand/enable 
[root@ondemand /]# gem install rdoc -v 6.3.3
james-s-willis commented 3 months ago

Thanks! That fix worked!

JAM2199562 commented 2 months ago

For those who still don't know how to solve this problem

cd hpc-toolset-tutorial
docker compose build --no-cache base
docker compose build

Additionally, you may encounter websockify compilation failures

add a new line sed -i 's/numpy/numpy==1.26/' setup.py

before https://github.com/ubccr/hpc-toolset-tutorial/blob/6baa9a3950cff280bb398aea26420f279bebf2df/slurm/install.sh#L53

mw201608 commented 1 month ago

Can we re-open this issue? JAM2199562's approach does not work for me and the certificate does not update after re-build. Am I missing anything here? Thanks.