search

OSC / ondemand

Supercomputing. Seamlessly. Open, Interactive HPC Via the Web
https://openondemand.org/
MIT License
295 stars 107 forks source link

3.0 to 3.1 upgrade issues #3833

Open btsherid opened 2 months ago

btsherid commented 2 months ago

Hi,

My Active Jobs apps was not working after upgrading from 3.0 to 3.1. I saw this error in the logs when I loaded the page.

App 415324 output: [2024-09-26 13:20:06 -0400 ] FATAL "ActionController::RoutingError (No route matches [GET] \"/assets/active_jobs-a658ac9a4d3841b112f8fcd7cd8437a541435e03c4f63c020ad95fa60a1f8634.js\"):"

I copied that .js file from a 3.0 install into /var/www/ood/apps/sys/dashboard/public/assets on the 3.1 install and the Active Jobs app started working again. Is this expected?

Also, all of my Passenger apps broke. The release notes state that ruby is updated as a depency during this upgrade, but required gems were either removed or never installed with the new version of ruby. The Passenger error log said " Could not find sinatra-3.0.6, sinatra-contrib-3.0.6, rack-2.2.6.4, tilt-2.1.0 in any of the sources (Bundler::GemNotFound)"

I ran the following commands and all of my Passenger apps started working again: gem install sinatra -v 3.0.6 gem install sinatra-contrib -v 3.0.6 gem install rack -v 2.2.6.4 gem install tilt -v 2.1.0

Thank You, Brendan Sheridan

btsherid commented 2 months ago

Other things were broken too like Interactive Apps submitting jobs. I ended up having to scp /var/www/ood/apps/sys/dashboard/public/assets/* from the 3.0 install to get things working.

I also had to create /var/www/ood/apps/sys/dashboard/tmp/cache manually and set it to 777 permissions.

johrstrom commented 2 months ago

Something's broken in your update process. You shouldn't have to create that tmp/cache and indeed may cause more errors down the line. I'd suggest maybe uninstalling then re-installing. Not sure what automation you have, but puppet for us has handled this nicely for years.

Scping assets from 3.0 to 3.1 is likely to give some errors down the road that are going to be a nightmare to debug. Again, I'd suggest maybe uninstalling then re-installing and avoiding any manual updates you need to make. If you have to make manual updates like that, they're not likely to work well long term. I'm 100% sure there'll be more issues for you that you just haven't encountered yet.

btsherid commented 2 months ago

I followed the install instructions for version 3.1. Is there something that I missed?

We use Ansible for automation and I can add automation for all of our custom OnDemand changes, but I hope to avoid having to uninstall and re-install the OnDemand software on our prod instance.

Thanks, Brendan

johrstrom commented 2 months ago

Oh you used the ansible role and still got into this situation. Unfortunate. We see issues like that come up sometimes, but we're never able to replicate. Something, somehow, didn't install quite right.