search

OSC / ondemand

Supercomputing. Seamlessly. Open, Interactive HPC Via the Web
https://openondemand.org/
MIT License
278 stars 104 forks source link

undefined method `bc_clean_old_dirs?' #3034

Closed evakichi closed 11 months ago

evakichi commented 1 year ago

Hello,

I encountered the error that "undefined method `bc_clean_old_dirs?'".

I captured the html which was created by Phusion Passenger.

This messages are as following,

Error details:

undefined method bc_clean_old_dirs?' for #<ConfigurationSingleton:0x00007f54355263f0 @config={}> (NoMethodError) /var/www/ood/apps/sys/dashboard/config/initializers/bc_cleanup.rb:4:inblock in <top (required)>' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:68:in block in execute_hook' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:61:inwith_execution_control' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:66:in execute_hook' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:52:inblock in run_load_hooks' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:51:in each' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:51:inrun_load_hooks' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/application/finisher.rb:140:in block in <module:Finisher>' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/initializable.rb:32:ininstance_exec' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/initializable.rb:32:in run' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/initializable.rb:61:inblock in run_initializers' /usr/share/ruby/tsort.rb:228:in block in tsort_each' /usr/share/ruby/tsort.rb:350:inblock (2 levels) in each_strongly_connected_component' /usr/share/ruby/tsort.rb:431:in each_strongly_connected_component_from' /usr/share/ruby/tsort.rb:349:inblock in each_strongly_connected_component' /usr/share/ruby/tsort.rb:347:in each' /usr/share/ruby/tsort.rb:347:incall' /usr/share/ruby/tsort.rb:347:in each_strongly_connected_component' /usr/share/ruby/tsort.rb:226:intsort_each' /usr/share/ruby/tsort.rb:205:in tsort_each' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/initializable.rb:60:inrun_initializers' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/application.rb:391:in initialize!' /var/www/ood/apps/sys/dashboard/config/environment.rb:5:in<top (required)>' config.ru:5:in require_relative' config.ru:5:inblock in

' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/rack-2.2.4/lib/rack/builder.rb:125:in instance_eval' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/rack-2.2.4/lib/rack/builder.rb:125:ininitialize' config.ru:1:in new' config.ru:1:in
' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:96:in eval' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:96:inload_app' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:116:in block in <module:App>' /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:397:inrun_block_and_record_step_progress' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:115:in <module:App>' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:28:in' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:27:in `
'

The stdout/stderr output of the subprocess so far is:

Error: The application encountered the following error: undefined method bc_clean_old_dirs?' for #<ConfigurationSingleton:0x00007f54355263f0 @config={}> (NoMethodError) /var/www/ood/apps/sys/dashboard/config/initializers/bc_cleanup.rb:4:inblock in <top (required)>' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:68:in block in execute_hook' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:61:inwith_execution_control' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:66:in execute_hook' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:52:inblock in run_load_hooks' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:51:in each' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/activesupport-6.1.7/lib/active_support/lazy_load_hooks.rb:51:inrun_load_hooks' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/application/finisher.rb:140:in block in <module:Finisher>' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/initializable.rb:32:ininstance_exec' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/initializable.rb:32:in run' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/initializable.rb:61:inblock in run_initializers' /usr/share/ruby/tsort.rb:228:in block in tsort_each' /usr/share/ruby/tsort.rb:350:inblock (2 levels) in each_strongly_connected_component' /usr/share/ruby/tsort.rb:431:in each_strongly_connected_component_from' /usr/share/ruby/tsort.rb:349:inblock in each_strongly_connected_component' /usr/share/ruby/tsort.rb:347:in each' /usr/share/ruby/tsort.rb:347:incall' /usr/share/ruby/tsort.rb:347:in each_strongly_connected_component' /usr/share/ruby/tsort.rb:226:intsort_each' /usr/share/ruby/tsort.rb:205:in tsort_each' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/initializable.rb:60:inrun_initializers' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/railties-6.1.7/lib/rails/application.rb:391:in initialize!' /var/www/ood/apps/sys/dashboard/config/environment.rb:5:in<top (required)>' config.ru:5:in require_relative' config.ru:5:inblock in

' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/rack-2.2.4/lib/rack/builder.rb:125:in instance_eval' /opt/ood/ondemand/root/usr/share/gems/3.0/gems/rack-2.2.4/lib/rack/builder.rb:125:ininitialize' config.ru:1:in new' config.ru:1:in
' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:96:in eval' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:96:inload_app' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:116:in block in <module:App>' /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:397:inrun_block_and_record_step_progress' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:115:in <module:App>' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:28:in' /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:27:in `
'

How can I solve by myself ? or are you able to fix this error.

Thanks !!

johrstrom commented 1 year ago

That is a very strange error. I know where it's coming from - but just don't quite know why it would happen.

What version & OS are you running?

johrstrom commented 1 year ago

Yea I don't really get how you could wind up in this state.

You could remove this file (or back it up really) /var/www/ood/apps/sys/dashboard/config/initializers/bc_cleanup.rb and that may solve this issue - but I fear there's something else going on here.

You didn't happen to modify /var/www/ood/apps/sys/dashboard/config/configuration_singleton.rb did you?

evakichi commented 1 year ago

Thank you for your reply.

I challenged remove bc_cleanup.rb along with your suggestion. After that I encountered 503 Service Unavailable.

My environment is rocky 8 running in Docker. Open OnDemand version is latest. It's including my customization, in detail , this Dockerfile includes 'CA cert and SSL Server cert generation', and 'setting for Japanese language representation'.

By the way this Dockerfile was used for older Open OnDemand, and I changed part that this can install latest Open OnDemand.

If you want my Dockerfile, I can describe it.

And I guess I didn't modify /var/www/ood/apps/sys/dashboard/config/configuration_singleton.rb.

thanks.

johrstrom commented 1 year ago

OK there's something really funny going on here.

If you want my Dockerfile, I can describe it.

Yes please. And/or use the Dockerfile.dev we have. We have this rake task to build it:

bundle config path --local vendor/bundle
bundle install
bundle exec rake dev:start

Where it'll prompt you for a password to use. Then you can mount things from ~/.config/ondemand/container/config/ to /etc/ood/config (like your japanese locale file).

evakichi commented 1 year ago

Here is my Dockerfile

FROM rockylinux/rockylinux:8
LABEL maintainer="tdockendorf@osc.edu; johrstrom@osc.edu"

ARG VERSION=latest
ARG CONCURRENCY=4
ENV PYTHON=/usr/libexec/platform-python

# setup the ondemand repositories
RUN dnf -y install https://yum.osc.edu/ondemand/3.0/ondemand-release-web-3.0-1.noarch.rpm 

# setup for PBS -start-
COPY files/pbspro-client-2021.1.4.20220802060528-0.el8.x86_64.rpm /tmp/pbspro-client-2021.1.4.20220802060528-0.el8.x86_64.rpm
RUN dnf localinstall -y /tmp/pbspro-client-2021.1.4.20220802060528-0.el8.x86_64.rpm
# setup for PBS -end-

# install all the dependencies
RUN dnf -y update && \
    dnf install -y dnf-utils epel-release && \
    dnf config-manager --set-enabled powertools && \
    dnf -y module enable nodejs:14 ruby:3.0 && \
    dnf install -y \
        file \
        lsof \
        sudo \
        gcc \
        gcc-c++ \
        git \
        patch \
        lua-posix \
        java-latest-openjdk \
# add -start-
        rsync \
        bind-utils \
        openldap-clients \
        sssd \
        sssd-ldap \
        oddjob-mkhomedir \
        authselect \
        tree \
        munge \
        easy-rsa \
# add -end-
        ondemand-gems \
        ondemand-runtime \
        ondemand-build \
        ondemand-apache \
        ondemand-ruby \
        ondemand-nodejs \
#        ondemand-python \
        ondemand-dex \
        ondemand-passenger \
        ondemand_exporter \
        ondemand-nginx && \
    dnf clean all && rm -rf /var/cache/dnf/*

RUN mkdir -p /opt/ood
RUN mkdir -p /var/www/ood/{apps,public,discover}
RUN mkdir -p /var/www/ood/apps/{sys,dev,usr}

# change file -start-
COPY files/launch-ood      /opt/ood/launch
# change file -end-

COPY mod_ood_proxy          /opt/ood/mod_ood_proxy
COPY nginx_stage            /opt/ood/nginx_stage
COPY ood-portal-generator   /opt/ood/ood-portal-generator
COPY ood_auth_map           /opt/ood/ood_auth_map
COPY apps                   /opt/ood/apps
COPY Rakefile               /opt/ood/Rakefile
COPY lib                    /opt/ood/lib
COPY Gemfile                /opt/ood/Gemfile

RUN cd /opt/ood; bundle install

RUN source /opt/rh/ondemand/enable && \
    rake -f /opt/ood/Rakefile -mj$CONCURRENCY build && \
# change command -start-
    cp -r /opt/ood/apps/* /var/www/ood/apps/sys/ && \
# change command -end-
    rm -rf /opt/ood/Rakefile /opt/ood/apps /opt/ood/lib

# copy customized ood_portal_example.yml
COPY files/ood_portal_example.yml /opt/ood/ood-portal-generator/share/ood_portal_example.yml 

# copy configuration files
RUN mkdir -p /etc/ood/config
RUN cp /opt/ood/nginx_stage/share/nginx_stage_example.yml            /etc/ood/config/nginx_stage.yml
RUN cp /opt/ood/ood-portal-generator/share/ood_portal_example.yml    /etc/ood/config/ood_portal.yml

# make some misc directories & files
RUN mkdir -p /var/lib/ondemand-nginx/config/apps/{sys,dev,usr}
RUN touch /var/lib/ondemand-nginx/config/apps/sys/{dashboard,shell,myjobs}.conf

# setup sudoers for apache
RUN echo -e 'Defaults:apache !requiretty, !authenticate \n\
Defaults:apache env_keep += "NGINX_STAGE_* OOD_*" \n\
apache ALL=(ALL) NOPASSWD: /opt/ood/nginx_stage/sbin/nginx_stage' >/etc/sudoers.d/ood

# copy the files -start-
COPY files/my_cluster.yml /etc/ood/config/clusters.d/my_cluster.yml
COPY files/env /etc/ood/config/apps/dashboard/env
COPY files/myjobs/ja.yml /var/www/ood/apps/sys/myjobs/config/locales/ja.yml
COPY files/dashboard/ja.yml /var/www/ood/apps/sys/dashboard/config/locales/ja.yml
# copy the files -end-

# run the OOD executables to setup the env
RUN /opt/ood/ood-portal-generator/sbin/update_ood_portal --insecure
RUN /opt/ood/nginx_stage/sbin/update_nginx_stage
RUN echo $VERSION > /opt/ood/VERSION
# this one bc centos:8 doesn't generate localhost cert
RUN /usr/libexec/httpd-ssl-gencerts

# setup for SSSD -start-
COPY sssd.conf.cskk /etc/sssd/sssd.conf
RUN  chmod 600 /etc/sssd/sssd.conf
RUN echo "URI ldap://192.168.100.1/" >> /etc/openldap/ldap.conf && \
    echo "BASE something " >> /etc/openldap/ldap.conf && \
    authselect select sssd with-mkhomedir --force
# setup for SSSD -end-

# setup pbs.conf -start-
RUN sed -i 's/CHANGE_THIS_TO_PBS_PRO_SERVER_HOSTNAME/foo/' /etc/pbs.conf
# setup pbs.conf -end-

# setup comsol -start-
RUN cd /var/www/ood/apps/sys/ && git clone https://github.com/OSC/bc_osc_comsol.git && mv bc_osc_comsol comsol
RUN cd /var/www/ood/apps/sys/comsol && rm .gitlab-ci.yml && sed -i "s/owens/my_cluster/g" form.yml.erb
# setup comsol -end-

# setup pin -start-
RUN mkdir -p /etc/odd/config/ondemand.d/
COPY files/ondemand.yml  /etc/ood/config/ondemand.d/ondemand.yml
# setup pin -end-

# setup easy-rsa -start-
RUN cd /root/ && \
    ln -s /usr/share/easy-rsa ./ && \
    cd easy-rsa/3 && \
    ./easyrsa init-pki && \
    yes something | ./easyrsa build-ca nopass && \
    ./easyrsa build-server-full something nopass && \
    cp /root/easy-rsa/3/pki/ca.crt /usr/share/pki/ca-trust-source/anchors/ && \
    update-ca-trust extract &&\
    sed -i "s/#ServerName www\.example\.com/ServerName something/g" /etc/httpd/conf.d/ssl.conf
# setup easy-rsa -end-

# add -start-
EXPOSE 80
EXPOSE 9301
# add -end-
EXPOSE 443
EXPOSE 8080
EXPOSE 5556
EXPOSE 3035

CMD [ "/opt/ood/launch" ]

and here is my launch file

#!/bin/bash

set -e

/opt/ood/ood-portal-generator/sbin/update_ood_portal --force --insecure
runuser -u ondemand-dex /usr/sbin/ondemand-dex serve /etc/ood/dex/config.yaml &
/usr/bin/ondemand_exporter & 
/usr/sbin/sssd &
/usr/sbin/oddjobd &
/usr/sbin/munged &
/usr/sbin/httpd -DFOREGROUND

Also I want to challenge for your suggestion later.

Thanks.

johrstrom commented 1 year ago

Yea so your Dockerfile is copying source files. You should instead install an RPM like Dockerfile.dev does.

Dockerfile.dev installs a nightly RPM - It seems like you can remove this line (and then you probably need to edit the next line on what ruby & nodejs versions to include or just remove them as the RPM will pull them as needed). https://github.com/OSC/ondemand/blob/4c5905b9968568a45f02531c814e043a2025f17f/Dockerfile.dev#L11

Or if you want a full suite with ldap & Slurm and everything else you can try to use https://github.com/ubccr/hpc-toolset-tutorial/

Note that all of these docker installations are for demonstration purposes - I wouldn't suggest building out a production environment from them.

evakichi commented 1 year ago

Thank you for your advice.

In fact, my environment is very strange, in detail that's completely the Internet unreachable environment. Thus, It is difficult to install by RPM for me, because I cannot download several RPMs such as Ruby and Node.js from the Internet. Therefore I choose followings,

  1. building docker image on environment that is the Internet reachable,
  2. saving it's image,
  3. moving it's image from 'reachable environment' to 'unreachable environment',
  4. loading it's image,
  5. running 'docker compose'.

Since this is my own business and I understood that Docker environment is not production purpose from your information, I must image RPM installation environment instead of Docker. But I cannot image the best way.

Can you help me if you know 'minimal set' of installation of Open OnDemand with RPMs installation?

I guess somehow I must install them along with your suggestion.

Thanks!!

johrstrom commented 1 year ago

Since this is my own business and I understood that Docker environment is not production purpose from your information, I must image RPM installation environment instead of Docker. But I cannot image the best way.

Can you help me if you know 'minimal set' of installation of Open OnDemand with RPMs installation?

I only meant install the RPM in the Dockerfile as Dockerfile.dev does. I think Dockerfile.dev is a better starting point for you than the other Dockerfile.

My suggestion is to start with Dockerfile.dev and modify as needed (like installing 3.0.x instead of a nightly RPM).

evakichi commented 1 year ago

Thank you very much!!

Now I am trying to build with Dockerfile.dev. But I encountered another problem.

I will try and error.

Which is better remain or close this issue?

Thanks.

johrstrom commented 1 year ago

Which is better remain or close this issue?

I think another ticket is best.

johrstrom commented 11 months ago

@evakichi I'm going to close this ticket as I think we've moved passed the original issue. Feel free to open a new ticket, a topic on discourse or just to request this ticket remain open.