search

bgruening / docker-galaxy

:whale::bar_chart::books: Docker Images tracking the stable Galaxy releases.
http://bgruening.github.io/docker-galaxy
MIT License
226 stars 134 forks source link

403/Forbidden and "Could not open job state file" #571

Closed marco-brandizi closed 3 days ago

marco-brandizi commented 4 years ago

I'm trying to set up a local Galaxy instance, using this Docker project.

I've tried this command:

# docker-run.sh
cd `dirname "$0"`
docker run -it -p 8080:80 -v `pwd`/galaxy-store/:/export/ --name galaxy bgruening/galaxy-stable

Initially, it says: "Checking /export..." for about 30 mins, then it starts yielding several messages, including errors like:

FATAL:  the database system is starting up # repeated multiple times
...
==> /home/galaxy/logs/slurmctld.log <==
[2020-11-04T14:14:55.442] error: Could not open job state file /tmp/slurm/job_state: No such file or directory
[2020-11-04T14:14:55.442] error: NOTE: Trying backup state save file. Jobs may be lost!
[2020-11-04T14:14:55.442] No job state file (/tmp/slurm/job_state.old) found

I get the messages about job_state after I tried http://localhost:8080 with the host browser. The latter returns "403 Forbidden".

Any ideas?

shenweiyan commented 3 years ago

I got the same error .

bgruening commented 3 years ago

@shenweiyan are you also using the export? What happens if you do not use the export?

shenweiyan commented 3 years ago

@shenweiyan are you also using the export? What happens if you do not use the export?

I got something wrong as follow:

$ docker run -it -p 9080:80 -p 9081:21 -p 9082:22 bgruening/galaxy-stable
Enable Galaxy reports authentification
Checking /export...
Disable Galaxy Interactive Environments. Start with --privileged to enable IE's.
Starting postgres
postgresql: started
Checking if database is up and running
Traceback (most recent call last):
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/check_database.py", line 25, in <module>
    query.count()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3749, in count
    return self.from_self(col).scalar()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3469, in scalar
    ret = self.one()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3436, in one
    ret = self.one_or_none()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3405, in one_or_none
    ret = list(self)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
    return self._execute_and_instances(context)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
    distilled_params,
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
    e, statement, parameters, cursor, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

.....

==> /home/galaxy/logs/uwsgi.log <==
Traceback (most recent call last):
  File "lib/galaxy/webapps/galaxy/buildapp.py", line 48, in app_factory
    app = galaxy.app.UniverseApplication(global_conf=global_conf, **kwargs)
  File "lib/galaxy/app.py", line 97, in __init__
    self._configure_models(check_migrate_databases=self.config.check_migrate_databases, check_migrate_tools=check_migrate_tools, config_file=config_file)
  File "lib/galaxy/config/__init__.py", line 1226, in _configure_models
    create_or_verify_database(db_url, config_file, self.config.database_engine_options, app=self, map_install_models=combined_install_database)
  File "lib/galaxy/model/migrate/check.py", line 140, in create_or_verify_database
    raise Exception("{}. {}{}".format(expect_msg, instructions, backup_msg))
Exception: Your database has version '1' but this code expects version '170'. Please backup your database and then migrate the database schema by running 'sh manage_db.sh upgrade'.

....
bgruening commented 3 years ago

Can you try docker run -it -p 9080:80 -p 9081:21 -p 9082:22 quay.io/bgruening/galaxy

shenweiyan commented 3 years ago

On my CentOS 7, the error is still the same. But I can see the galaxy web page on http://localhost:9080.

$ docker run -it -p 9080:80 -p 9081:21 -p 9082:22 quay.io/bgruening/galaxy
Enable Galaxy reports authentification
Checking /export...
Disable Galaxy Interactive Environments. Start with --privileged to enable IE's.
Starting postgres
postgresql: started
Checking if database is up and running
Traceback (most recent call last):
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/check_database.py", line 25, in <module>
    query.count()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3749, in count
    return self.from_self(col).scalar()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3469, in scalar
    ret = self.one()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3436, in one
    ret = self.one_or_none()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3405, in one_or_none
    ret = list(self)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
    return self._execute_and_instances(context)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
    distilled_params,
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
    e, statement, parameters, cursor, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

[SQL: SELECT count(*) AS count_1
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token
FROM galaxy_user
WHERE galaxy_user.email = %(email_1)s) AS anon_1]
[parameters: {'email_1': 'admin@galaxy.org'}]
(Background on this error at: http://sqlalche.me/e/f405)
Waiting for database
Traceback (most recent call last):
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/check_database.py", line 25, in <module>
    query.count()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3749, in count
    return self.from_self(col).scalar()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3469, in scalar
    ret = self.one()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3436, in one
    ret = self.one_or_none()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3405, in one_or_none
    ret = list(self)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
    return self._execute_and_instances(context)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
    distilled_params,
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
    e, statement, parameters, cursor, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

[SQL: SELECT count(*) AS count_1
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token
FROM galaxy_user
WHERE galaxy_user.email = %(email_1)s) AS anon_1]
[parameters: {'email_1': 'admin@galaxy.org'}]
(Background on this error at: http://sqlalche.me/e/f405)
Waiting for database
Traceback (most recent call last):
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/check_database.py", line 25, in <module>
    query.count()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3749, in count
    return self.from_self(col).scalar()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3469, in scalar
    ret = self.one()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3436, in one
    ret = self.one_or_none()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3405, in one_or_none
    ret = list(self)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
    return self._execute_and_instances(context)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
    distilled_params,
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
    e, statement, parameters, cursor, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

[SQL: SELECT count(*) AS count_1
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token
FROM galaxy_user
WHERE galaxy_user.email = %(email_1)s) AS anon_1]
[parameters: {'email_1': 'admin@galaxy.org'}]
(Background on this error at: http://sqlalche.me/e/f405)
Waiting for database
Traceback (most recent call last):
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/check_database.py", line 25, in <module>
    query.count()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3749, in count
    return self.from_self(col).scalar()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3469, in scalar
    ret = self.one()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3436, in one
    ret = self.one_or_none()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3405, in one_or_none
    ret = list(self)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
    return self._execute_and_instances(context)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
    distilled_params,
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
    e, statement, parameters, cursor, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

[SQL: SELECT count(*) AS count_1
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token
FROM galaxy_user
WHERE galaxy_user.email = %(email_1)s) AS anon_1]
[parameters: {'email_1': 'admin@galaxy.org'}]
(Background on this error at: http://sqlalche.me/e/f405)
Waiting for database
Database connected
Starting cron
cron: started
Starting ProFTP
proftpd: started
Starting Galaxy reports webapp
reports: started
Starting nodejs
galaxy:galaxy_nodejs_proxy: started
Starting condor
condor: started
Starting slurmctld
Starting slurmd
Creating admin user admin with key fakekey and password password if not existing
==> /home/galaxy/logs/handler0.log <==
galaxy.jobs.handler INFO 2021-10-15 09:07:50,893 job handler queue started
galaxy.jobs.handler INFO 2021-10-15 09:07:50,910 job handler stop queue started
galaxy.workflow.scheduling_manager INFO 2021-10-15 09:07:50,979 No workflow schedulers plugin config file defined, using default scheduler.
galaxy.workflow.scheduling_manager DEBUG 2021-10-15 09:07:50,979 Starting workflow schedulers
galaxy.queue_worker INFO 2021-10-15 09:07:51,524 Binding and starting galaxy control worker for handler0
galaxy.queue_worker INFO 2021-10-15 09:07:51,798 Queuing async task rebuild_toolbox_search_index for handler0.
galaxy.app INFO 2021-10-15 09:07:54,088 Galaxy app startup finished (96604.370 ms)
galaxy.web_stack INFO 2021-10-15 09:07:54,088 Galaxy server instance 'handler0' is running
galaxy.queue_worker INFO 2021-10-15 09:07:55,100 Instance 'handler0' received 'rebuild_toolbox_search_index' task, executing now.
galaxy.queue_worker DEBUG 2021-10-15 09:07:55,100 App is not a webapp, not building a search index

==> /home/galaxy/logs/handler1.log <==
galaxy.workflow.scheduling_manager INFO 2021-10-15 09:07:46,616 No workflow schedulers plugin config file defined, using default scheduler.
galaxy.workflow.scheduling_manager DEBUG 2021-10-15 09:07:46,616 Starting workflow schedulers
galaxy.queue_worker INFO 2021-10-15 09:07:47,097 Binding and starting galaxy control worker for handler1
galaxy.queue_worker INFO 2021-10-15 09:07:47,217 Queuing async task rebuild_toolbox_search_index for handler1.
galaxy.model.database_heartbeat DEBUG 2021-10-15 09:07:48,212 handler1 is config watcher
galaxy.util.watcher DEBUG 2021-10-15 09:07:48,896 Watching for changes in directory (recursively): /galaxy-central/tool-data
galaxy.app INFO 2021-10-15 09:07:49,682 Galaxy app startup finished (92769.326 ms)
galaxy.web_stack INFO 2021-10-15 09:07:49,683 Galaxy server instance 'handler1' is running
galaxy.queue_worker INFO 2021-10-15 09:07:50,611 Instance 'handler1' received 'rebuild_toolbox_search_index' task, executing now.
galaxy.queue_worker DEBUG 2021-10-15 09:07:50,611 App is not a webapp, not building a search index

==> /home/galaxy/logs/reports.log <==
galaxy.web.framework.base DEBUG 2021-10-15 09:07:50,801 Enabling 'jobs' controller, class: Jobs
galaxy.web.framework.base DEBUG 2021-10-15 09:07:50,806 Enabling 'users' controller, class: Users
galaxy.web.framework.base DEBUG 2021-10-15 09:07:50,827 Enabling 'system' controller, class: System
galaxy.webapps.util DEBUG 2021-10-15 09:07:50,879 Enabling 'paste.httpexceptions' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:07:51,885 Enabling 'RecursiveMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:07:52,079 Enabling 'ErrorMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:07:52,087 Enabling 'TransLogger' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:07:52,092 Enabling 'XForwardedHostMiddleware' middleware
Starting server in PID 616.
serving on http://127.0.0.1:9001

==> /home/galaxy/logs/slurmctld.log <==
[2021-10-15T09:06:26.382] No trigger state file (/tmp/slurm/trigger_state.old) to recover
[2021-10-15T09:06:26.382] _preserve_plugins: backup_controller not specified
[2021-10-15T09:06:26.382] Reinitializing job accounting state
[2021-10-15T09:06:26.382] cons_res: select_p_reconfigure
[2021-10-15T09:06:26.382] cons_res: select_p_node_init
[2021-10-15T09:06:26.382] cons_res: preparing for 1 partitions
[2021-10-15T09:06:26.383] Running as primary controller
[2021-10-15T09:06:26.414] No parameter for mcs plugin, default values set
[2021-10-15T09:06:26.414] mcs: MCSParameters = (null). ondemand set.
[2021-10-15T09:06:29.806] SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=2,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=2

==> /home/galaxy/logs/slurmd.log <==
[2021-10-15T09:06:28.014] error: Domain socket directory /var/spool/slurmd: No such file or directory
[2021-10-15T09:06:28.182] Message aggregation disabled
[2021-10-15T09:06:28.186] CPU frequency setting not configured for this node
[2021-10-15T09:06:28.325] slurmd version 17.11.2 started
[2021-10-15T09:06:28.386] slurmd started on Fri, 15 Oct 2021 09:06:28 +0000
[2021-10-15T09:06:28.398] CPUs=4 Boards=1 Sockets=4 Cores=1 Threads=1 Memory=15871 TmpDisk=1007799 Uptime=4720391 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)

==> /home/galaxy/logs/uwsgi.log <==
galaxy.web.framework.base DEBUG 2021-10-15 09:07:42,413 [p:499,w:0,m:0] [MainThread] Enabling 'error' controller, class: Error
galaxy.web.framework.base DEBUG 2021-10-15 09:07:42,716 [p:499,w:0,m:0] [MainThread] Enabling 'visualization' controller, class: VisualizationController
galaxy.web.framework.base DEBUG 2021-10-15 09:07:43,505 [p:499,w:0,m:0] [MainThread] Enabling 'history' controller, class: HistoryController
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,799 [p:499,w:0,m:0] [MainThread] Enabling 'workflow' controller, class: WorkflowController
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,891 [p:499,w:0,m:0] [MainThread] Enabling 'authnz' controller, class: JSAppLauncher
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,892 [p:499,w:0,m:0] [MainThread] Enabling 'authnz' controller, class: OIDC
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,907 [p:499,w:0,m:0] [MainThread] Enabling 'root' controller, class: RootController
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,908 [p:499,w:0,m:0] [MainThread] Enabling 'library' controller, class: Library
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,909 [p:499,w:0,m:0] [MainThread] Enabling 'admin' controller, class: AdminGalaxy
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,916 [p:499,w:0,m:0] [MainThread] Enabling 'forms' controller, class: Forms
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,199 [p:499,w:0,m:0] [MainThread] Enabling 'page' controller, class: PageController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,208 [p:499,w:0,m:0] [MainThread] Enabling 'tag' controller, class: TagsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,216 [p:499,w:0,m:0] [MainThread] Enabling 'user' controller, class: User
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,227 [p:499,w:0,m:0] [MainThread] Enabling 'roles' API controller, class: RoleAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,280 [p:499,w:0,m:0] [MainThread] Enabling 'tags' API controller, class: TagsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,288 [p:499,w:0,m:0] [MainThread] Enabling 'tool_shed_repositories' API controller, class: ToolShedRepositoriesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,291 [p:499,w:0,m:0] [MainThread] Enabling 'trs_consumer' API controller, class: TrsConsumeAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,297 [p:499,w:0,m:0] [MainThread] Enabling 'groups' API controller, class: GroupAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,300 [p:499,w:0,m:0] [MainThread] Enabling 'webhooks' API controller, class: WebhooksController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,304 [p:499,w:0,m:0] [MainThread] Enabling 'dataset_collections' API controller, class: DatasetCollectionsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,315 [p:499,w:0,m:0] [MainThread] Enabling 'remote_files' API controller, class: RemoteFilesAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,320 [p:499,w:0,m:0] [MainThread] Enabling 'folders' API controller, class: FoldersController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,322 [p:499,w:0,m:0] [MainThread] Enabling 'tool_entry_points' API controller, class: ToolEntryPointsAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,326 [p:499,w:0,m:0] [MainThread] Enabling 'authenticate' API controller, class: AuthenticationController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,903 [p:499,w:0,m:0] [MainThread] Enabling 'search' API controller, class: SearchController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,908 [p:499,w:0,m:0] [MainThread] Enabling 'folder_contents' API controller, class: FolderContentsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,911 [p:499,w:0,m:0] [MainThread] Enabling 'history_content_provenance' API controller, class: HDAProvenanceController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,912 [p:499,w:0,m:0] [MainThread] Enabling 'ldda_provenance' API controller, class: LDDAProvenanceController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,914 [p:499,w:0,m:0] [MainThread] Enabling 'page_revisions' API controller, class: PageRevisionsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,987 [p:499,w:0,m:0] [MainThread] Enabling 'tools' API controller, class: ToolsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,989 [p:499,w:0,m:0] [MainThread] Enabling 'metrics' API controller, class: MetricsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,993 [p:499,w:0,m:0] [MainThread] Enabling 'job_files' API controller, class: JobFilesAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,999 [p:499,w:0,m:0] [MainThread] Enabling 'quotas' API controller, class: QuotaAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,001 [p:499,w:0,m:0] [MainThread] Enabling 'display_applications' API controller, class: DisplayApplicationsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,017 [p:499,w:0,m:0] [MainThread] Enabling 'cloudauthz' API controller, class: CloudAuthzController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,020 [p:499,w:0,m:0] [MainThread] Enabling 'configuration' API controller, class: ConfigurationController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,103 [p:499,w:0,m:0] [MainThread] Enabling 'cloud' API controller, class: CloudController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,105 [p:499,w:0,m:0] [MainThread] Enabling 'history_content_tags' API controller, class: HistoryContentTagsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,106 [p:499,w:0,m:0] [MainThread] Enabling 'history_tags' API controller, class: HistoryTagsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,106 [p:499,w:0,m:0] [MainThread] Enabling 'workflow_tags' API controller, class: WorkflowTagsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,116 [p:499,w:0,m:0] [MainThread] Enabling 'history_contents' API controller, class: HistoryContentsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,119 [p:499,w:0,m:0] [MainThread] Enabling 'datatypes' API controller, class: DatatypesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,122 [p:499,w:0,m:0] [MainThread] Enabling 'visualizations' API controller, class: VisualizationsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,124 [p:499,w:0,m:0] [MainThread] Enabling 'genomes' API controller, class: GenomesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,198 [p:499,w:0,m:0] [MainThread] Enabling 'job_ports' API controller, class: JobPortsAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,206 [p:499,w:0,m:0] [MainThread] Enabling 'libraries' API controller, class: LibrariesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,208 [p:499,w:0,m:0] [MainThread] Enabling 'group_users' API controller, class: GroupUsersAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,211 [p:499,w:0,m:0] [MainThread] Enabling 'tours' API controller, class: ToursController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,218 [p:499,w:0,m:0] [MainThread] Enabling 'histories' API controller, class: HistoriesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,284 [p:499,w:0,m:0] [MainThread] Enabling 'workflows' API controller, class: WorkflowsAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,299 [p:499,w:0,m:0] [MainThread] Enabling 'library_datasets' API controller, class: LibraryDatasetsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,302 [p:499,w:0,m:0] [MainThread] Enabling 'history_annotations' API controller, class: HistoryAnnotationsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,302 [p:499,w:0,m:0] [MainThread] Enabling 'history_content_annotations' API controller, class: HistoryContentAnnotationsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,303 [p:499,w:0,m:0] [MainThread] Enabling 'workflow_annotations' API controller, class: WorkflowAnnotationsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,309 [p:499,w:0,m:0] [MainThread] Enabling 'jobs' API controller, class: JobController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,311 [p:499,w:0,m:0] [MainThread] Enabling 'group_roles' API controller, class: GroupRolesAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,319 [p:499,w:0,m:0] [MainThread] Enabling 'forms' API controller, class: FormDefinitionAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,320 [p:499,w:0,m:0] [MainThread] Enabling 'trs_search' API controller, class: TrsSearchAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,387 [p:499,w:0,m:0] [MainThread] Enabling 'users' API controller, class: UserAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,390 [p:499,w:0,m:0] [MainThread] Enabling 'history_dataset_extended_metadata' API controller, class: HistoryDatasetExtendMetadataController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,390 [p:499,w:0,m:0] [MainThread] Enabling 'library_dataset_extended_metadata' API controller, class: LibraryDatasetExtendMetadataController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,391 [p:499,w:0,m:0] [MainThread] Enabling 'container_resolution' API controller, class: ContainerResolutionAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,394 [p:499,w:0,m:0] [MainThread] Enabling 'uploads' API controller, class: UploadsAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,396 [p:499,w:0,m:0] [MainThread] Enabling 'pages' API controller, class: PagesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,397 [p:499,w:0,m:0] [MainThread] Enabling 'toolshed' API controller, class: ToolShedController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,400 [p:499,w:0,m:0] [MainThread] Enabling 'tool_dependencies' API controller, class: ToolDependenciesAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,403 [p:499,w:0,m:0] [MainThread] Enabling 'tool_data' API controller, class: ToolData
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,404 [p:499,w:0,m:0] [MainThread] Enabling 'plugins' API controller, class: PluginsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,408 [p:499,w:0,m:0] [MainThread] Enabling 'library_contents' API controller, class: LibraryContentsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,413 [p:499,w:0,m:0] [MainThread] Enabling 'datasets' API controller, class: DatasetsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,415 [p:499,w:0,m:0] [MainThread] Enabling 'dynamic_tools' API controller, class: DynamicToolsController
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,654 [p:499,w:0,m:0] [MainThread] Enabling 'paste.httpexceptions' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,690 [p:499,w:0,m:0] [MainThread] Enabling 'RecursiveMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,690 [p:499,w:0,m:0] [MainThread] Enabling 'ErrorMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,696 [p:499,w:0,m:0] [MainThread] Enabling 'TransLogger' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,702 [p:499,w:0,m:0] [MainThread] Enabling 'XForwardedHostMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,702 [p:499,w:0,m:0] [MainThread] Enabling 'RequestIDMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,702 [p:499,w:0,m:0] [MainThread] Enabling 'BatchMiddleware' middleware
galaxy.webapps.galaxy.buildapp DEBUG 2021-10-15 09:08:11,709 [p:499,w:0,m:0] [MainThread] Prior to webapp return, Galaxy thread <_MainThread(MainThread, started 140052499973952)> is alive.
galaxy.webapps.galaxy.buildapp DEBUG 2021-10-15 09:08:11,710 [p:499,w:0,m:0] [MainThread] Prior to webapp return, Galaxy thread <SqliteMultithread(Thread-4, started daemon 140052303128320)> is alive.
Fri Oct 15 09:08:11 2021 - WSGI app 0 (mountpoint='') ready in 202 seconds on interpreter 0x55b1989f3dc0 pid: 499 (default app)
Fri Oct 15 09:08:11 2021 - *** uWSGI is running in multiple interpreter mode ***
Fri Oct 15 09:08:11 2021 - spawned uWSGI master process (pid: 499)
Fri Oct 15 09:08:11 2021 - spawned uWSGI worker 1 (pid: 853, cores: 4)
Fri Oct 15 09:08:11 2021 - spawned 2 offload threads for uWSGI worker 1
Fri Oct 15 09:08:11 2021 - spawned uWSGI worker 2 (pid: 856, cores: 4)
Fri Oct 15 09:08:11 2021 - spawned 2 offload threads for uWSGI worker 2
galaxy.web_stack DEBUG 2021-10-15 09:08:11,820 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method ApplicationStack.set_postfork_server_name of <galaxy.web_stack.UWSGIApplicationStack object at 0x7f605e9ed358>>
Fri Oct 15 09:08:11 2021 - *** Stats server enabled on 127.0.0.1:9191 fd: 17 ***
Fri Oct 15 09:08:12 2021 - spawned uWSGI http 1 (pid: 859)
Fri Oct 15 09:08:12 2021 - running "unix_signal:2 gracefully_kill_them_all" (master-start)...
Fri Oct 15 09:08:12 2021 - running "unix_signal:15 gracefully_kill_them_all" (master-start)...
galaxy.web_stack DEBUG 2021-10-15 09:08:11,961 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method ApplicationStack.set_postfork_server_name of <galaxy.web_stack.UWSGIApplicationStack object at 0x7f605e9ed358>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,320 [p:856,w:2,m:0] [MainThread] server_name set to: main.web.2
galaxy.web_stack DEBUG 2021-10-15 09:08:12,320 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method ToolDocumentCache.reopen_ro of <galaxy.tools.cache.ToolDocumentCache object at 0x7f605d7c6588>>
sqlitedict INFO 2021-10-15 09:08:12,322 [p:856,w:2,m:0] [MainThread] opening Sqlite table 'unnamed' in '/galaxy-central/database/tool_cache/cache.sqlite'
galaxy.web_stack DEBUG 2021-10-15 09:08:12,391 [p:853,w:1,m:0] [MainThread] server_name set to: main.web.1
galaxy.web_stack DEBUG 2021-10-15 09:08:12,393 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method ToolDocumentCache.reopen_ro of <galaxy.tools.cache.ToolDocumentCache object at 0x7f605d7c6588>>
sqlitedict INFO 2021-10-15 09:08:12,414 [p:853,w:1,m:0] [MainThread] opening Sqlite table 'unnamed' in '/galaxy-central/database/tool_cache/cache.sqlite'
galaxy.web_stack DEBUG 2021-10-15 09:08:12,626 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method JobManager.start of <galaxy.jobs.manager.JobManager object at 0x7f605cc84d68>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,680 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method DatabaseHeartbeat.start of <galaxy.model.database_heartbeat.DatabaseHeartbeat object at 0x7f605cc78898>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,802 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method UWSGIApplicationStack.start of <galaxy.web_stack.UWSGIApplicationStack object at 0x7f605e9ed358>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,804 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method GalaxyQueueWorker.bind_and_start of <GalaxyQueueWorker(Thread-1, initial daemon)>>
galaxy.queue_worker INFO 2021-10-15 09:08:12,807 [p:856,w:2,m:0] [MainThread] Binding and starting galaxy control worker for main.web.2
galaxy.web_stack DEBUG 2021-10-15 09:08:12,907 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method JobManager.start of <galaxy.jobs.manager.JobManager object at 0x7f605cc84d68>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,910 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method DatabaseHeartbeat.start of <galaxy.model.database_heartbeat.DatabaseHeartbeat object at 0x7f605cc78898>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,985 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method UWSGIApplicationStack.start of <galaxy.web_stack.UWSGIApplicationStack object at 0x7f605e9ed358>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,987 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method GalaxyQueueWorker.bind_and_start of <GalaxyQueueWorker(Thread-1, initial daemon)>>
galaxy.queue_worker INFO 2021-10-15 09:08:12,990 [p:853,w:1,m:0] [MainThread] Binding and starting galaxy control worker for main.web.1
galaxy.web_stack DEBUG 2021-10-15 09:08:13,001 [p:856,w:2,m:0] [MainThread] Calling postfork function: <function UniverseApplication.__init__.<locals>.<lambda> at 0x7f605ccdad90>
galaxy.queue_worker INFO 2021-10-15 09:08:13,006 [p:856,w:2,m:0] [MainThread] Queuing async task rebuild_toolbox_search_index for main.web.2.
galaxy.web_stack DEBUG 2021-10-15 09:08:13,318 [p:853,w:1,m:0] [MainThread] Calling postfork function: <function UniverseApplication.__init__.<locals>.<lambda> at 0x7f605ccdad90>
galaxy.queue_worker INFO 2021-10-15 09:08:13,325 [p:853,w:1,m:0] [MainThread] Queuing async task rebuild_toolbox_search_index for main.web.1.
galaxy.model.database_heartbeat DEBUG 2021-10-15 09:08:15,214 [p:856,w:2,m:0] [database_heartbeart_main.web.2.thread] main.web.2 is config watcher
galaxy.web_stack DEBUG 2021-10-15 09:08:15,615 [p:856,w:2,m:0] [MainThread] Calling postfork function: <built-in function register>
galaxy.web_stack DEBUG 2021-10-15 09:08:15,620 [p:856,w:2,m:0] [MainThread] Calling postfork function: <function postfork_setup at 0x7f605e9ebae8>
galaxy.web_stack INFO 2021-10-15 09:08:15,622 [p:856,w:2,m:0] [MainThread] Galaxy server instance 'main.web.2' is running
galaxy.queue_worker INFO 2021-10-15 09:08:15,723 [p:856,w:2,m:0] [Thread-1] Instance 'main.web.2' received 'rebuild_toolbox_search_index' task, executing now.
galaxy.tools.search DEBUG 2021-10-15 09:08:15,726 [p:856,w:2,m:0] [Thread-1] Starting to build toolbox index.
galaxy.web_stack DEBUG 2021-10-15 09:08:15,831 [p:853,w:1,m:0] [MainThread] Calling postfork function: <built-in function register>
galaxy.web_stack DEBUG 2021-10-15 09:08:15,834 [p:853,w:1,m:0] [MainThread] Calling postfork function: <function postfork_setup at 0x7f605e9ebae8>
galaxy.web_stack INFO 2021-10-15 09:08:15,883 [p:853,w:1,m:0] [MainThread] Galaxy server instance 'main.web.1' is running
Starting server in PID 499.
serving on http://127.0.0.1:8080
serving on uwsgi://127.0.0.1:4001
galaxy.util.watcher DEBUG 2021-10-15 09:08:15,907 [p:856,w:2,m:0] [database_heartbeart_main.web.2.thread] Watching for changes in directory (recursively): /galaxy-central/tool-data
galaxy.queue_worker INFO 2021-10-15 09:08:16,623 [p:853,w:1,m:0] [Thread-1] Instance 'main.web.1' received 'rebuild_toolbox_search_index' task, executing now.
galaxy.tools.search DEBUG 2021-10-15 09:08:16,639 [p:853,w:1,m:0] [Thread-1] Starting to build toolbox index.
galaxy.tools.search DEBUG 2021-10-15 09:08:17,314 [p:853,w:1,m:0] [Thread-1] Toolbox index finished (675.020 ms)
galaxy.tools.search DEBUG 2021-10-15 09:08:19,086 [p:856,w:2,m:0] [Thread-1] Toolbox index finished (3361.779 ms)

==> /home/galaxy/logs/handler1.log <==
galaxy.model.database_heartbeat DEBUG 2021-10-15 09:08:50,705 handler1 is not config watcher

==> /home/galaxy/logs/slurmctld.log <==
[2021-10-15T09:11:26.709] error: Could not open job state file /tmp/slurm/job_state: No such file or directory
[2021-10-15T09:11:26.710] error: NOTE: Trying backup state save file. Jobs may be lost!
[2021-10-15T09:11:26.710] No job state file (/tmp/slurm/job_state.old) found
hexylena commented 3 years ago

I see this in the logs

migrate.versioning.repository DEBUG 2021-11-09 12:42:13,037 [p:1430,w:0,m:0] [MainThread] Config: OrderedDict([('db_settings', OrderedDict([('repository_id', 'Galaxy'), ('version_table', 'migrate_version'), ('required_dbs', '[]')]))])
Traceback (most recent call last):
  File "lib/galaxy/webapps/galaxy/buildapp.py", line 48, in app_factory
    app = galaxy.app.UniverseApplication(global_conf=global_conf, **kwargs)
  File "lib/galaxy/app.py", line 97, in __init__
    self._configure_models(check_migrate_databases=self.config.check_migrate_databases, check_migrate_tools=check_migrate_tools, config_file=config_file)
  File "lib/galaxy/config/__init__.py", line 1226, in _configure_models
    create_or_verify_database(db_url, config_file, self.config.database_engine_options, app=self, map_install_models=combined_install_database)
  File "lib/galaxy/model/migrate/check.py", line 140, in create_or_verify_database
    raise Exception("{}. {}{}".format(expect_msg, instructions, backup_msg))
Exception: Your database has version '1' but this code expects version '170'. Please backup your database and then migrate the database schema by running 'sh manage_db.sh upgrade'.
bgruening commented 3 days ago

Thanks to @jyotipm29 we are back on track :smile: The new 24.1 image contains a lot of changes and reflects the latest developments in Galaxy. I would like to close this PR, but please feel free to reopen and rebase against the latest version.

Please give it a try:

docker run -p 8080:80 -p 8021:21 -p 4002:4002 --privileged=true -e "GALAXY_DESTINATIONS_DEFAULT=slurm_cluster_docker"   -v /tmp/galaxy-data/:/export/ quay.io/bgruening/galaxy:24.1

... or any other combination. The readme has been updated. Please add any useful tip to it.

For a list of changes, see the Changelog.