search

galaxyproject / galaxy

Data intensive science for everyone.
https://galaxyproject.org
Other
1.41k stars 1.01k forks source link

db migration fail upgrade 24.1 from a5da8acfc5 to 2a7954c51b #18877

Closed alpapan closed 1 month ago

alpapan commented 1 month ago

Galaxy Version and/or server at which you observed the bug Galaxy Version: 24.1 Commit: 2a7954c51bcc2fb6a33879fe157735fb68198f83 upgrade to 2a7954c51b

Browser and Operating System Ubuntu 20.04.6 LTS psql 12

Describe the bug

I noticed that /admin/users (roles and groups) hangs with my current commit even though the relevant API call returns immediately (and no errors in log even with database logging turned on), as per issue https://github.com/galaxyproject/galaxy/issues/18878#issue-2544336463

So I did a git pull on my release_24.1 branch to check if that was an update issue

but the upgrade failed

Issue 1 and solution i did

$ ./manage_db.sh upgrade
Activating virtualenv at .venv
INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
INFO:alembic.runtime.migration:Will assume transactional DDL.
INFO:alembic.runtime.migration:Running upgrade c63848676caf -> 04288b6a5b25, make dataset uuids unique
Traceback (most recent call last):
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedFunction: function gen_random_uuid() does not exist
LINE 3:         SET uuid=REPLACE(gen_random_uuid()::text, '-', '')
                                 ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./scripts/db.py", line 37, in <module>
    main()
  File "./scripts/db.py", line 33, in main
    args.func(args)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/base.py", line 265, in upgrade
    self._exec_command("upgrade", args)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/base.py", line 289, in _exec_command
    getattr(dbscript, command)(args)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/base.py", line 202, in upgrade
    self._upgrade_to_head(args.sql)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/dbscript.py", line 88, in _upgrade_to_head
    self._upgrade_to_revision("gxy@head", is_sql_mode)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/base.py", line 234, in _upgrade_to_revision
    command.upgrade(self.alembic_config, rev, is_sql_mode)
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/command.py", line 403, in upgrade
    script.run_env()
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/alembic/env.py", line 146, in <module>
    run_migrations_online()
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/alembic/env.py", line 40, in run_migrations_online
    _configure_and_run_migrations_online(url)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/alembic/env.py", line 123, in _configure_and_run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/runtime/environment.py", line 948, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
    step.migration_fn(**kw)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/alembic/versions_gxy/04288b6a5b25_make_dataset_uuids_unique.py", line 70, in upgrade
    _randomize_uuids_for_purged_datasets_with_duplicated_uuids(connection)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/alembic/versions_gxy/04288b6a5b25_make_dataset_uuids_unique.py", line 225, in _randomize_uuids_for_purged_datasets_with_duplicated_uuids
    connection.execute(updated_purged_uuids)
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
    return meth(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
    ret = self._execute_context(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2353, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedFunction) function gen_random_uuid() does not exist
LINE 3:         SET uuid=REPLACE(gen_random_uuid()::text, '-', '')
                                 ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

[SQL:
        UPDATE dataset
        SET uuid=REPLACE(gen_random_uuid()::text, '-', '')
        WHERE uuid IN (SELECT uuid FROM temp_duplicate_datasets_purged) AND purged = true
        ]
(Background on this error at: https://sqlalche.me/e/20/f405)

solution that worked

# psql into db
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE OR REPLACE FUNCTION gen_random_uuid()
RETURNS uuid
AS $$
BEGIN
  RETURN uuid_generate_v4();
END;
$$ LANGUAGE plpgsql;

#bash
 ./manage_db.sh upgrade
Activating virtualenv at .venv
INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
INFO:alembic.runtime.migration:Will assume transactional DDL.
INFO:alembic.runtime.migration:Running upgrade c63848676caf -> 04288b6a5b25, make dataset uuids unique
DEBUG:alembic.runtime.migration:update c63848676caf to 04288b6a5b25
INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
INFO:alembic.runtime.migration:Will assume transactional DDL.

issue 2, i have not fixed this

./manage_toolshed_db.sh upgrade
Activating virtualenv at .venv
INFO:alembic.runtime.migration:Context impl SQLiteImpl.
INFO:alembic.runtime.migration:Will assume non-transactional DDL.
INFO:alembic.runtime.migration:Running upgrade 969bbf7bcc29 -> 1b5bf427db25, add non-nullable column deleted to API keys
Traceback (most recent call last):
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: api_keys

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./scripts/toolshed_db.py", line 37, in <module>
    main()
  File "./scripts/toolshed_db.py", line 33, in main
    args.func(args)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/base.py", line 265, in upgrade
    self._exec_command("upgrade", args)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/base.py", line 289, in _exec_command
    getattr(dbscript, command)(args)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/base.py", line 202, in upgrade
    self._upgrade_to_head(args.sql)
  File "/mnt/lazebnik/home/galaxy/lib/tool_shed/webapp/model/migrations/dbscript.py", line 61, in _upgrade_to_head
    self._upgrade_to_revision("head", is_sql_mode)
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/base.py", line 234, in _upgrade_to_revision
    command.upgrade(self.alembic_config, rev, is_sql_mode)
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/command.py", line 403, in upgrade
    script.run_env()
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/mnt/lazebnik/home/galaxy/lib/tool_shed/webapp/model/migrations/alembic/env.py", line 88, in <module>
    run_migrations_online()
  File "/mnt/lazebnik/home/galaxy/lib/tool_shed/webapp/model/migrations/alembic/env.py", line 30, in run_migrations_online
    _configure_and_run_migrations_online(url)
  File "/mnt/lazebnik/home/galaxy/lib/tool_shed/webapp/model/migrations/alembic/env.py", line 69, in _configure_and_run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/runtime/environment.py", line 948, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
    step.migration_fn(**kw)
  File "/mnt/lazebnik/home/galaxy/lib/tool_shed/webapp/model/migrations/alembic/versions/1b5bf427db25_add_non_nullable_column_deleted_to_api_.py", line 44, in upgrade
    add_column(table_name, Column(column_name, Boolean(), nullable=False, index=True, server_default=false()))
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/util.py", line 310, in add_column
    AddColumn(table_name, column).run()
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/util.py", line 29, in run
    return self.execute()
  File "/mnt/lazebnik/home/galaxy/lib/galaxy/model/migrations/util.py", line 167, in execute
    op.add_column(self.table_name, self.column)
  File "<string>", line 8, in add_column
  File "<string>", line 3, in add_column
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/operations/ops.py", line 2142, in add_column
    return operations.invoke(op)
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/operations/base.py", line 445, in invoke
    return fn(self, operation)
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/operations/toimpl.py", line 171, in add_column
    operations.impl.add_column(table_name, column, schema=schema, **kw)
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/ddl/impl.py", line 334, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/alembic/ddl/impl.py", line 207, in _exec
    return conn.execute(construct, multiparams)
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
    return meth(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 180, in _execute_on_connection
    return connection._execute_ddl(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1529, in _execute_ddl
    ret = self._execute_context(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2353, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/mnt/lazebnik/home/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: api_keys
[SQL: ALTER TABLE api_keys ADD COLUMN deleted BOOLEAN DEFAULT (0) NOT NULL]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

The api_keys exists in the database and it already has deleted

\d api_keys
                                         Table "public.api_keys"
   Column    |            Type             | Collation | Nullable |               Default
-------------+-----------------------------+-----------+----------+--------------------------------------
 id          | integer                     |           | not null | nextval('api_keys_id_seq'::regclass)
 create_time | timestamp without time zone |           |          |
 user_id     | integer                     |           |          |
 key         | character varying(32)       |           |          |
 deleted     | boolean                     |           | not null | false
Indexes:
    "api_keys_pkey" PRIMARY KEY, btree (id)
    "ix_api_keys_key" UNIQUE, btree (key)
    "ix_api_keys_deleted" btree (deleted)
    "ix_api_keys_user_id" btree (user_id)
Foreign-key constraints:
    "api_keys_user_id_fkey" FOREIGN KEY (user_id) REFERENCES galaxy_user(id)

./manage_toolshed_db.sh version
Activating virtualenv at .venv
1b5bf427db25 (head)
galaxy@lazebnik:~$ ./manage_toolshed_db.sh dbversion
Activating virtualenv at .venv
INFO:alembic.runtime.migration:Context impl SQLiteImpl.
INFO:alembic.runtime.migration:Will assume non-transactional DDL.
969bbf7bcc29

FWIW my git pull log

git pull
remote: Enumerating objects: 2116, done.
remote: Counting objects: 100% (1778/1778), done.
remote: Compressing objects: 100% (369/369), done.
remote: Total 2116 (delta 1465), reused 1667 (delta 1408), pack-reused 338 (from 1)
Receiving objects: 100% (2116/2116), 1.35 MiB | 1.28 MiB/s, done.
Resolving deltas: 100% (1565/1565), completed with 292 local objects.
From https://github.com/galaxyproject/galaxy
   a5da8acfc5..2a7954c51b  release_24.1                                   -> origin/release_24.1
 * [new branch]            dependabot/npm_and_yarn/client/dompurify-3.1.3 -> origin/dependabot/npm_and_yarn/client/dompurify-3.1.3
   0d4cb1091f..d5f8feb3ae  dev                                            -> origin/dev
   5b4d8a1688..3e17d8e75d  master                                         -> origin/master
   6df30d6cdf..99b3d29e69  release_20.05                                  -> origin/release_20.05
   9d31330c4c..13ab9430b3  release_20.09                                  -> origin/release_20.09
   2b79cf2aea..b1da9d832e  release_21.01                                  -> origin/release_21.01
   6b9f0902b8..9a9c4a8eb8  release_21.05                                  -> origin/release_21.05
   72bfa2b76d..0a039981bb  release_21.09                                  -> origin/release_21.09
   a03856c8e5..101d4a6d41  release_22.01                                  -> origin/release_22.01
   569d653fa0..3e511c1b2b  release_22.05                                  -> origin/release_22.05
   2384c89f15..262d48a711  release_23.0                                   -> origin/release_23.0
   60d063a179..80c55eb659  release_23.1                                   -> origin/release_23.1
   ef4e32f443..4a845a4b46  release_23.2                                   -> origin/release_23.2
   b1b905120f..0642b5940f  release_24.0                                   -> origin/release_24.0
 + 656633ede3...2fef7e339e usegalaxy                                      -> origin/usegalaxy  (forced update)
Updating a5da8acfc5..2a7954c51b
Fast-forward
 client/src/api/datasets.ts                                                                 |   3 +
 client/src/api/index.ts                                                                    |  21 ++++++
 client/src/api/schema/schema.ts                                                            |  37 +++++++---
 client/src/components/Form/Elements/FormData/FormData.vue                                  |  15 ++++-
 client/src/components/History/CurrentCollection/CollectionPanel.vue                        |   2 +
 client/src/components/History/CurrentHistory/HistoryPanel.vue                              | 113 +++----------------------------
 client/src/components/History/Multiple/MultipleViewList.vue                                |  73 ++------------------
 client/src/components/History/SwitchToHistoryLink.test.ts                                  |  46 ++++++++++---
 client/src/components/History/SwitchToHistoryLink.vue                                      |  22 ++++--
 client/src/components/Libraries/LibraryEditField.vue                                       |   7 +-
 client/src/components/Workflow/Editor/NodeOutput.vue                                       |   2 +-
 client/src/components/Workflow/Editor/composables/useUniqueLabelError.ts                   |   4 +-
 client/src/components/Workflow/Editor/modules/terminals.test.ts                            |  10 +++
 client/src/components/Workflow/Editor/modules/terminals.ts                                 |   3 +
 client/src/components/Workflow/Editor/test-data/parameter_steps.json                       |  64 ++++++++++++++++++
 client/src/components/Workflow/WorkflowFilters.js                                          |   7 ++
 client/src/components/Workflow/WorkflowList.vue                                            |  22 ++----
 client/src/composables/historyDragDrop.ts                                                  | 190 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 client/src/stores/historyStore.ts                                                          |  10 ++-
 client/src/utils/setDrag.ts                                                                |  29 ++++++++
 config/plugins/visualizations/editor/templates/editor.mako                                 |   2 +-
 lib/galaxy/app.py                                                                          |   8 +--
 lib/galaxy/authnz/custos_authnz.py                                                         |   3 +
 lib/galaxy/authnz/managers.py                                                              |   2 +
 lib/galaxy/authnz/psa_authnz.py                                                            |   3 +-
 lib/galaxy/celery/__init__.py                                                              |   2 +-
 lib/galaxy/config/__init__.py                                                              |   2 +-
 lib/galaxy/config/sample/tool_conf.xml.sample                                              |   1 +
 lib/galaxy/datatypes/sniff.py                                                              |   6 +-
 lib/galaxy/job_execution/container_monitor.py                                              |   3 +-
 lib/galaxy/jobs/__init__.py                                                                |  24 ++++++-
 lib/galaxy/jobs/runners/local.py                                                           |   4 +-
 lib/galaxy/jobs/runners/pulsar.py                                                          |  11 +--
 lib/galaxy/managers/datasets.py                                                            |  14 +++-
 lib/galaxy/managers/jobs.py                                                                |   2 +-
 lib/galaxy/managers/quotas.py                                                              |   9 ++-
 lib/galaxy/managers/workflows.py                                                           |   6 ++
 lib/galaxy/model/__init__.py                                                               |  22 ++++--
 lib/galaxy/model/migrations/alembic/versions_gxy/04288b6a5b25_make_dataset_uuids_unique.py | 330 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/galaxy/model/migrations/dbscript.py                                                    |   4 +-
 lib/galaxy/model/store/__init__.py                                                         |  39 +++++++----
 lib/galaxy/model/store/discover.py                                                         |  17 +++--
 lib/galaxy/schema/help.py                                                                  |  16 ++---
 lib/galaxy/schema/schema.py                                                                |   3 +
 lib/galaxy/tools/__init__.py                                                               |  12 +++-
 lib/galaxy/tools/actions/__init__.py                                                       |  68 ++++++++++++-------
 lib/galaxy/util/__init__.py                                                                |  70 +++++++++----------
 lib/galaxy/webapps/galaxy/api/workflows.py                                                 |  20 +++++-
 lib/galaxy/webapps/galaxy/controllers/dataset.py                                           |  31 ++-------
 lib/galaxy/webapps/galaxy/services/datasets.py                                             |   6 +-
 lib/galaxy/workflow/trs_proxy.py                                                           |  18 ++++-
 lib/galaxy_test/api/test_jobs.py                                                           |  26 ++++++++
 lib/galaxy_test/api/test_workflows.py                                                      |  37 ++++++++++
 lib/galaxy_test/base/populators.py                                                         |  11 +--
 lib/tool_shed/test/base/playwrightbrowser.py                                               |   1 +
 scripts/check_and_update_purged_on_duplicated_uuid.py                                      |  64 ++++++++++++++++++
 scripts/galaxy_main.py                                                                     |   7 ++
 test/integration/objectstore/test_jobs.py                                                  |  26 ++++++--
 test/integration/resubmission_runners.py                                                   |   4 +-
 test/integration/test_extended_metadata.py                                                 |   1 +
 test/integration/test_job_resubmission.py                                                  |  31 +++++++--
 test/integration/test_quota.py                                                             |  20 ++++++
 test/integration/test_workflow_invocation.py                                               |  50 ++++++++++++++
 test/unit/app/managers/test_HDAManager.py                                                  |   2 +-
 test/unit/data/model/test_model_store.py                                                   |  62 +++++++++++++++--
 test/unit/workflows/test_trs_proxy.py                                                      |  32 +++++----
 tools/data_source/ebi_sra.xml                                                              |   2 +-
 tools/interactive/interactivetool_blobtoolkit.xml                                          |  53 +++++++++++++++
 68 files changed, 1448 insertions(+), 419 deletions(-)
 create mode 100644 client/src/composables/historyDragDrop.ts
 create mode 100644 lib/galaxy/model/migrations/alembic/versions_gxy/04288b6a5b25_make_dataset_uuids_unique.py
 create mode 100644 scripts/check_and_update_purged_on_duplicated_uuid.py
 create mode 100644 tools/interactive/interactivetool_blobtoolkit.xml
galaxy@lazebnik:~$ git status
On branch release_24.1
Your branch is up to date with 'origin/release_24.1'.

i tried run.sh after git pull but it failed due to database not migrated.

alpapan commented 1 month ago

NB: galaxy starts fine. i don't know if manage_toolkit db is needed.

providing my experience for manage_db as FYI or for educating me i did something wrong

hexylena commented 1 month ago

Your postgres is 12 or less (which I believe it still officially supported by galaxy?) but the migration makes use of a postgres 13+ function, you'll need to CREATE EXTENSION pgcrypto; to make it available and then re-run the migration.

hexylena commented 1 month ago

cc @jdavcs who was talking about this in the admin matrix channel.

jdavcs commented 1 month ago

Reproduced. Working on a fix.

jdavcs commented 1 month ago

@alpapan Correction: not reproduced (the similar error I encountered was unrelated).

It seems you are looking at two different databases: the error raised by the manage_toolshed_db.sh script is a sqlite error (sqlite3.OperationalError), whereas the database you're checking is postgresql (\d api_keys). The sqlite error you're seeing will be raised when you run the upgrade/downgrade command on a database that has not been initialized (it will contain only one table: alembic_version). Please check your configuration and let us know if that's indeed the case.

alpapan commented 1 month ago

Hello

hexylena is right on the manage_db.sh. my workaround did the same thing. I can update postgres if needed

I have a community.sqlite and a results.sqlite (these are new right?) i don't have a universe.sqlite though

  # By default, Galaxy uses a SQLite database at
  # '<data_dir>/universe.sqlite'. You may use a SQLAlchemy connection
  # string to specify an external database instead.
  # Sample default
  # 'sqlite:///<data_dir>/universe.sqlite?isolation_level=IMMEDIATE'
  # You may specify additional options that will be passed to the
  # SQLAlchemy database engine by using the prefix
  # "database_engine_option_". For some of these options, default values
  # are provided (e.g. see database_engine_option_pool_size, etc.).
  # The same applies to `install_database_connection`, for which you
  # should use the "install_database_engine_option_" prefix.
  # For more options, please check SQLAlchemy's documentation at
  # https://docs.sqlalchemy.org/en/14/core/engines.html?highlight=create_engine#sqlalchemy.create_engine
  #database_connection: null
jdavcs commented 1 month ago

You'd only need manage_toolshed_db.sh if you're running your own tool shed. community.sqlite is the default sqlite database for tool shed. universe.sqlite is the default for galaxy. They are set in your config files: galaxy.yml for galaxy and tool_shed.yml for tool shed. If a config file is missing, the sample config will be used (e.g. tool_shed.yml.sample). The config property in both cases is database_connection. results.sqlite is the Celery results backend galaxy uses by default. In other words, you only need them if that's what you specified in your configuration files.

jdavcs commented 1 month ago

Closing this. Please let us know if you run into any further issues.