search

galaxyproject / galaxy

Data intensive science for everyone.
https://galaxyproject.org
Other
1.34k stars 978 forks source link

mySQL specific query json_extract #17371

Closed alpapan closed 5 months ago

alpapan commented 5 months ago

Describe the bug There is a mysql specific function somewhere, it doesn't exit in postgres.

tail end of log (see below)

psycopg2.errors.UndefinedFunction: function json_extract(bytea, unknown) does not exist
LINE 4: ...ry_dataset_association.history_id = 23 AND (lower(json_extra...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

Galaxy Version and/or server at which you observed the bug Galaxy Version: 23.1.5.dev0 Commit: 1fe37b099f6f7da20c2483157bb2c1d94a766c67

To Reproduce Not sure, this happened overnight when running cleanup_datasets.py via cron. It's not happening currently.

Longer log

[2024-01-29 10:14:10 +1100] [84913] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1905, in _execute_context
    self.dialect.do_execute(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedFunction: function json_extract(bytea, unknown) does not exist
LINE 4: ...ry_dataset_association.deleted = false AND (lower(json_extra...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 428, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/applications.py", line 282, in __call__
    await super().__call__(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette_context/middleware/raw_middleware.py", line 93, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/fast_app.py", line 103, in add_x_frame_options
    response = await call_next(request)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/base.py", line 84, in call_next
    raise app_exc
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/routing.py", line 241, in app
    raw_response = await run_endpoint_function(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/routing.py", line 169, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/api/history_contents.py", line 469, in index
    items = self.service.index(
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/services/history_contents.py", line 314, in index
    return self.__index_v2(trans, history_id, params, serialization_params, filter_query_params, accept)
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/services/history_contents.py", line 978, in __index_v2
    contents = self.history_contents_manager.contents(
  File "/apps/others/galaxy/lib/galaxy/managers/history_contents.py", line 125, in contents
    return self._union_of_contents(
  File "/apps/others/galaxy/lib/galaxy/managers/history_contents.py", line 259, in _union_of_contents
    contents_results = self._union_of_contents_query(container, **kwargs).all()
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2773, in all
    return self._iter().all()
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
    result = self.session.execute(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1948, in _execute_context
    self._handle_dbapi_exception(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2129, in _handle_dbapi_exception
    util.raise_(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1905, in _execute_context
    self.dialect.do_execute(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedFunction) function json_extract(bytea, unknown) does not exist
LINE 4: ...ry_dataset_association.deleted = false AND (lower(json_extra...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

[SQL: SELECT anon_1.history_id AS anon_1_history_id, anon_1.history_content_type AS anon_1_history_content_type, anon_1.id AS anon_1_id, anon_1.type_id AS anon_1_type_id, anon_1.hid AS anon_1_hid, anon_1.extension AS anon_1_extension, anon_1.dataset_id AS anon_1_dataset_id, anon_1.collection_id AS anon_1_collection_id, anon_1.name AS anon_1_name, anon_1.state AS anon_1_state, anon_1.size AS anon_1_size, anon_1.deleted AS anon_1_deleted, anon_1.purged AS anon_1_purged, anon_1.visible AS anon_1_visible, anon_1.create_time AS anon_1_create_time, anon_1.update_time AS anon_1_update_time
FROM (SELECT history_dataset_association.history_id AS history_id, %(param_1)s AS history_content_type, history_dataset_association.id AS id, %(param_2)s || %(param_3)s || history_dataset_association.id AS type_id, history_dataset_association.hid AS hid, history_dataset_association.extension AS extension, history_dataset_association.dataset_id AS dataset_id, %(param_4)s AS collection_id, history_dataset_association.name AS name, dataset.state AS state, dataset.file_size AS size, history_dataset_association.deleted AS deleted, history_dataset_association.purged AS purged, history_dataset_association.visible AS visible, history_dataset_association.create_time AS create_time, history_dataset_association.update_time AS update_time
FROM history_dataset_association JOIN dataset ON dataset.id = history_dataset_association.dataset_id
WHERE history_dataset_association.history_id = %(history_id_1)s AND history_dataset_association.deleted = false AND (lower(json_extract(history_dataset_association.metadata, %(json_extract_1)s)) LIKE '%%' || %(lower_1)s || '%%' ESCAPE '/') UNION ALL SELECT history_dataset_collection_association.history_id AS history_id, %(param_5)s AS history_content_type, history_dataset_collection_association.id AS id, %(param_6)s || %(param_7)s || history_dataset_collection_association.id AS type_id, history_dataset_collection_association.hid AS hid, %(param_8)s AS extension, %(param_9)s AS dataset_id, history_dataset_collection_association.collection_id AS collection_id, history_dataset_collection_association.name AS name, dataset_collection.populated_state AS state, %(param_10)s AS size, history_dataset_collection_association.deleted AS deleted, %(param_11)s AS purged, history_dataset_collection_association.visible AS visible, history_dataset_collection_association.create_time AS create_time, history_dataset_collection_association.update_time AS update_time
FROM history_dataset_collection_association JOIN dataset_collection ON dataset_collection.id = history_dataset_collection_association.collection_id
WHERE false) AS anon_1 ORDER BY anon_1.hid DESC
 LIMIT %(param_12)s OFFSET %(param_13)s]
[parameters: {'param_1': 'dataset', 'param_2': 'dataset', 'param_3': '-', 'param_4': None, 'history_id_1': 39, 'json_extract_1': '$.dbkey', 'lower_1': 'murcottphoenix', 'param_5': 'dataset_collection', 'param_6': 'dataset_collection', 'param_7': '-', 'param_8': None, 'param_9': None, 'param_10': None, 'param_11': False, 'param_12': 100, 'param_13': 0}]
(Background on this error at: https://sqlalche.me/e/14/f405)
[2024-01-29 10:14:26 +1100] [84913] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1905, in _execute_context
    self.dialect.do_execute(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedFunction: function json_extract(bytea, unknown) does not exist
LINE 4: ...ry_dataset_association.deleted = false AND (lower(json_extra...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 428, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/applications.py", line 282, in __call__
    await super().__call__(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette_context/middleware/raw_middleware.py", line 93, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/fast_app.py", line 103, in add_x_frame_options
    response = await call_next(request)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/base.py", line 84, in call_next
    raise app_exc
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/routing.py", line 241, in app
    raw_response = await run_endpoint_function(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/routing.py", line 169, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/api/history_contents.py", line 469, in index
    items = self.service.index(
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/services/history_contents.py", line 314, in index
    return self.__index_v2(trans, history_id, params, serialization_params, filter_query_params, accept)
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/services/history_contents.py", line 978, in __index_v2
    contents = self.history_contents_manager.contents(
  File "/apps/others/galaxy/lib/galaxy/managers/history_contents.py", line 125, in contents
    return self._union_of_contents(
  File "/apps/others/galaxy/lib/galaxy/managers/history_contents.py", line 259, in _union_of_contents
    contents_results = self._union_of_contents_query(container, **kwargs).all()
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2773, in all
    return self._iter().all()
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
    result = self.session.execute(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1948, in _execute_context
    self._handle_dbapi_exception(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2129, in _handle_dbapi_exception
    util.raise_(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1905, in _execute_context
    self.dialect.do_execute(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedFunction) function json_extract(bytea, unknown) does not exist
LINE 4: ...ry_dataset_association.deleted = false AND (lower(json_extra...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

[SQL: SELECT anon_1.history_id AS anon_1_history_id, anon_1.history_content_type AS anon_1_history_content_type, anon_1.id AS anon_1_id, anon_1.type_id AS anon_1_type_id, anon_1.hid AS anon_1_hid, anon_1.extension AS anon_1_extension, anon_1.dataset_id AS anon_1_dataset_id, anon_1.collection_id AS anon_1_collection_id, anon_1.name AS anon_1_name, anon_1.state AS anon_1_state, anon_1.size AS anon_1_size, anon_1.deleted AS anon_1_deleted, anon_1.purged AS anon_1_purged, anon_1.visible AS anon_1_visible, anon_1.create_time AS anon_1_create_time, anon_1.update_time AS anon_1_update_time
FROM (SELECT history_dataset_association.history_id AS history_id, %(param_1)s AS history_content_type, history_dataset_association.id AS id, %(param_2)s || %(param_3)s || history_dataset_association.id AS type_id, history_dataset_association.hid AS hid, history_dataset_association.extension AS extension, history_dataset_association.dataset_id AS dataset_id, %(param_4)s AS collection_id, history_dataset_association.name AS name, dataset.state AS state, dataset.file_size AS size, history_dataset_association.deleted AS deleted, history_dataset_association.purged AS purged, history_dataset_association.visible AS visible, history_dataset_association.create_time AS create_time, history_dataset_association.update_time AS update_time
FROM history_dataset_association JOIN dataset ON dataset.id = history_dataset_association.dataset_id
WHERE history_dataset_association.history_id = %(history_id_1)s AND history_dataset_association.deleted = false AND (lower(json_extract(history_dataset_association.metadata, %(json_extract_1)s)) LIKE '%%' || %(lower_1)s || '%%' ESCAPE '/') UNION ALL SELECT history_dataset_collection_association.history_id AS history_id, %(param_5)s AS history_content_type, history_dataset_collection_association.id AS id, %(param_6)s || %(param_7)s || history_dataset_collection_association.id AS type_id, history_dataset_collection_association.hid AS hid, %(param_8)s AS extension, %(param_9)s AS dataset_id, history_dataset_collection_association.collection_id AS collection_id, history_dataset_collection_association.name AS name, dataset_collection.populated_state AS state, %(param_10)s AS size, history_dataset_collection_association.deleted AS deleted, %(param_11)s AS purged, history_dataset_collection_association.visible AS visible, history_dataset_collection_association.create_time AS create_time, history_dataset_collection_association.update_time AS update_time
FROM history_dataset_collection_association JOIN dataset_collection ON dataset_collection.id = history_dataset_collection_association.collection_id
WHERE false) AS anon_1 ORDER BY anon_1.hid DESC
 LIMIT %(param_12)s OFFSET %(param_13)s]
[parameters: {'param_1': 'dataset', 'param_2': 'dataset', 'param_3': '-', 'param_4': None, 'history_id_1': 39, 'json_extract_1': '$.dbkey', 'lower_1': 'murcottphoenix', 'param_5': 'dataset_collection', 'param_6': 'dataset_collection', 'param_7': '-', 'param_8': None, 'param_9': None, 'param_10': None, 'param_11': False, 'param_12': 100, 'param_13': 0}]
(Background on this error at: https://sqlalche.me/e/14/f405)
/apps/others/galaxy/lib/galaxy/managers/notification.py:225: SAWarning: TypeDecorator DoubleEncodedJsonType() will not produce a cache key because the ``cache_ok`` attribute is not set to True.  This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions.  Set this attribute to True if this type object's state is safe to use in a cache key, or False to disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)
  result = self.sa_session.execute(stmt).fetchall()
/apps/others/galaxy/lib/galaxy/managers/notification.py:178: SAWarning: TypeDecorator DoubleEncodedJsonType() will not produce a cache key because the ``cache_ok`` attribute is not set to True.  This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions.  Set this attribute to True if this type object's state is safe to use in a cache key, or False to disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)
  result = self.sa_session.execute(stmt).fetchall()
[2024-01-29 10:27:14 +1100] [84925] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1905, in _execute_context
    self.dialect.do_execute(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedFunction: function json_extract(bytea, unknown) does not exist
LINE 4: ...ry_dataset_association.deleted = false AND (lower(json_extra...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 428, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/applications.py", line 282, in __call__
    await super().__call__(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette_context/middleware/raw_middleware.py", line 93, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/fast_app.py", line 103, in add_x_frame_options
    response = await call_next(request)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/base.py", line 84, in call_next
    raise app_exc
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/routing.py", line 241, in app
    raw_response = await run_endpoint_function(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/fastapi/routing.py", line 169, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/api/history_contents.py", line 469, in index
    items = self.service.index(
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/services/history_contents.py", line 314, in index
    return self.__index_v2(trans, history_id, params, serialization_params, filter_query_params, accept)
  File "/apps/others/galaxy/lib/galaxy/webapps/galaxy/services/history_contents.py", line 978, in __index_v2
    contents = self.history_contents_manager.contents(
  File "/apps/others/galaxy/lib/galaxy/managers/history_contents.py", line 125, in contents
    return self._union_of_contents(
  File "/apps/others/galaxy/lib/galaxy/managers/history_contents.py", line 259, in _union_of_contents
    contents_results = self._union_of_contents_query(container, **kwargs).all()
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2773, in all
    return self._iter().all()
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
    result = self.session.execute(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1948, in _execute_context
    self._handle_dbapi_exception(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2129, in _handle_dbapi_exception
    util.raise_(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1905, in _execute_context
    self.dialect.do_execute(
  File "/apps/others/galaxy/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedFunction) function json_extract(bytea, unknown) does not exist
LINE 4: ...ry_dataset_association.deleted = false AND (lower(json_extra...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

[SQL: SELECT anon_1.history_id AS anon_1_history_id, anon_1.history_content_type AS anon_1_history_content_type, anon_1.id AS anon_1_id, anon_1.type_id AS anon_1_type_id, anon_1.hid AS anon_1_hid, anon_1.extension AS anon_1_extension, anon_1.dataset_id AS anon_1_dataset_id, anon_1.collection_id AS anon_1_collection_id, anon_1.name AS anon_1_name, anon_1.state AS anon_1_state, anon_1.size AS anon_1_size, anon_1.deleted AS anon_1_deleted, anon_1.purged AS anon_1_purged, anon_1.visible AS anon_1_visible, anon_1.create_time AS anon_1_create_time, anon_1.update_time AS anon_1_update_time
FROM (SELECT history_dataset_association.history_id AS history_id, %(param_1)s AS history_content_type, history_dataset_association.id AS id, %(param_2)s || %(param_3)s || history_dataset_association.id AS type_id, history_dataset_association.hid AS hid, history_dataset_association.extension AS extension, history_dataset_association.dataset_id AS dataset_id, %(param_4)s AS collection_id, history_dataset_association.name AS name, dataset.state AS state, dataset.file_size AS size, history_dataset_association.deleted AS deleted, history_dataset_association.purged AS purged, history_dataset_association.visible AS visible, history_dataset_association.create_time AS create_time, history_dataset_association.update_time AS update_time
FROM history_dataset_association JOIN dataset ON dataset.id = history_dataset_association.dataset_id

and then continues (i didn't copy paste whole log as it was not spit out on a file)

mvdbeek commented 5 months ago

@ahmedhamidawan do you have a moment to look into this ? It's https://github.com/galaxyproject/galaxy/pull/15505#discussion_r1329271096

alpapan commented 5 months ago

Oh ok i found when it happens: when search the history with histories/view_multiple. i used this search string extension:cram genome_build:murcott

err log attached galaxy.err.log

ahmedhamidawan commented 5 months ago

Ah I see, thank you, this will help reproducing the error. Working on a solution for this

alpapan commented 5 months ago

postgres logs attached. me needs to go to bed now! postgresql-12-genomics.err.log

ahmedhamidawan commented 5 months ago

Fixed in #17384