search

galaxyproject / galaxy

Data intensive science for everyone.
https://galaxyproject.org
Other
1.41k stars 1.01k forks source link

Cannot import RO-Crate of an Invocation using Collection as input #18927

Open patrick-austin opened 1 month ago

patrick-austin commented 1 month ago

Describe the bug When attempting to import an exported RO-Crate for a Workflow Invocation that uses a Dataset List Collection as input, an error is returned to the UI: History import failed.

Galaxy Version and/or server at which you observed the bug Galaxy Version: 24.1.1 Commit: 940a87c8bf248c0d1cecd61b2e19fd37ce097777

Also tried on usegalaxy.eu and got the same error. Also tried on 23.2, and the import did not succeed but also did not error (job stays running indefinitely).

Browser and Operating System Operating System: Windows Browser: Firefox

To Reproduce Steps to reproduce the behavior:

  1. Import an invocation with Dataset List Collection as input Invocation-of-Input-Dataset-Collection-Test-at-2024-10-02T170508077452.rocrate.zip

Since this will fail, I also provide the simplest example workflow I could think of (used in the above): Galaxy-Workflow-Input_Dataset_Collection_Test.txt This should be a .ga but had to rename to .txt to allow upload to GitHub.

Expected behavior Import to succeed.

Screenshot image

Additional context Ultimately, the error seems to be:

CircularDependencyError
Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70dc3fd280>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70dc3fd280>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70dc3fd280>, delete=False))

We found this in our log files, wrapped in a SQLAlchemy rollback error. I don't have the knowledge of Galaxy server to dig any further into the cause, but we did a lot of trial and error on our instance and the presence of the Collection as input seemed to be the common link, and is referenced in the error message above.

bgruening commented 1 month ago

@patrick-austin do you have the entire stacktrace? Thanks for the detailed description!

patrick-austin commented 1 month ago

After some more experimentation it looks like this might only be a problem when using the "Send results to a new history" option. Maybe there's a problem in how the metadata is formatted at export/import time for collections that are sent to a new history?

For a "real" example we had, not sending to new history resulted in an RO-Crate that could be re-imported. But for the example I gave above it seems to fail with a different error, this time (see bottom). An even more simple example that doesn't even run a tool does work though:

Invocation without sending to new history (should work): Invocation-of-Collection-input-only-at-2024-10-07T101050454964.rocrate.zip

Invocaion with sending to new history (doesn't work for me): Invocation-of-Collection-input-only-at-2024-10-07T102428277441.rocrate.zip

Apologies for not including the full stack for the original error, it seemed to be a lot of SQLAlchemy rolling back for the same underlying error but perhaps there is something useful in there:

galaxy.workflow.run_request INFO 2024-10-04 15:19:44,279 [pN:main.1,p:3238871,tN:AnyIO worker thread] Creating a step_state for step.id 9295
galaxy.workflow.run_request INFO 2024-10-04 15:19:44,280 [pN:main.1,p:3238871,tN:AnyIO worker thread] Creating a step_state for step.id 9296
galaxy.web_stack.handlers INFO 2024-10-04 15:19:44,281 [pN:main.1,p:3238871,tN:AnyIO worker thread] (WorkflowInvocation[unflushed]) Handler '_default_' assigned using 'HANDLER_ASSIGNMENT_METHODS.DB_SKIP_LOCKED' assignment method
multipart.multipart DEBUG 2024-10-04 15:20:15,007 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_begin with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_field with data[60:79]
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_value with data[81:193]
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_end with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_field with data[195:207]
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_value with data[209:224]
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_end with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_headers_finished with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,010 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_data with data[228:208896]
multipart.multipart DEBUG 2024-10-04 15:20:15,011 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_data with data[0:56404]
multipart.multipart DEBUG 2024-10-04 15:20:15,011 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_end with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_begin with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_field with data[56466:56485]
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_value with data[56487:56519]
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_end with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_headers_finished with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_end with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_end with no data
galaxy.web_stack.handlers INFO 2024-10-04 15:20:15,052 [pN:main.1,p:3238871,tN:AnyIO worker thread] Flushed transaction for Job[id=17137,tool_id=__IMPORT_HISTORY__] (10.270 ms)
galaxy.web_stack.handlers INFO 2024-10-04 15:20:15,052 [pN:main.1,p:3238871,tN:AnyIO worker thread] (Job[id=17137,tool_id=__IMPORT_HISTORY__]) Handler '_default_' assigned using 'HANDLER_ASSIGNMENT_METHODS.DB_SKIP_LOCKED' assignment method
galaxy.jobs.handler DEBUG 2024-10-04 15:20:15,446 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] Grabbed Job(s): 17137
galaxy.jobs.mapper DEBUG 2024-10-04 15:20:15,469 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] (17137) Mapped job to destination id: local_dest
galaxy.jobs.handler DEBUG 2024-10-04 15:20:15,479 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] (17137) Dispatching to local_runner runner
galaxy.jobs DEBUG 2024-10-04 15:20:15,492 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] (17137) Persisting job destination (destination id: local_dest)
galaxy.jobs DEBUG 2024-10-04 15:20:15,508 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] (17137) Working directory for job is: /mnt/nfs/data/job_work_dir/017/17137
galaxy.jobs.runners DEBUG 2024-10-04 15:20:15,517 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] Job [17137] queued (38.066 ms)
galaxy.jobs.handler INFO 2024-10-04 15:20:15,522 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] (17137) Job dispatched
galaxy.jobs DEBUG 2024-10-04 15:20:15,591 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] Job wrapper for Job [17137] prepared (60.970 ms)
galaxy.jobs.command_factory INFO 2024-10-04 15:20:15,638 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] Built script [/mnt/nfs/data/job_work_dir/017/17137/tool_script.sh] for tool command [python '/srv/galaxy/server/lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py' 'L3Nydi9nYWxheHkvdmFyL3RtcC90bXBlOXhzN2JkZS56aXA=' 'L3Nydi9nYWxheHkvdmFyL3RtcC90bXAxcl8xZ3RscQ==' --file --file-sources '/mnt/nfs/data/job_work_dir/017/17137/configs/tmpx548bpin' --encoded]
galaxy.jobs.runners DEBUG 2024-10-04 15:20:15,708 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] (17137) command is: mkdir -p working outputs configs
if [ -d _working ]; then
    rm -rf working/ outputs/ configs/; cp -R _working working; cp -R _outputs outputs; cp -R _configs configs
else
    cp -R working _working; cp -R outputs _outputs; cp -R configs _configs
fi
cd working; /bin/bash /mnt/nfs/data/job_work_dir/017/17137/tool_script.sh > '../outputs/tool_stdout' 2> '../outputs/tool_stderr'; return_code=$?; echo $return_code > /mnt/nfs/data/job_work_dir/017/17137/galaxy_17137.ec; cd '/mnt/nfs/data/job_work_dir/017/17137'; 
[ "$GALAXY_VIRTUAL_ENV" = "None" ] && GALAXY_VIRTUAL_ENV="$_GALAXY_VIRTUAL_ENV"; _galaxy_setup_environment True; python metadata/set.py; sh -c "exit $return_code"
galaxy.jobs.runners.local DEBUG 2024-10-04 15:20:15,763 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] (17137) executing job script: /mnt/nfs/data/job_work_dir/017/17137/galaxy_17137.sh
galaxy.jobs.runners.util.process_groups DEBUG 2024-10-04 15:20:24,081 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] check_pg(): No process found in process group 3827210
galaxy.jobs.runners.local DEBUG 2024-10-04 15:20:24,085 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] execution finished: /mnt/nfs/data/job_work_dir/017/17137/galaxy_17137.sh
galaxy.jobs ERROR 2024-10-04 15:20:24,219 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] exec_after_process hook failed for job 17137
Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 81, in cleanup_after_job
    model_store.perform_import(new_history, job=job, new_history=True)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 418, in perform_import
    self._import_jobs(object_import_tracker, history)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1224, in _import_jobs
    self._flush()
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1330, in _flush
    self.sa_session.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 597, in commit
    return self._proxied.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2017, in commit
    trans.commit(_to_root=True)
  File "<string>", line 2, in commit
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1302, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl
    self.session.flush()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4341, in flush
    self._flush(objects)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4477, in _flush
    transaction.rollback(_capture_exception=True)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush
    flush_context.execute()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 457, in execute
    for subset in topological.sort_as_subsets(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/topological.py", line 47, in sort_as_subsets
    raise CircularDependencyError(
sqlalchemy.exc.CircularDependencyError: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2039, in finish
    task_wrapper = self.tool.exec_after_process(
  File "/srv/galaxy/server/lib/galaxy/tools/__init__.py", line 3100, in exec_after_process
    JobImportHistoryArchiveWrapper(self.app, job.id).cleanup_after_job()
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 87, in cleanup_after_job
    jiha.job.tool_stderr += f"Error cleaning up history import job: {e}"
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 566, in __get__
    return self.impl.get(state, dict_)  # type: ignore[no-any-return]
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1121, in _fire_loader_callables
    return self.callable_(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 927, in _load_for_state
    primary_key_identity = self._get_ident_for_use_get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 989, in _get_ident_for_use_get
    return [
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 990, in <listcomp>
    get_attr(state, dict_, self._equated_columns[pk], passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3581, in _get_state_attr_by_column
    return state.manager[prop.key].impl.get(state, dict_, passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1116, in _fire_loader_callables
    return state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)
galaxy.jobs.runners ERROR 2024-10-04 15:20:24,264 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] (17137/3827210) Job wrapper finish method failed
Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 81, in cleanup_after_job
    model_store.perform_import(new_history, job=job, new_history=True)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 418, in perform_import
    self._import_jobs(object_import_tracker, history)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1224, in _import_jobs
    self._flush()
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1330, in _flush
    self.sa_session.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 597, in commit
    return self._proxied.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2017, in commit
    trans.commit(_to_root=True)
  File "<string>", line 2, in commit
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1302, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl
    self.session.flush()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4341, in flush
    self._flush(objects)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4477, in _flush
    transaction.rollback(_capture_exception=True)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush
    flush_context.execute()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 457, in execute
    for subset in topological.sort_as_subsets(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/topological.py", line 47, in sort_as_subsets
    raise CircularDependencyError(
sqlalchemy.exc.CircularDependencyError: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2039, in finish
    task_wrapper = self.tool.exec_after_process(
  File "/srv/galaxy/server/lib/galaxy/tools/__init__.py", line 3100, in exec_after_process
    JobImportHistoryArchiveWrapper(self.app, job.id).cleanup_after_job()
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 87, in cleanup_after_job
    jiha.job.tool_stderr += f"Error cleaning up history import job: {e}"
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 566, in __get__
    return self.impl.get(state, dict_)  # type: ignore[no-any-return]
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1121, in _fire_loader_callables
    return self.callable_(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 927, in _load_for_state
    primary_key_identity = self._get_ident_for_use_get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 989, in _get_ident_for_use_get
    return [
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 990, in <listcomp>
    get_attr(state, dict_, self._equated_columns[pk], passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3581, in _get_state_attr_by_column
    return state.manager[prop.key].impl.get(state, dict_, passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1116, in _fire_loader_callables
    return state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 676, in _finish_or_resubmit_job
    job_wrapper.finish(
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2044, in finish
    return fail("exec_after_process hook failed", exception=e)
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1849, in fail
    return self.fail(
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1396, in fail
    job = self.get_job()
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1188, in get_job
    job = self.sa_session.get(Job, self.job_id)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 1060, in get
    return self._proxied.get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3682, in get
    return self._get_impl(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3826, in _get_impl
    instance = self._identity_lookup(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2999, in _identity_lookup
    return_value = loading.get_from_identity(self, mapper, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 478, in get_from_identity
    state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)
galaxy.jobs.runners ERROR 2024-10-04 15:20:24,327 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] (17137) Unhandled exception calling queue_job
Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 81, in cleanup_after_job
    model_store.perform_import(new_history, job=job, new_history=True)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 418, in perform_import
    self._import_jobs(object_import_tracker, history)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1224, in _import_jobs
    self._flush()
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1330, in _flush
    self.sa_session.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 597, in commit
    return self._proxied.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2017, in commit
    trans.commit(_to_root=True)
  File "<string>", line 2, in commit
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1302, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl
    self.session.flush()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4341, in flush
    self._flush(objects)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4477, in _flush
    transaction.rollback(_capture_exception=True)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush
    flush_context.execute()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 457, in execute
    for subset in topological.sort_as_subsets(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/topological.py", line 47, in sort_as_subsets
    raise CircularDependencyError(
sqlalchemy.exc.CircularDependencyError: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2039, in finish
    task_wrapper = self.tool.exec_after_process(
  File "/srv/galaxy/server/lib/galaxy/tools/__init__.py", line 3100, in exec_after_process
    JobImportHistoryArchiveWrapper(self.app, job.id).cleanup_after_job()
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 87, in cleanup_after_job
    jiha.job.tool_stderr += f"Error cleaning up history import job: {e}"
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 566, in __get__
    return self.impl.get(state, dict_)  # type: ignore[no-any-return]
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1121, in _fire_loader_callables
    return self.callable_(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 927, in _load_for_state
    primary_key_identity = self._get_ident_for_use_get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 989, in _get_ident_for_use_get
    return [
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 990, in <listcomp>
    get_attr(state, dict_, self._equated_columns[pk], passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3581, in _get_state_attr_by_column
    return state.manager[prop.key].impl.get(state, dict_, passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1116, in _fire_loader_callables
    return state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 676, in _finish_or_resubmit_job
    job_wrapper.finish(
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2044, in finish
    return fail("exec_after_process hook failed", exception=e)
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1849, in fail
    return self.fail(
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1396, in fail
    job = self.get_job()
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1188, in get_job
    job = self.sa_session.get(Job, self.job_id)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 1060, in get
    return self._proxied.get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3682, in get
    return self._get_impl(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3826, in _get_impl
    instance = self._identity_lookup(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2999, in _identity_lookup
    return_value = loading.get_from_identity(self, mapper, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 478, in get_from_identity
    state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 174, in run_next
    method(arg)
  File "/srv/galaxy/server/lib/galaxy/jobs/runners/local.py", line 155, in queue_job
    self._finish_or_resubmit_job(job_state, stdout, stderr, job_id=job_id)
  File "/srv/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 686, in _finish_or_resubmit_job
    job_wrapper.fail("Unable to finish job", exception=True)
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1396, in fail
    job = self.get_job()
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1188, in get_job
    job = self.sa_session.get(Job, self.job_id)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 1060, in get
    return self._proxied.get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3682, in get
    return self._get_impl(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3826, in _get_impl
    instance = self._identity_lookup(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2999, in _identity_lookup
    return_value = loading.get_from_identity(self, mapper, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 478, in get_from_identity
    state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)

As for trying to do a dataset extraction, re-importing the invocation, e.g.: Invocation-of-Input-Dataset-Collection-Test-at-2024-10-07T084853539674.rocrate.zip

fails with a different error:

galaxy.web_stack.handlers INFO 2024-10-07 11:10:12,231 [pN:main.1,p:3238871,tN:AnyIO worker thread] Flushed transaction for Job[id=17505,tool_id=__IMPORT_HISTORY__] (10.455 ms)
galaxy.web_stack.handlers INFO 2024-10-07 11:10:12,231 [pN:main.1,p:3238871,tN:AnyIO worker thread] (Job[id=17505,tool_id=__IMPORT_HISTORY__]) Handler '_default_' assigned using 'HANDLER_ASSIGNMENT_METHODS.DB_SKIP_LOCKED' assignment method
galaxy.jobs.handler DEBUG 2024-10-07 11:10:12,477 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] Grabbed Job(s): 17505
galaxy.jobs.mapper DEBUG 2024-10-07 11:10:12,500 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] (17505) Mapped job to destination id: local_dest
galaxy.jobs.handler DEBUG 2024-10-07 11:10:12,510 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] (17505) Dispatching to local_runner runner
galaxy.jobs DEBUG 2024-10-07 11:10:12,524 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] (17505) Persisting job destination (destination id: local_dest)
galaxy.jobs DEBUG 2024-10-07 11:10:12,538 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] (17505) Working directory for job is: /mnt/nfs/data/job_work_dir/017/17505
galaxy.jobs.runners DEBUG 2024-10-07 11:10:12,545 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] Job [17505] queued (34.395 ms)
galaxy.jobs.handler INFO 2024-10-07 11:10:12,549 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] (17505) Job dispatched
galaxy.jobs DEBUG 2024-10-07 11:10:12,616 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] Job wrapper for Job [17505] prepared (58.727 ms)
galaxy.jobs.command_factory INFO 2024-10-07 11:10:12,656 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] Built script [/mnt/nfs/data/job_work_dir/017/17505/tool_script.sh] for tool command [python '/srv/galaxy/server/lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py' 'L3Nydi9nYWxheHkvdmFyL3RtcC90bXB4cDh1c3JrZi56aXA=' 'L3Nydi9nYWxheHkvdmFyL3RtcC90bXBpYzd5cW8zZg==' --file --file-sources '/mnt/nfs/data/job_work_dir/017/17505/configs/tmpfahhef6b' --encoded]
galaxy.jobs.runners DEBUG 2024-10-07 11:10:12,720 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] (17505) command is: mkdir -p working outputs configs
if [ -d _working ]; then
    rm -rf working/ outputs/ configs/; cp -R _working working; cp -R _outputs outputs; cp -R _configs configs
else
    cp -R working _working; cp -R outputs _outputs; cp -R configs _configs
fi
cd working; /bin/bash /mnt/nfs/data/job_work_dir/017/17505/tool_script.sh > '../outputs/tool_stdout' 2> '../outputs/tool_stderr'; return_code=$?; echo $return_code > /mnt/nfs/data/job_work_dir/017/17505/galaxy_17505.ec; cd '/mnt/nfs/data/job_work_dir/017/17505'; 
[ "$GALAXY_VIRTUAL_ENV" = "None" ] && GALAXY_VIRTUAL_ENV="$_GALAXY_VIRTUAL_ENV"; _galaxy_setup_environment True; python metadata/set.py; sh -c "exit $return_code"
galaxy.jobs.runners.local DEBUG 2024-10-07 11:10:12,762 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] (17505) executing job script: /mnt/nfs/data/job_work_dir/017/17505/galaxy_17505.sh
galaxy.jobs.runners.util.process_groups DEBUG 2024-10-07 11:10:21,271 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] check_pg(): No process found in process group 380764
galaxy.jobs.runners.local DEBUG 2024-10-07 11:10:21,273 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] execution finished: /mnt/nfs/data/job_work_dir/017/17505/galaxy_17505.sh
/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py:89: SAWarning: Object of type <WorkflowInvocationStep> not in session, add operation along 'WorkflowInvocation.steps' will not proceed
  self.sa_session.commit()
galaxy.jobs ERROR 2024-10-07 11:10:21,493 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] exec_after_process hook failed for job 17505
Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2039, in finish
    task_wrapper = self.tool.exec_after_process(
  File "/srv/galaxy/server/lib/galaxy/tools/__init__.py", line 3100, in exec_after_process
    JobImportHistoryArchiveWrapper(self.app, job.id).cleanup_after_job()
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 81, in cleanup_after_job
    model_store.perform_import(new_history, job=job, new_history=True)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 420, in perform_import
    self._import_workflow_invocations(object_import_tracker, history)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1084, in _import_workflow_invocations
    job = object_import_tracker.jobs_by_key[step_attrs["job"][object_key]]
KeyError: '040106f9b0b0dbd5'