Closed mvdbeek closed 1 year ago
I suspect the first chunk to fail, the weekly tests seem to be failing on this:
galaxy.objectstore CRITICAL 2023-01-17 18:45:59,219 [pN:main.1,p:19633,tN:LocalRunner.work_thread-0] Error copying /tmp/tmpqorr81b5/job_working_directory/000/60/working/dataset_60_files/mpa_vJan21_CHOCOPhlAnSGB_202103/mpa_vJan21_CHOCOPhlAnSGB_202103_SGB.fna to /tmp/tmpqorr81b5/files/000/dataset_60_files/mpa_vJan21_CHOCOPhlAnSGB_202103/mpa_vJan21_CHOCOPhlAnSGB_202103_SGB.fna: [Errno 28] No space left on device
galaxy.job_execution.output_collect DEBUG 2023-01-17 18:45:59,221 [pN:main.1,p:19633,tN:LocalRunner.work_thread-0] Error in collect_associated_files: [Errno 28] No space left on device
galaxy.jobs DEBUG 2023-01-17 18:45:59,222 [pN:main.1,p:19633,tN:LocalRunner.work_thread-0] (60) setting dataset 60 state to ERROR
Shut down
supervisord has terminated
There were problems with 1 test(s) - out of 1 test(s) executed. See tool_test_output.html for detailed breakdown.
data_manager_metaphlan_download (Test #1): failed
Interestingly after cleaning up the container tests continue and pass, but the runner crashes on shutdown. I think I'll just disable data_manager_metaphlan_download tests for now.
/run-all-tool-tests branch=dev
/run-all-tool-tests branch=release_22.05
Nice, so I think the difference comes from a new assertion in the tool data table branch:
Traceback (most recent call last):
File "/tmp/tmpmk87ahga/galaxy-dev/lib/galaxy/jobs/__init__.py", line 1909, in finish
self.app, inp_data, out_data, param_dict, job=job, final_job_state=final_job_state
File "/tmp/tmpmk87ahga/galaxy-dev/lib/galaxy/tools/__init__.py", line 3061, in exec_after_process
data_manager.process_result(out_data)
File "/tmp/tmpmk87ahga/galaxy-dev/lib/galaxy/tools/data_manager/manager.py", line 231, in process_result
options,
File "/tmp/tmpmk87ahga/galaxy-dev/lib/galaxy/tool_util/data/__init__.py", line 1133, in process_bundle
return _process_bundle(out_data, bundle, options, self)
File "/tmp/tmpmk87ahga/galaxy-dev/lib/galaxy/tool_util/data/__init__.py", line 1271, in _process_bundle
**data_table_value,
File "/tmp/tmpmk87ahga/galaxy-dev/lib/galaxy/tool_util/data/__init__.py", line 1332, in _process_move
assert source
AssertionError
that should not stop us from upgrading test & main, but we should look into this (@jmchilton any idea ?)
/run-all-tool-tests branch=dev