Open tischi opened 10 months ago
constantinpape
commented
10 months ago In principle everything should run on Mac and Windows and there are unit tests for it that run through without issues. https://github.com/mobie/mobie-utils-python/actions/runs/7186625802/job/19572434858
Maybe these don't cover all the cases for ome-zarr conversion?! To fix this I would need a minimal self-contained reproducible example where it fails.
eschalnajmi
commented
3 weeks ago Hi! I'm facing a very similar issue right now where code works as a .py on linux but not on macOS or windows. However, the code works in mac and windows when run as .ipynb file. It seems to get stuck on the downscaling when run as a .py file containing the exact same code. Potentially jupyter notebook is adding a layer of virtualisation?
The exact error traceback:
`/Users/najmie/miniconda3/envs/mobie3/bin/python /Users/najmie/Downloads/mobie_python_project/project_creation.py
najmie@CH7H1X6L5F mobie_python_project % /Users/najmie/miniconda3/envs/mobie3/bin/python /Users/najmie/Downloads/mobie_python_project/project_creation.py
Only one channel found in data/em_20nm_z_40_145.tif - no conversion needed :)
/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/parameter.py:296: UserWarning: Parameter "dtype" with value "None" is not of type string.
warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if DownscalingWorkflow(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, target=local, dependency=DummyTask, input_path=data/em_20nm_z_40_145.tif, input_key=, scale_factors=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], halos=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], dtype=None, int_to_uint=False, metadata_format=ome.zarr, metadata_dict={"resolution": [10.0, 10.0, 10.0], "unit": "nanometer", "setup_name": "em_20nm_z_40_145"}, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key_prefix=, force_copy=False, skip_existing_levels=False, scale_offset=0) is complete
/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/parameter.py:296: UserWarning: Parameter "scale_factor" with value "(1, 2, 2)" is not of type string.
warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if WriteDownscalingMetadata(tmp_folder=tmp_dataset1_em_20nm_z_40_145, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, scale_factors=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], dependency=DownscalingLocal, metadata_format=ome.zarr, metadata_dict={"resolution": [10.0, 10.0, 10.0], "unit": "nanometer", "setup_name": "em_20nm_z_40_145"}, output_key_prefix=, scale_offset=0, prefix=downscaling) is complete
INFO: Informed scheduler that task DownscalingWorkflow_tmp_dataset1_emDummyTask_None_f01a3395a1 has status PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s3, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s4, scale_factor=(1, 2, 2), scale_prefix=s4, halo=[1, 2, 2], effective_scale_factor=[1, 16, 16], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task WriteDownscalingMetadataDownscalingLocalresolution___ome_zarr_784397d098 has status PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s2, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s3, scale_factor=(1, 2, 2), scale_prefix=s3, halo=[1, 2, 2], effective_scale_factor=[1, 8, 8], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task DownscalingLocal_tmp_dataset1_emDownscalingLocal_d657c21b11 has status PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s1, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s2, scale_factor=(1, 2, 2), scale_prefix=s2, halo=[1, 2, 2], effective_scale_factor=[1, 4, 4], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task DownscalingLocal_tmp_dataset1_emDownscalingLocal_d82adb898c has status PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s0, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s1, scale_factor=(1, 2, 2), scale_prefix=s1, halo=[1, 2, 2], effective_scale_factor=[1, 2, 2], dimension_separator=/, dependency=CopyVolumeLocal) is complete
INFO: Informed scheduler that task DownscalingLocal_tmp_dataset1_emDownscalingLocal_7fa9471af6 has status PENDING
DEBUG: Checking if CopyVolumeLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=data/em_20nm_z_40_145.tif, input_key=, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s0, prefix=initial_scale, dtype=None, int_to_uint=False, fit_to_roi=False, effective_scale_factor=[], dimension_separator=/, dependency=DummyTask) is complete
INFO: Informed scheduler that task DownscalingLocal_tmp_dataset1_emCopyVolumeLocal_452731ba01 has status PENDING
DEBUG: Checking if DummyTask() is complete
INFO: Informed scheduler that task CopyVolumeLocal_tmp_dataset1_emDummyTask_07bedbbf06 has status PENDING
INFO: Informed scheduler that task DummyTask__99914b932b has status DONE
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 7
INFO: [pid 11765] Worker Worker(salt=8065394207, workers=1, host=CH7H1X6L5F, username=najmie, pid=11765) running CopyVolumeLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=data/em_20nm_z_40_145.tif, input_key=, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s0, prefix=initial_scale, dtype=None, int_to_uint=False, fit_to_roi=False, effective_scale_factor=[], dimension_separator=/, dependency=DummyTask)
Only one channel found in data/em_20nm_z_40_145.tif - no conversion needed :)
/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/parameter.py:296: UserWarning: Parameter "dtype" with value "None" is not of type string.
warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if DownscalingWorkflow(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, target=local, dependency=DummyTask, input_path=data/em_20nm_z_40_145.tif, input_key=, scale_factors=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], halos=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], dtype=None, int_to_uint=False, metadata_format=ome.zarr, metadata_dict={"resolution": [10.0, 10.0, 10.0], "unit": "nanometer", "setup_name": "em_20nm_z_40_145"}, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key_prefix=, force_copy=False, skip_existing_levels=False, scale_offset=0) is complete
/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/parameter.py:296: UserWarning: Parameter "scale_factor" with value "(1, 2, 2)" is not of type string.
warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if WriteDownscalingMetadata(tmp_folder=tmp_dataset1_em_20nm_z_40_145, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, scale_factors=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], dependency=DownscalingLocal, metadata_format=ome.zarr, metadata_dict={"resolution": [10.0, 10.0, 10.0], "unit": "nanometer", "setup_name": "em_20nm_z_40_145"}, output_key_prefix=, scale_offset=0, prefix=downscaling) is complete
INFO: Informed scheduler that task DownscalingWorkflow_tmp_dataset1_emDummyTask_None_f01a3395a1 has status PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s3, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s4, scale_factor=(1, 2, 2), scale_prefix=s4, halo=[1, 2, 2], effective_scale_factor=[1, 16, 16], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task WriteDownscalingMetadata_DownscalingLocal_resolution___ome_zarr_784397d098 has status PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s2, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s3, scale_factor=(1, 2, 2), scale_prefix=s3, halo=[1, 2, 2], effective_scale_factor=[1, 8, 8], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task DownscalingLocal_tmp_dataset1_emDownscalingLocal___d657c21b11 has status PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s1, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s2, scale_factor=(1, 2, 2), scale_prefix=s2, halo=[1, 2, 2], effective_scale_factor=[1, 4, 4], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task DownscalingLocal_tmp_dataset1_emDownscalingLocal_d82adb898c has status PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s0, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s1, scale_factor=(1, 2, 2), scale_prefix=s1, halo=[1, 2, 2], effective_scale_factor=[1, 2, 2], dimension_separator=/, dependency=CopyVolumeLocal) is complete
INFO: Informed scheduler that task DownscalingLocal_tmp_dataset1_emDownscalingLocal_7fa9471af6 has status PENDING
DEBUG: Checking if CopyVolumeLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=data/em_20nm_z_40_145.tif, input_key=, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s0, prefix=initial_scale, dtype=None, int_to_uint=False, fit_to_roi=False, effective_scale_factor=[], dimension_separator=/, dependency=DummyTask) is complete
INFO: Informed scheduler that task DownscalingLocal_tmp_dataset1_emCopyVolumeLocal_452731ba01 has status PENDING
INFO: Informed scheduler that task CopyVolumeLocal_tmp_dataset1_emDummyTask_07bedbbf06 has status DONE
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 6
INFO: [pid 11783] Worker Worker(salt=2341089979, workers=1, host=CH7H1X6L5F, username=najmie, pid=11783) running DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s0, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s1, scale_factor=(1, 2, 2), scale_prefix=s1, halo=[1, 2, 2], effective_scale_factor=[1, 2, 2], dimension_separator=/, dependency=CopyVolumeLocal)
ERROR: [pid 11783] Worker Worker(salt=2341089979, workers=1, host=CH7H1X6L5F, username=najmie, pid=11783) failed DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s0, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s1, scale_factor=(1, 2, 2), scale_prefix=s1, halo=[1, 2, 2], effective_scale_factor=[1, 2, 2], dimension_separator=/, dependency=CopyVolumeLocal)
Traceback (most recent call last):
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/worker.py", line 210, in run
new_deps = self._run_get_new_deps()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
task_gen = self.task.run()
^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 95, in run
raise e
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 81, in run
self.run_impl()
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/downscaling/downscaling.py", line 184, in run_impl
self.submit_jobs(n_jobs, self.scale_prefix)
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 567, in submit_jobs
tasks = [pp.submit(self._submit, job_id, job_prefix) for job_id in range(n_jobs)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 567, in
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
To fix this issue, refer to the "Safe importing of main module"
section in https://docs.python.org/3/library/multiprocessing.htmlDEBUG: 1 running tasks, waiting for next task to finish INFO: Informed scheduler that task DownscalingLocal_tmp_dataset1_em__CopyVolumeLocal___452731ba01 has status FAILED DEBUG: Asking scheduler for work... DEBUG: Done DEBUG: There are no more tasks to run at this time DEBUG: There are 6 pending tasks possibly being run by other workers DEBUG: There are 6 pending tasks unique to this worker DEBUG: There are 6 pending tasks last scheduled by this worker INFO: Worker Worker(salt=2341089979, workers=1, host=CH7H1X6L5F, username=najmie, pid=11783) was stopped. Shutting down Keep-Alive thread INFO: ===== Luigi Execution Summary =====
Scheduled 7 tasks of which:
This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====
Could not add em_20nm_z_40_145 to project :( ERROR: [pid 11765] Worker Worker(salt=8065394207, workers=1, host=CH7H1X6L5F, username=najmie, pid=11765) failed CopyVolumeLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=data/em_20nm_z_40_145.tif, input_key=, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s0, prefix=initial_scale, dtype=None, int_to_uint=False, fit_to_roi=False, effective_scale_factor=[], dimension_separator=/, dependency=DummyTask) concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 547, in _submit assert os.path.exists(script_path), script_path AssertionError: tmp_dataset1_em_20nm_z_40_145/copy_volume.py """
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 81, in run
self.run_impl()
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/copy_volume/copy_volume.py", line 163, in run_impl
self.submit_jobs(n_jobs, self.prefix)
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 568, in submit_jobs
[t.result() for t in tasks]
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 568, in
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/worker.py", line 210, in run
new_deps = self._run_get_new_deps()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
task_gen = self.task.run()
^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 92, in run
self._write_log("task failed in run_impl with %s" % msg)
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 281, in _write_log
with open(log_file, 'a') as f:
^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'tmp_dataset1_em_20nm_z_40_145/copy_volume_initial_scale.log'
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task CopyVolumeLocal_tmp_dataset1_em__DummyTask___07bedbbf06 has status FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 7 pending tasks possibly being run by other workers
DEBUG: There are 7 pending tasks unique to this worker
DEBUG: There are 7 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=8065394207, workers=1, host=CH7H1X6L5F, username=najmie, pid=11765) was stopped. Shutting down Keep-Alive thread
INFO:
===== Luigi Execution Summary =====
Scheduled 8 tasks of which:
This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====
Could not add em_20nm_z_40_145 to project :( `
The code being run is in this repo: https://github.com/eschalnajmi/load_mobie_project
If anyone has figured out a way to fix this easily I would very very much appreciate it :)
@constantinpape @martinschorb
I did not manage to run the conversion to OME-Zarr on my Mac (see MoBIE mattermost channel).
This raises a general question: on which OS does the code here run?