search

kirchhausenlab / incasem

Automated Segmentation of cellular substructures in Electron Microscopy
BSD 3-Clause "New" or "Revised" License
16 stars 1 forks source link

Prediction error on MacOS #12

Closed GenevieveBuckley closed 1 year ago

GenevieveBuckley commented 1 year ago

We (Ben and me) managed to get everything installed on Mac (MacOS Monterey (M1 chip) version 12.3.1 (21E258)), following the installation instructions here.

But when we tried to run the prediction command on the test dataset, we got this error:

Full error traceback (click to expand) ``` (omniboard_environment) (incasem) genevieb@Admins-MacBook-Pro 03_predict % python predict.py --run_id 1847 --name example_prediction_cell6_ER with config_prediction.yaml 'prediction.data=data_configs/example_cell6.json' 'prediction.checkpoint=../../models/pretrained_checkpoints/model_checkpoint_1847_mito_CF.pt' INFO:__main__:Attach Mongo observer INFO:__main__:['predict.py', '-C', 'no', '--name', 'example_prediction_cell6_ER', 'with', 'config_prediction.yaml', 'prediction.data=data_configs/example_cell6.json', 'prediction.checkpoint=../../models/pretrained_checkpoints/model_checkpoint_1847_mito_CF.pt', 'prediction.run_id_training=1847'] INFO:example_prediction_cell6_ER:Running command 'predict' INFO:example_prediction_cell6_ER:Started run with ID "2" Added application/json as content-type of artifact /Users/genevieb/Documents/GitHub/incasem/scripts/03_predict/data_configs/example_cell6.json. INFO:__main__:total_params=5837730 INFO:incasem.pipeline.sources.data_sources_base:Setting up Cell_6_example_roi_nickname INFO:incasem.pipeline.sources.data_sources_semantic:No mask given, add dummy mask of all 1s. INFO:incasem.pipeline.sources.data_sources_semantic:No metric mask given, add dummy metric mask of all 1s. INFO:incasem.pipeline.sources.data_sources_semantic:No labels given, add dummy background labels. ERROR:gunpowder.build:something went wrong during the setup of the pipeline, calling tear down Exception ignored in: Traceback (most recent call last): File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/producer_pool.py", line 33, in __del__ self.stop() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/producer_pool.py", line 82, in stop self.__watch_dog.join() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/process.py", line 146, in join self._check_closed() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/process.py", line 100, in _check_closed if self._closed: AttributeError: 'Process' object has no attribute '_closed' Traceback (most recent call last): File "", line 1, in File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/spawn.py", line 126, in _main self = reduction.pickle.load(from_parent) AttributeError: 'ProducerPool' object has no attribute '__run_watch_dog' ERROR:example_prediction_cell6_ER:Failed after 0:00:02! Traceback (most recent call last): File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/pipeline.py", line 97, in node_setup node.output.setup() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/nodes/scan.py", line 60, in setup self.workers.start() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/producer_pool.py", line 46, in start self.__watch_dog.start() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/context.py", line 284, in _Popen return Popen(process_obj) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__ super().__init__(process_obj) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__ self._launch(process_obj) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch reduction.dump(process_obj, fp) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) TypeError: cannot pickle 'weakref' object The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/build.py", line 12, in __enter__ self.pipeline.setup() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/pipeline.py", line 101, in setup self.traverse( File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/pipeline.py", line 69, in traverse result.append(callback(self)) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/pipeline.py", line 99, in node_setup raise PipelineSetupError(node.output) from e gunpowder.pipeline.PipelineSetupError: Exception in Scan while calling setup() During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/pipeline.py", line 118, in node_teardown node.output.internal_teardown() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/nodes/batch_filter.py", line 119, in internal_teardown self.teardown() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/nodes/scan.py", line 65, in teardown self.workers.stop() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/producer_pool.py", line 82, in stop self.__watch_dog.join() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/process.py", line 148, in join assert self._popen is not None, 'can only join a started process' AssertionError: can only join a started process The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/sacred/experiment.py", line 312, in run_commandline return self.run( File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/sacred/experiment.py", line 276, in run run() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/sacred/run.py", line 238, in __call__ self.result = self.main_function(*args) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/sacred/config/captured_function.py", line 42, in captured_function result = wrapped(*args, **kwargs) File "predict.py", line 327, in predict with gp.build(prediction.pipeline) as p: File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/build.py", line 15, in __enter__ self.pipeline.internal_teardown() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/pipeline.py", line 123, in internal_teardown self.traverse( File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/pipeline.py", line 69, in traverse result.append(callback(self)) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/pipeline.py", line 120, in node_teardown raise PipelineTeardownError(node.output) from e gunpowder.pipeline.PipelineTeardownError: Exception in Scan while calling teardown() During handling of the above exception, another exception occurred: Traceback (most recent call last): File "predict.py", line 445, in ex.run_commandline(argv) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/sacred/experiment.py", line 347, in run_commandline print_filtered_stacktrace() File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/sacred/utils.py", line 494, in print_filtered_stacktrace print(format_filtered_stacktrace(filter_traceback), file=sys.stderr) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/sacred/utils.py", line 529, in format_filtered_stacktrace return "".join(filtered_traceback_format(tb_exception)) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/sacred/utils.py", line 560, in filtered_traceback_format yield from filtered_traceback_format(tb_exception.__cause__, chain=chain) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/sacred/utils.py", line 566, in filtered_traceback_format yield from filtered_traceback_format(tb_exception.__context__, chain=chain) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/sacred/utils.py", line 560, in filtered_traceback_format yield from filtered_traceback_format(tb_exception.__cause__, chain=chain) File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/sacred/utils.py", line 569, in filtered_traceback_format current_tb = tb_exception.exc_traceback AttributeError: 'TracebackException' object has no attribute 'exc_traceback' Exception ignored in: Traceback (most recent call last): File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/producer_pool.py", line 33, in __del__ File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/site-packages/gunpowder/producer_pool.py", line 82, in stop File "/Users/genevieb/mambaforge/envs/incasem/lib/python3.8/multiprocessing/process.py", line 148, in join AssertionError: can only join a started process ```
bentaculum commented 1 year ago

Thanks for filing, fixed in 0dedc1cbafc84bee17d4cdccd445616054b6529e.