search

hammerlab / cytokit

Microscopy Image Cytometry Toolkit
Apache License 2.0
115 stars 18 forks source link

Tensorflow DType error #24

Closed ninning closed 4 years ago

ninning commented 4 years ago

Hi Cytokit team,

Using Python 3.6 Tensorflow 1.12.0

I was testing out some of the cytokit python pipelines and was testing out the example in the marker_profiling_example

I got to the point where I was going to run the process pipeline on the example datasets in https://storage.googleapis.com/cytokit/datasets/cellular-marker/{exp}/1_00001_Z{z:03d}_CH{ch}.tif' when I got this error message:

!cytokit processor run_all --config-path=$variant_dir/config --data-dir=$raw_dir --output-dir=$variant_dir

/Users/ninning/venv/cytokit/lib/python3.6/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. data = yaml.load(f.read()) or {} /Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/config.py:21: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. return yaml.load(fd) 2020-01-31 18:45:57,734:INFO:52737:root: Execution arguments and environment saved to "/var/folders/sq/qy7gh_jj1lgfdyh2wf4vt8p00000gn/T/cytokit-example/cellular-marker/20181116-d40-r1-20x-5by5/output/v00/processor/execution/202001312345.json" /Users/ninning/venv/cytokit/lib/python3.6/site-packages/bokeh/themes/theme.py:131: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. json = yaml.load(f) /Users/ninning/venv/cytokit/lib/python3.6/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. data = yaml.load(f.read()) or {} /Users/ninning/venv/cytokit/lib/python3.6/site-packages/bokeh/themes/theme.py:131: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. json = yaml.load(f) 2020-01-31 18:46:07,021:INFO:52737:cytokit.exec.pipeline: Starting Pre-processing pipeline for 1 tasks (1 workers) distributed.worker - WARNING - Compute Failed Function: run_preprocess_task args: ({'region_indexes': array([0]), 'tile_indexes': array([0]), 'data_dir': '/var/folders/sq/qy7gh_jj1lgfdyh2wf4vt8p00000gn/T/cytokit-example/cellular-marker/20181116-d40-r1-20x-5by5/raw', 'output_dir': '/var/folders/sq/qy7gh_jj1lgfdyh2wf4vt8p00000gn/T/cytokit-example/cellular-marker/20181116-d40-r1-20x-5by5/output/v00', 'gpu': 0, 'op_flags': <cytokit.exec.pipeline.OpFlags object at 0x12c4cd898>, 'tile_prefetch_capacity': 1}) kwargs: {} Exception: Exception('Long error message', <class 'tensorflow.python.framework.errors_impl.OutOfRangeError'>, 'Read less bytes than requested\n\t [[node save/RestoreV2 (defined at /Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/miq/prediction.py:61) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]\n\nCaused by op \'save/RestoreV2\', defined at:\n File "/Users/ninning/.pyenv/versions/3.6.10/lib/python3.6/threading.py", line 884, in _bootstrap\n self._bootstrap_inner()\n File "/Users/ninning/.pyenv/versions/3.6.10/lib/python3.6/threading.py", line 916, in _bootstrap_inner\n self.run()\n File "/Users/ninning/.pyenv/versions/3.6.10/lib/python3.6/threading.py", line 864, in run\n self._target(*self._args, **self._kwargs)\n File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/distributed/threadpoolexecutor.py", line 57, in _worker\n task.run()\n File "/Users/ninning/venv/cytokit/lib/')

Traceback (most recent call last): File "/Users/ninning/venv/cytokit/bin/cytokit", line 8, in sys.exit(main()) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/cli/main.py", line 28, in main fire.Fire(Cytokit) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/fire/core.py", line 138, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/fire/core.py", line 471, in _Fire target=component.name) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/fire/core.py", line 675, in _CallAndUpdateTrace component = fn(varargs, kwargs) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/cli/init.py", line 167, in run_all fn({config[op], params}) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/cli/processor.py", line 131, in run pipeline.run(pl_config, logging_init_fn=self._logging_init_fn) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/exec/pipeline.py", line 458, in run run_tasks(pl_conf, 'Pre-processing', run_preprocess_task, logging_init_fn) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/exec/pipeline.py", line 421, in run_tasks res = [r.result() for r in res] File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/exec/pipeline.py", line 421, in res = [r.result() for r in res] File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/distributed/client.py", line 227, in result six.reraise(result) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/six.py", line 702, in reraise raise value.with_traceback(tb) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/exec/pipeline.py", line 441, in run_preprocess_task return run_task(task, ops, preprocess_tile) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/exec/pipeline.py", line 355, in run_task with ops: File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/ops/op.py", line 200, in enter v.enter() File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/ops/op.py", line 152, in enter self.initialize() File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/ops/best_focus.py", line 46, in initialize graph=self.graph, session_config=get_tf_config(self) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/miq/prediction.py", line 63, in init saver.restore(self._sess, model_ckpt) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1546, in restore {self.saver_def.filename_tensor_name: save_path}) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) Exception: ('Long error message', <class 'tensorflow.python.framework.errors_impl.OutOfRangeError'>, 'Read less bytes than requested\n\t [[node save/RestoreV2 (defined at /Users/ninning/venv/cytokit/lib/python3.6/site-packages/cytokit/miq/prediction.py:61) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]\n\nCaused by op \'save/RestoreV2\', defined at:\n File "/Users/ninning/.pyenv/versions/3.6.10/lib/python3.6/threading.py", line 884, in _bootstrap\n self._bootstrap_inner()\n File "/Users/ninning/.pyenv/versions/3.6.10/lib/python3.6/threading.py", line 916, in _bootstrap_inner\n self.run()\n File "/Users/ninning/.pyenv/versions/3.6.10/lib/python3.6/threading.py", line 864, in run\n self._target(*self._args, **self._kwargs)\n File "/Users/ninning/venv/cytokit/lib/python3.6/site-packages/distributed/threadpoolexecutor.py", line 57, in _worker\n task.run()\n File "/Users/ninning/venv/cytokit/lib/')

Appreciate any help and thank you in advance!

eric-czech commented 4 years ago

The suggestion here will likely fix it.

ninning commented 4 years ago

Thanks Eric!

Deleted the old NN file and re-ran the entire notebook from the start and it fixed it. Not sure why the download was incomplete initially, I suspect i might have terminated the kernel before the download was completed initially but I cannot seem to reproduce that error now.