search

vqdang / hover_net

Simultaneous Nuclear Instance Segmentation and Classification in H&E Histology Images.
MIT License
523 stars 220 forks source link

Error running the code #9

Closed ChristianEschen closed 5 years ago

ChristianEschen commented 5 years ago

Hi

Thanks again again for the awesome code. I have now switched to a working Ubuntu machine. However, I still have problems running your code. As I am a bit new to the model tensorpack I will appreciate some help here.

$python train.py --gpu='0'

OpenCV is built with OpenMP support. This usually results in poor performance. For details, see https://github.com/tensorpack/benchmarks/blob/master/ImageNet/benchmark-opencv-resize.py
[270, 270] [80, 80]
[0811 16:53:46 @parallel.py:327] [MultiProcessRunnerZMQ] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[270, 270] [80, 80]
[0811 16:53:46 @logger.py:90] Argv: train.py --gpu=0
2019-08-11 16:53:46.244932: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-08-11 16:53:46.273391: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-11 16:53:46.273857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.665
pciBusID: 0000:01:00.0
2019-08-11 16:53:46.273996: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-08-11 16:53:46.274974: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-08-11 16:53:46.275938: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-08-11 16:53:46.276112: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-08-11 16:53:46.277188: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-08-11 16:53:46.277829: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-08-11 16:53:46.280153: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-08-11 16:53:46.280315: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-11 16:53:46.280833: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-11 16:53:46.281246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
[0811 16:53:46 @interface.py:31] Automatically applying QueueInput on the DataFlow.
Traceback (most recent call last):
  File "train.py", line 268, in <module>
    trainer.run()
  File "train.py", line 236, in run
    self.run_once(opt, sess_init=init_weights, save_dir=log_dir)
  File "train.py", line 208, in run_once
    launch_train_with_config(config, SyncMultiGPUTrainerParameterServer(nr_gpus))
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/train/interface.py", line 90, in launch_train_with_config
    model.get_input_signature(), input,
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/utils/argtools.py", line 200, in wrapper
    value = func(*args, **kwargs)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/graph_builder/model_desc.py", line 86, in get_input_signature
    inputs = self.inputs()
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/graph_builder/model_desc.py", line 118, in inputs
    raise NotImplementedError()
NotImplementedError
simongraham commented 5 years ago

Hi @ChristianEschen ,

Can you please confirm the version of tensorpack and tensorflow that you are using.

Simon

ChristianEschen commented 5 years ago

tensorflow 1.14.0 tensorpack 0.9.7.1

simongraham commented 5 years ago

First, please try running with tensorpack version 0.9.0.1

ChristianEschen commented 5 years ago 
python train.py --gpu='0'
OpenCV is built with OpenMP support. This usually results in poor performance. For details, see https://github.com/tensorpack/benchmarks/blob/master/ImageNet/benchmark-opencv-resize.py
WARNING: Logging before flag parsing goes to stderr.
W0812 07:56:54.924021 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/tfutils/common.py:151: The name tf.VERSION is deprecated. Please use tf.version.VERSION instead.

W0812 07:56:55.220246 139644337526592 lazy_loader.py:50] 
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W0812 07:56:55.222888 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/callbacks/graph.py:81: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

W0812 07:56:55.223329 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/callbacks/hooks.py:13: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.

W0812 07:56:55.227744 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/tfutils/optimizer.py:16: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

W0812 07:56:55.228045 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/tfutils/sesscreate.py:20: The name tf.train.SessionCreator is deprecated. Please use tf.compat.v1.train.SessionCreator instead.

W0812 07:56:55.628230 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/projects/Nuclei_segmentation/hover_net_tensorflow/src/opt/hover.py:59: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W0812 07:56:55.628424 139644337526592 deprecation_wrapper.py:119] From train.py:227: The name tf.random.set_random_seed is deprecated. Please use tf.compat.v1.random.set_random_seed instead.

W0812 07:56:55.628721 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/tfutils/sessinit.py:258: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.

[270, 270] [80, 80]
[0812 07:56:56 @parallel.py:293] [PrefetchDataZMQ] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[270, 270] [80, 80]
[0812 07:56:56 @logger.py:73] Argv: train.py --gpu=0
2019-08-12 07:56:56.180655: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-08-12 07:56:56.200837: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3400240000 Hz
2019-08-12 07:56:56.201180: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5575dbe088d0 executing computations on platform Host. Devices:
2019-08-12 07:56:56.201198: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-08-12 07:56:56.202119: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-08-12 07:56:56.238656: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-12 07:56:56.239110: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.665
pciBusID: 0000:01:00.0
2019-08-12 07:56:56.239290: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-08-12 07:56:56.240430: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-08-12 07:56:56.241469: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-08-12 07:56:56.241662: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-08-12 07:56:56.242868: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-08-12 07:56:56.243516: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-08-12 07:56:56.246143: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-08-12 07:56:56.246283: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-12 07:56:56.246829: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-12 07:56:56.247238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-08-12 07:56:56.247284: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-08-12 07:56:56.335924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-12 07:56:56.335948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2019-08-12 07:56:56.335956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2019-08-12 07:56:56.336146: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-12 07:56:56.336740: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-12 07:56:56.337231: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-12 07:56:56.337642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:0 with 9786 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-08-12 07:56:56.338852: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5575df9de510 executing computations on platform CUDA. Devices:
2019-08-12 07:56:56.338863: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
W0812 07:56:56.339349 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/callbacks/saver.py:41: The name tf.gfile.IsDirectory is deprecated. Please use tf.io.gfile.isdir instead.

W0812 07:56:56.339528 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/projects/Nuclei_segmentation/hover_net_tensorflow/src/model/graph.py:123: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0812 07:56:56.342355 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/projects/Nuclei_segmentation/hover_net_tensorflow/src/model/graph.py:125: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W0812 07:56:56.343324 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/tfutils/common.py:32: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

[0812 07:56:56 @interface.py:34] Automatically applying QueueInput on the DataFlow.
[0812 07:56:56 @develop.py:96] WRN [Deprecated] ModelDescBase._get_inputs() interface will be deprecated after 30 Mar. Use inputs() instead!
W0812 07:56:56.343840 139644337526592 deprecation_wrapper.py:119] From /home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/graph_builder/model_desc.py:62: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

[0812 07:56:56 @input_source.py:220] Setting up the queue 'QueueInput/input_queue' for CPU prefetching ...
[0812 07:56:56 @training.py:114] Building graph for training tower 0 ...
Traceback (most recent call last):
  File "train.py", line 268, in <module>
    trainer.run()
  File "train.py", line 236, in run
    self.run_once(opt, sess_init=init_weights, save_dir=log_dir)
  File "train.py", line 208, in run_once
    launch_train_with_config(config, SyncMultiGPUTrainerParameterServer(nr_gpus))
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/train/interface.py", line 87, in launch_train_with_config
    model._build_graph_get_cost, model.get_optimizer)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/utils/argtools.py", line 176, in wrapper
    return func(*args, **kwargs)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/train/tower.py", line 204, in setup_graph
    train_callbacks = self._setup_graph(input, get_cost_fn, get_opt_fn)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/train/trainers.py", line 106, in _setup_graph
    self._make_get_grad_fn(input, get_cost_fn, get_opt_fn), get_opt_fn)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/graph_builder/training.py", line 161, in build
    grad_list = DataParallelBuilder.build_on_towers(self.towers, get_grad_fn, devices)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/graph_builder/training.py", line 119, in build_on_towers
    ret.append(func())
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/train/tower.py", line 232, in get_grad_fn
    cost = get_cost_fn(*input.get_input_tensors())
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/input_source/input_source_base.py", line 82, in get_input_tensors
    return self._get_input_tensors()
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/input_source/input_source.py", line 267, in _get_input_tensors
    ret = self.queue.dequeue(name='input_deque')
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 445, in dequeue
    self._queue_ref, self._dtypes, name=name)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 4140, in queue_dequeue_v2
    timeout_ms=timeout_ms, name=name)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3617, in create_op
    self._create_op_helper(ret, compute_device=compute_device)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3690, in _create_op_helper
    self._apply_device_functions(op)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 4576, in _apply_device_functions
    device_string = device_spec.string_merge(op)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 128, in string_merge
    return compat.as_str(_device_string(self.function(node_def)))
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/graph_builder/utils.py", line 84, in __call__
    from tensorflow.python.training.device_util import canonicalize
ModuleNotFoundError: No module named 'tensorflow.python.training.device_util'
vqdang commented 5 years ago

Hi @ChristianEschen, please running with tensorflow 1.12.

EDIT: please also use markdown to separate your message out from the logging output. I have edited your comment to make it look clearer

ChristianEschen commented 5 years ago

With my current anaconda enviroment I get conflicts by installing tensorflow version 1.12. Could you please list the specific version of the modules you have used?

vqdang commented 5 years ago

@ChristianEschen The modules you need can be found here https://github.com/vqdang/hover_net/blob/master/requirements.txt. I think it would be faster if you just make a new environment.

p/s: also please pull the latest changes as I just fixed a minor bug.

ChristianEschen commented 5 years ago

I still have errors with the configuration you list in requurements.txt $conda install -c conda-forge matplotlib

Collecting package metadata (current_repodata.json): done Solving environment: failed with current_repodata.json, will retry with next repodata source. Initial quick solve with frozen env failed. Unfreezing env and trying again. Solving environment: failed with current_repodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): done Solving environment: failed Initial quick solve with frozen env failed. Unfreezing env and trying again. Solving environment: failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Package numpy conflicts for: keras-applications -> keras[version='>=2.1.6'] -> tensorflow -> keras-preprocessing[version='>=1.0.3,>=1.0.5'] -> scipy[version='>=0.14'] -> numpy[version='1.10.,>=1.11.3,<2.0a0,>=1.14.6,<2.0a0,>=1.9,>=1.9.3,<2.0a0'] mkl_random -> numpy[version='>=1.11,>=1.11.3,<2.0a0,>=1.9.3,<2.0a0'] mkl-service -> numpy[version='>=1.11.3,<2.0a0'] keras-preprocessing -> keras[version='>=2.1.6'] -> tensorflow -> keras-applications[version='>=1.0.5,>=1.0.6'] -> numpy[version='>=1.9.1'] numpy tensorflow-gpu=1.12 -> tensorflow==1.12.0 -> keras-preprocessing[version='>=1.0.3,>=1.0.5'] -> keras[version='>=2.1.6'] -> keras-base=2.2.4 -> keras-applications[version='>=1.0.6'] -> numpy[version='>=1.9.1'] mkl_fft -> numpy[version='>=1.11,>=1.11.3,<2.0a0'] matplotlib -> numpy tensorflow -> tensorflow-estimator[version='>=1.14.0,<1.15.0'] -> tensorflow-base[version='>=1.14.0,<1.15.0a0'] -> keras-preprocessing[version='>=1.0.3,>=1.0.5'] -> keras[version='>=2.1.6'] -> keras-base=2.2.4 -> keras-applications[version='>=1.0.6'] -> numpy[version='>=1.9.1'] h5py -> numpy[version='1.10.,1.12.,>=1.11.3,<2.0a0,>=1.14.6,<2.0a0,>=1.8,>=1.9.3,<2.0a0'] tensorboard -> numpy[version='>=1.12,>=1.12.0'] opencv=3.2.0 -> numpy[version='1.11.,1.12.'] scipy -> numpy[version='1.10.,>=1.11.3,<2.0a0,>=1.14.6,<2.0a0,>=1.9,>=1.9.3,<2.0a0'] tensorflow-base -> tensorflow-estimator[version='>=1.13.0,<1.14.0a0'] -> numpy[version='>=1.13.3,>=1.16.1'] Package freetype conflicts for: harfbuzz -> cairo[version='1.14.,>=1.14.12,<1.15.0a0,>=1.16.0,<1.17.0a0'] -> fontconfig[version='2.11.,2.12.,>=2.13.1,<3.0a0'] -> freetype[version='2.6.,2.7,2.8.1,>=2.8.1,<2.9.0a0,>=2.9.1,<3.0a0'] cairo -> fontconfig[version='2.11.,2.12.,>=2.13.1,<3.0a0'] -> freetype[version='2.6.,2.7,2.8.1,>=2.8.1,<2.9.0a0,>=2.9.1,<3.0a0'] matplotlib -> pyqt[version='5.6.,5.9.'] -> qt[version='4.8.,5.6.,5.9.,>=5.6.2,<5.7.0a0,>=5.9.4,<5.10.0a0,>=5.9.7,<5.10.0a0'] -> gtk2 -> gdk-pixbuf -> gobject-introspection -> cairo[version='>=1.14.12,<1.15.0a0,>=1.14.12,<2.0a0,>=1.16.0,<1.17.0a0'] -> fontconfig[version='2.12.,>=2.13.1,<3.0a0'] -> freetype[version='2.7,2.8.1,>=2.8.1,<2.9.0a0,>=2.9.1,<3.0a0'] fontconfig -> freetype[version='2.6.,2.7,2.8.1,>=2.8.1,<2.9.0a0,>=2.9.1,<3.0a0'] freetype ffmpeg -> freetype[version='2.8.1,>=2.8,<2.9.0a0,>=2.9.1,<3.0a0'] opencv=3.2.0 -> fontconfig[version='2.11.,2.12.'] -> freetype[version='2.6.*,2.7']

simongraham commented 5 years ago

It seems that keras and tensorflow are in conflict with each other.

We recommend setting up a new environment without keras to help resolve this conflict.

vqdang commented 5 years ago

@ChristianEschen, please use a new conda environment and do as follow

conda create --name test python=3.6
conda activate test
pip install opencv-python=3.2 scipy scikit-image pandas matplotlib
pip install --upgrade git+https://github.com/tensorpack/tensorpack.git@0.9.0.1
pip install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.12.0-cp36-cp36m-linux_x86_64.whl
source activate test

I have tested this on a different linux machine and I can run the training from the terminal without any conflict or error.

vqdang commented 5 years ago

Close due to lack of activity, I assume you have resolved the problem. Feel free to reopen if you encounter anything else.