search

NifTK / NiftyNet

[unmaintained] An open-source convolutional neural networks platform for research in medical image analysis and image-guided therapy
http://niftynet.io
Apache License 2.0
1.36k stars 404 forks source link

CRITICAL:niftynet: Don't know how to generate sampling locations: Spatial dimensions of the grouped input sources are not consistent. {(512, 512, 102), (512, 512, 92)} 2019-09-27 17:21:51.189240: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: NotImplementedError: #443

Closed zhaoxiaoqian666 closed 4 years ago

zhaoxiaoqian666 commented 4 years ago

What you were trying to do (and why)

I encountered the problem in training Unet,I saw the previous posts, but my problem has not been solved. Maybe the wrong situation is different. Below are my configuration files and errors:

What happened (include command output)

CRITICAL:tensorflow:Optional Python module cv2 not found, please install cv2 and retry if the application fails.
CRITICAL:tensorflow:Optional Python module skimage.io not found, please install skimage.io and retry if the application fails.
INFO:niftynet: Optional Python module yaml not found, please install yaml and retry if the application fails.
INFO:niftynet: Optional Python module yaml version None not found, please install yaml-None and retry if the application fails.
NiftyNet version 0.5.0
INFO:niftynet: Optional Python module yaml not found, please install yaml and retry if the application fails.
INFO:niftynet: Optional Python module yaml version None not found, please install yaml-None and retry if the application fails.
INFO:niftynet: Resolving ./group1test.csv as /home/deserts/Projects/NiftyNet-dev/config/g1/./group1test.csv
INFO:niftynet: Resolving ./group1label.csv as /home/deserts/Projects/NiftyNet-dev/config/g1/./group1label.csv
[CUSTOM]
-- num_classes: 10
-- output_prob: False
-- label_normalisation: True
-- softmax: True
-- min_sampling_ratio: 0
-- compulsory_labels: (0, 1)
-- rand_samples: 0
-- min_numb_labels: 1
-- proba_connect: True
-- evaluation_units: foreground
-- image: ('image',)
-- inferred: ()
-- label: ('label',)
-- sampler: ('label',)
-- weight: ()
-- name: net_segment
[CONFIG_FILE]
-- path: /home/deserts/Projects/NiftyNet-dev/config/g1/unet_config_g1.ini
[IMAGE]
-- csv_file: /home/deserts/Projects/NiftyNet-dev/config/g1/group1test.csv
-- filename_contains: None
-- filename_not_contains: ()
-- filename_removefromid: 
-- interp_order: 1
-- loader: None
-- pixdim: ()
-- axcodes: ()
-- spatial_window_size: (96, 96, 96)
[LABEL]
-- csv_file: /home/deserts/Projects/NiftyNet-dev/config/g1/group1label.csv
-- filename_contains: None
-- filename_not_contains: ()
-- filename_removefromid: 
-- interp_order: 1
-- loader: None
-- pixdim: ()
-- axcodes: ()
-- spatial_window_size: (80, 80, 80)
[SYSTEM]
-- cuda_devices: 1
-- num_threads: 2
-- num_gpus: 1
-- model_dir: /home/deserts/Projects/NiftyNet-dev/models/model_unet_g1
-- dataset_split_file: ./dataset_split.csv
-- event_handler: ('model_saver', 'model_restorer', 'sampler_threading', 'apply_gradients', 'output_interpreter', 'console_logger', 'tensorboard_logger')
-- iteration_generator: iteration_generator
-- action: training
[NETWORK]
-- name: niftynet.network.unet.UNet3D
-- activation_function: relu
-- batch_size: 2
-- smaller_final_batch_mode: pad
-- decay: 0.0
-- reg_type: L2
-- volume_padding_size: (0, 0, 0)
-- volume_padding_mode: minimum
-- window_sampling: weighted
-- queue_length: 10
-- multimod_foreground_type: and
-- histogram_ref_file: ./histogram_ref_file.txt
-- norm_type: percentile
-- cutoff: (0.01, 0.99)
-- foreground_type: otsu_plus
-- normalisation: False
-- whitening: False
-- normalise_foreground_only: False
-- weight_initializer: he_normal
-- bias_initializer: zeros
-- keep_prob: 1.0
[TRAINING]
-- optimiser: adam
-- sample_per_volume: 32
-- rotation_angle: ()
-- rotation_angle_x: ()
-- rotation_angle_y: ()
-- rotation_angle_z: ()
-- scaling_percentage: ()
-- antialiasing: True
-- bias_field_range: ()
-- bf_order: 3
-- random_flipping_axes: -1
-- do_elastic_deformation: False
-- num_ctrl_points: 4
-- deformation_sigma: 15
-- proportion_to_deform: 0.5
-- lr: 0.001
-- loss_type: Dice
-- starting_iter: 0
-- save_every_n: 2000
-- tensorboard_every_n: 20
-- max_iter: 20000
-- max_checkpoints: 200
-- validation_every_n: -1
-- validation_max_iter: 1
-- exclude_fraction_for_validation: 0.0
-- exclude_fraction_for_inference: 0.0
-- vars_to_restore: 
-- vars_to_freeze: 
[INFERENCE]
-- spatial_window_size: (128, 128, 56)
-- inference_iter: -1
-- dataset_to_infer: 
-- save_seg_dir: ./output/unet
-- output_postfix: _niftynet_out
-- output_interp_order: 0
-- border: (20, 20, 20)
INFO:niftynet: set CUDA_VISIBLE_DEVICES to 1
INFO:niftynet: starting segmentation application
INFO:niftynet: [image] using existing csv file /home/deserts/Projects/NiftyNet-dev/config/g1/group1test.csv, skipped filenames search
INFO:niftynet: [label] using existing csv file /home/deserts/Projects/NiftyNet-dev/config/g1/group1label.csv, skipped filenames search
INFO:niftynet: 

Number of subjects 50, input section names: ['subject_id', 'image', 'label']
-- using all subjects (without data partitioning).

INFO:niftynet: Image reader: loading 33 subjects from sections ('image',) as input [image]
INFO:niftynet: Image reader: loading 33 subjects from sections ('label',) as input [label]
INFO:niftynet: Image reader: loading 33 subjects from sections ('label',) as input [sampler]
INFO:niftynet: label mapping ready for label:('label',), 10 classes
2019-09-27 17:21:41.164694: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2019-09-27 17:21:41.905813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.645
pciBusID: 0000:65:00.0
totalMemory: 10.91GiB freeMemory: 10.77GiB
2019-09-27 17:21:41.905842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-09-27 17:21:42.244710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-27 17:21:42.244753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-09-27 17:21:42.244761: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-09-27 17:21:42.245500: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 10419 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:65:00.0, compute capability: 6.1)
INFO:niftynet: initialised uniform sampler {'image': (32, 96, 96, 96, 1, 1), 'image_location': (32, 7), 'label': (32, 80, 80, 80, 1, 1), 'label_location': (32, 7), 'sampler': (32, 80, 80, 80, 1, 1), 'sampler_location': (32, 7)} 
INFO:niftynet: Initialised weighted sampler window instance
INFO:niftynet: Import [UNet3D] from /home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/network/unet.py.
WARNING:niftynet: From /home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_initializer.py:106: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with distribution=normal is deprecated and will be removed in a future version.
Instructions for updating:
`normal` is a deprecated alias for `truncated_normal`
using UNet
2019-09-27 17:21:42.290413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-09-27 17:21:42.290445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-27 17:21:42.290453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-09-27 17:21:42.290457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-09-27 17:21:42.290724: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 10419 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:65:00.0, compute capability: 6.1)
INFO:niftynet: Initialising Dataset from 33 subjects...
[Layer] UNet/L1 [Trainable] 32_bn_relu/conv_/w, 32_bn_relu/bn_/beta, 32_bn_relu/bn_/gamma, 64_bn_relu/conv_/w, 64_bn_relu/bn_/beta, 64_bn_relu/bn_/gamma (56352)
[Layer] UNet/L2 [Trainable] 64_bn_relu/conv_/w, 64_bn_relu/bn_/beta, 64_bn_relu/bn_/gamma, 128_bn_relu/conv_/w, 128_bn_relu/bn_/beta, 128_bn_relu/bn_/gamma (332160)
[Layer] UNet/L3 [Trainable] 128_bn_relu/conv_/w, 128_bn_relu/bn_/beta, 128_bn_relu/bn_/gamma, 256_bn_relu/conv_/w, 256_bn_relu/bn_/beta, 256_bn_relu/bn_/gamma (1327872)
[Layer] UNet/L4 [Trainable] 256_bn_relu/conv_/w, 256_bn_relu/bn_/beta, 256_bn_relu/bn_/gamma, 512_bn_relu/conv_/w, 512_bn_relu/bn_/beta, 512_bn_relu/bn_/gamma, up_2x2_bn/deconv_/w, up_2x2_bn/bn_/beta, up_2x2_bn/bn_/gamma (7408128)
[Layer] UNet/R3 [Trainable] 256_bn_relu/conv_/w, 256_bn_relu/bn_/beta, 256_bn_relu/bn_/gamma, 256_bn_relu_1/conv_/w, 256_bn_relu_1/bn_/beta, 256_bn_relu_1/bn_/gamma, up_2x2_bn/deconv_/w, up_2x2_bn/bn_/beta, up_2x2_bn/bn_/gamma (7603712)
[Layer] UNet/R2 [Trainable] 128_bn_relu/conv_/w, 128_bn_relu/bn_/beta, 128_bn_relu/bn_/gamma, 128_bn_relu_1/conv_/w, 128_bn_relu_1/bn_/beta, 128_bn_relu_1/bn_/gamma, up_2x2_bn/deconv_/w, up_2x2_bn/bn_/beta, up_2x2_bn/bn_/gamma (1901312)
[Layer] UNet/R1_FC [Trainable] 64_bn_relu/conv_/w, 64_bn_relu/bn_/beta, 64_bn_relu/bn_/gamma, 64_bn_relu_1/conv_/w, 64_bn_relu_1/bn_/beta, 64_bn_relu_1/bn_/gamma, 10_bn_relu/conv_/w, 10_bn_relu/bn_/beta, 10_bn_relu/bn_/gamma (443284)
2019-09-27 17:21:45.419235: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-09-27 17:21:45.419282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-27 17:21:45.419289: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-09-27 17:21:45.419294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-09-27 17:21:45.419572: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10419 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:65:00.0, compute capability: 6.1)
INFO:niftynet: Parameters from random initialisations ...
CRITICAL:niftynet: Don't know how to generate sampling locations: Spatial dimensions of the grouped input sources are not consistent. {(512, 512, 102), (512, 512, 92)}
2019-09-27 17:21:51.189240: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

Traceback (most recent call last):

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 298, in <lambda>
    func=lambda subject_id: nest.flatten(self(subject_id)),

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 34, in __call__
    return self._op(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 360, in __call__
    return self._call_func(args, kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 311, in _call_func
    result = self._func(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 73, in layer_op
    n_samples=self.window.n_samples)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 141, in _spatial_coordinates_generator
    _infer_spatial_size(img_sizes, win_sizes)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 229, in _infer_spatial_size
    raise NotImplementedError

NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

INFO:niftynet: cleaning up...
CRITICAL:niftynet: Don't know how to generate sampling locations: Spatial dimensions of the grouped input sources are not consistent. {(512, 512, 118), (512, 512, 124)}
2019-09-27 17:21:51.715303: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

Traceback (most recent call last):

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 298, in <lambda>
    func=lambda subject_id: nest.flatten(self(subject_id)),

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 34, in __call__
    return self._op(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 360, in __call__
    return self._call_func(args, kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 311, in _call_func
    result = self._func(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 73, in layer_op
    n_samples=self.window.n_samples)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 141, in _spatial_coordinates_generator
    _infer_spatial_size(img_sizes, win_sizes)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 229, in _infer_spatial_size
    raise NotImplementedError

NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

INFO:niftynet: iter 1 saved: /home/deserts/Projects/NiftyNet-dev/models/model_unet_g1/models/model.ckpt
INFO:niftynet: stopping sampling threads
CRITICAL:niftynet: Don't know how to generate sampling locations: Spatial dimensions of the grouped input sources are not consistent. {(512, 512, 117), (512, 512, 108)}
2019-09-27 17:21:53.808477: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

Traceback (most recent call last):

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 298, in <lambda>
    func=lambda subject_id: nest.flatten(self(subject_id)),

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 34, in __call__
    return self._op(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 360, in __call__
    return self._call_func(args, kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 311, in _call_func
    result = self._func(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 73, in layer_op
    n_samples=self.window.n_samples)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 141, in _spatial_coordinates_generator
    _infer_spatial_size(img_sizes, win_sizes)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 229, in _infer_spatial_size
    raise NotImplementedError

NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

CRITICAL:niftynet: Don't know how to generate sampling locations: Spatial dimensions of the grouped input sources are not consistent. {(512, 512, 117), (512, 512, 96)}
2019-09-27 17:21:54.333707: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

Traceback (most recent call last):

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 298, in <lambda>
    func=lambda subject_id: nest.flatten(self(subject_id)),

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 34, in __call__
    return self._op(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 360, in __call__
    return self._call_func(args, kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 311, in _call_func
    result = self._func(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 73, in layer_op
    n_samples=self.window.n_samples)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 141, in _spatial_coordinates_generator
    _infer_spatial_size(img_sizes, win_sizes)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 229, in _infer_spatial_size
    raise NotImplementedError

NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

CRITICAL:niftynet: Don't know how to generate sampling locations: Spatial dimensions of the grouped input sources are not consistent. {(512, 512, 120), (512, 512, 100)}
2019-09-27 17:21:56.029506: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

Traceback (most recent call last):

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 298, in <lambda>
    func=lambda subject_id: nest.flatten(self(subject_id)),

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 34, in __call__
    return self._op(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 360, in __call__
    return self._call_func(args, kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 311, in _call_func
    result = self._func(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 73, in layer_op
    n_samples=self.window.n_samples)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 141, in _spatial_coordinates_generator
    _infer_spatial_size(img_sizes, win_sizes)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 229, in _infer_spatial_size
    raise NotImplementedError

NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

Traceback (most recent call last):
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnimplementedError: NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

Traceback (most recent call last):

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 298, in <lambda>
    func=lambda subject_id: nest.flatten(self(subject_id)),

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 34, in __call__
    return self._op(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 360, in __call__
    return self._call_func(args, kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 311, in _call_func
    result = self._func(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 73, in layer_op
    n_samples=self.window.n_samples)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 141, in _spatial_coordinates_generator
    _infer_spatial_size(img_sizes, win_sizes)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 229, in _infer_spatial_size
    raise NotImplementedError

NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

     [[{{node PyFunc}} = PyFunc[Tin=[DT_INT64], Tout=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], token="pyfunc_0", _device="/device:GPU:0"](arg0)]]
     [[{{node worker_0/train/IteratorGetNext}} = IteratorGetNext[output_shapes=[[2,96,96,96,1,1], [2,7], [2,80,80,80,1,1], [2,7], [2,80,80,80,1,1], [2,7]], output_types=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](worker_0/train/OneShotIterator)]]
     [[{{node ApplyGradients/Adam/learning_rate/_45}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_2099_ApplyGradients/Adam/learning_rate", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 205, in run
    loop_status=loop_status)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 331, in loop
    ApplicationDriver.loop_step(application, iter_msg)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 363, in loop_step
    feed_dict=iteration_message.data_feed_dict)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnimplementedError: NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

Traceback (most recent call last):

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 298, in <lambda>
    func=lambda subject_id: nest.flatten(self(subject_id)),

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 34, in __call__
    return self._op(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 360, in __call__
    return self._call_func(args, kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 311, in _call_func
    result = self._func(*args, **kwargs)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 73, in layer_op
    n_samples=self.window.n_samples)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 141, in _spatial_coordinates_generator
    _infer_spatial_size(img_sizes, win_sizes)

  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 229, in _infer_spatial_size
    raise NotImplementedError

NotImplementedError: 

originally defined at:
  File "/home/deserts/anaconda3/envs/niftynet/bin/net_segment", line 10, in <module>
    sys.exit(main())
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/__init__.py", line 142, in main
    app_driver.run(app_driver.app)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 189, in run
    is_training_action=self.is_training_action)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/application_driver.py", line 252, in create_graph
    application.initialise_sampler()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 251, in initialise_sampler
    self.SUPPORTED_SAMPLING[self.net_param.window_sampling][0]()
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 201, in initialise_weighted_sampler
    self.readers]]
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/application/segmentation_application.py", line 200, in <listcomp>
    queue_length=self.net_param.queue_length) for reader in
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_weighted_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/sampler_uniform_v2.py", line 43, in __init__
    name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py", line 61, in __init__
    Layer.__init__(self, name=name)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/layer/base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "/home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/tensorflow/python/ops/template.py", line 154, in make_template
    **kwargs)

     [[{{node PyFunc}} = PyFunc[Tin=[DT_INT64], Tout=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], token="pyfunc_0", _device="/device:GPU:0"](arg0)]]
     [[node worker_0/train/IteratorGetNext (defined at /home/deserts/anaconda3/envs/niftynet/lib/python3.6/site-packages/niftynet/engine/image_window_dataset.py:201)  = IteratorGetNext[output_shapes=[[2,96,96,96,1,1], [2,7], [2,80,80,80,1,1], [2,7], [2,80,80,80,1,1], [2,7]], output_types=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](worker_0/train/OneShotIterator)]]
     [[{{node ApplyGradients/Adam/learning_rate/_45}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_2099_ApplyGradients/Adam/learning_rate", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
(niftynet) deserts@apollo:~/Projects/NiftyNet-dev$

I hope you can help me point out where the problem is,Thank you!

fepegar commented 4 years ago

Related to #274

fepegar commented 4 years ago

Hi @zhaoxiaoqian666,

It looks like your input images and labels have different size.

zhaoxiaoqian666 commented 4 years ago

Thank you ！But I check the image size of input and label,I’m sure they are consistent。------------------ 原始邮件 ------------------ 发件人: "Fernando Perez-Garcia"notifications@github.com 发送时间: 2019年9月30日(星期一) 晚上6:48 收件人: "NifTK/NiftyNet"NiftyNet@noreply.github.com; 抄送: "zhaoxiaoqian666"1476585534@qq.com;"Mention"mention@noreply.github.com; 主题: Re: [NifTK/NiftyNet] CRITICAL:niftynet: Don't know how to generatesampling locations: Spatial dimensions of the grouped input sources are notconsistent. {(512, 512, 102), (512, 512, 92)} 2019-09-27 17:21:51.189240: Wtensorflow/core/framework/op_kernel.cc:1261] Unimplemented:NotImplementedError: (#443)

Hi @zhaoxiaoqian666,

It looks like your input images and labels have different size.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

brynolff commented 4 years ago

The images may have the same size, but the blocks you extract from the images are different sizes:

[IMAGE] -- csv_file: /home/deserts/Projects/NiftyNet-dev/config/g1/group1test.csv -- filename_contains: None -- filename_not_contains: () -- filename_removefromid: -- interp_order: 1 -- loader: None -- pixdim: () -- axcodes: () -- spatial_window_size: (96, 96, 96) <------------------------------------------------- Block size of image [LABEL] -- csv_file: /home/deserts/Projects/NiftyNet-dev/config/g1/group1label.csv -- filename_contains: None -- filename_not_contains: () -- filename_removefromid: -- interp_order: 1 -- loader: None -- pixdim: () -- axcodes: () -- spatial_window_size: (80, 80, 80) <-------------------------------------------------- Block size of label

zhaoxiaoqian666 commented 4 years ago

I modified the parameter border of croplayer in unet.py as follows: crop_layer = CropLayer (border = 8, name ='crop-16'), so the size of image is equal to the size of label after removing the border.I don't know if that's true？

wyli commented 4 years ago

I modified the parameter border of croplayer in unet.py as follows: crop_layer = CropLayer (border = 8, name ='crop-16'), so the size of image is equal to the size of label after removing the border.I don't know if that's true？

That's true, alternatively you could try name= nonewnet which uses an improved version of unet 3d https://github.com/NifTK/NiftyNet/blob/dev/niftynet/network/no_new_net.py#L18

zhaoxiaoqian666 commented 4 years ago

Thank you!I found the mistake,I mistakenly wrote the test file into a training set.