`developer@linux:/shared/bonnet/train_py$ ./cnn_train.py -d cfg/cityscapes/data.yaml -n cfg/cityscapes/net_bonnet.yaml -t cfg/cityscapes/train_bonnet.yaml -l ./log
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
INTERFACE:
data yaml: cfg/cityscapes/data.yaml
net yaml: cfg/cityscapes/net_bonnet.yaml
train yaml: cfg/cityscapes/train_bonnet.yaml
log dir ./log
model path None
model type iou
Commit hash (training version): b'7bf03b0'
Opening desired data file cfg/cityscapes/data.yaml
Opening desired net file cfg/cityscapes/net_bonnet.yaml
Opening desired train file cfg/cityscapes/train_bonnet.yaml
Copying files to ./log for further reference.
Training from scratch
Fetching dataset
Training with 1 GPU's
Training with batch size 2
DEVICE AVAIL: /device:CPU:0
DEVICE AVAIL: /device:GPU:0
Number of GPU's available is 1
This means 2 images per GPU
Data depth: 3
Parsing directory /shared/datasets/cityscapes/bonnet_data/train
Total number of pixels: 6239027200
Total number of pixels of class road: 2036049408
Total number of pixels of class train: 12863940
Total number of pixels of class car: 386502976
Total number of pixels of class sky: 221459584
Total number of pixels of class trafficsign: 30521320
Total number of pixels of class fence: 48487192
Total number of pixels of class sidewalk: 336031712
Total number of pixels of class truck: 14775009
Total number of pixels of class person: 67202392
Total number of pixels of class vegetation: 878734336
Total number of pixels of class pole: 67768880
Total number of pixels of class building: 1259774336
Total number of pixels of class bus: 12995807
Total number of pixels of class rider: 7444910
Total number of pixels of class bicycle: 22849672
Total number of pixels of class terrain: 63965240
Total number of pixels of class trafficlight: 11509943
Total number of pixels of class motorcycle: 5445904
Total number of pixels of class wall: 36211256
Total number of pixels of class crap: 718432320
Content percentage of class road in dataset: 0.326341
Content percentage of class car in dataset: 0.061949
Content percentage of class trafficsign in dataset: 0.004892
Content percentage of class sidewalk in dataset: 0.053860
Content percentage of class truck in dataset: 0.002368
Content percentage of class vegetation in dataset: 0.140845
Content percentage of class building in dataset: 0.201918
Content percentage of class bus in dataset: 0.002083
Content percentage of class terrain in dataset: 0.010252
Content percentage of class wall in dataset: 0.005804
Content percentage of class train in dataset: 0.002062
Content percentage of class sky in dataset: 0.035496
Content percentage of class fence in dataset: 0.007772
Content percentage of class motorcycle in dataset: 0.000873
Content percentage of class person in dataset: 0.010771
Content percentage of class pole in dataset: 0.010862
Content percentage of class bicycle in dataset: 0.003662
Content percentage of class rider in dataset: 0.001193
Content percentage of class trafficlight in dataset: 0.001845
Content percentage of class crap in dataset: 0.115151
Total amount of images: 2975
OpenCV Error: Assertion failed (dims <= 2 && step[0] > 0) in locateROI, file /io/opencv/modules/core/src/matrix.cpp, line 991
Exception in thread ImgBufftrain:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/shared/bonnet/train_py/dataset/abstract_dataset.py", line 84, in run
img, lbl = self.augment(img, lbl)
File "/shared/bonnet/train_py/dataset/abstract_dataset.py", line 68, in augment
img = cv2.blur(img,(ksize,ksize))
cv2.error: /io/opencv/modules/core/src/matrix.cpp:991: error: (-215) dims <= 2 && step[0] > 0 in function locateROI
Parsing directory /shared/datasets/cityscapes/bonnet_data/valid
Total number of pixels: 1048576000
Total number of pixels of class road: 345222080
Total number of pixels of class train: 1032100
Total number of pixels of class car: 59759312
Total number of pixels of class sky: 30708080
Total number of pixels of class trafficsign: 6110454
Total number of pixels of class fence: 7527026
Total number of pixels of class sidewalk: 49559052
Total number of pixels of class truck: 2760469
Total number of pixels of class person: 11890229
Total number of pixels of class vegetation: 158682896
Total number of pixels of class pole: 13564731
Total number of pixels of class building: 200895344
Total number of pixels of class bus: 3564221
Total number of pixels of class rider: 1970543
Total number of pixels of class bicycle: 6500852
Total number of pixels of class terrain: 7625936
Total number of pixels of class trafficlight: 1813749
Total number of pixels of class motorcycle: 728922
Total number of pixels of class wall: 6720678
Total number of pixels of class crap: 131939472
Content percentage of class road in dataset: 0.329229
Content percentage of class car in dataset: 0.056991
Content percentage of class trafficsign in dataset: 0.005827
Content percentage of class sidewalk in dataset: 0.047263
Content percentage of class truck in dataset: 0.002633
Content percentage of class vegetation in dataset: 0.151332
Content percentage of class building in dataset: 0.191589
Content percentage of class bus in dataset: 0.003399
Content percentage of class terrain in dataset: 0.007273
Content percentage of class wall in dataset: 0.006409
Content percentage of class train in dataset: 0.000984
Content percentage of class sky in dataset: 0.029286
Content percentage of class fence in dataset: 0.007178
Content percentage of class motorcycle in dataset: 0.000695
Content percentage of class person in dataset: 0.011339
Content percentage of class pole in dataset: 0.012936
Content percentage of class bicycle in dataset: 0.006200
Content percentage of class rider in dataset: 0.001879
Content percentage of class trafficlight in dataset: 0.001730
Content percentage of class crap in dataset: 0.125827
Total amount of images: 500
SPECIFIC TO CITYSCAPES
Don't weigh the 'crap' class (key 255)
Content percentage of class crap in dataset: inf
SPECIFIC TO CITYSCAPES
Parsing directory /shared/datasets/cityscapes/bonnet_data/test
Total number of pixels: 15250000
Total number of pixels of class road: 0
Total number of pixels of class train: 0
Total number of pixels of class car: 0
Total number of pixels of class sky: 0
Total number of pixels of class trafficsign: 0
Total number of pixels of class fence: 0
Total number of pixels of class sidewalk: 0
Total number of pixels of class truck: 0
Total number of pixels of class person: 0
Total number of pixels of class vegetation: 0
Total number of pixels of class pole: 0
Total number of pixels of class building: 0
Total number of pixels of class bus: 0
Total number of pixels of class rider: 0
Total number of pixels of class bicycle: 0
Total number of pixels of class terrain: 0
Total number of pixels of class trafficlight: 0
Total number of pixels of class motorcycle: 0
Total number of pixels of class wall: 0
Total number of pixels of class crap: 15250000
Content percentage of class road in dataset: 0.000000
Content percentage of class car in dataset: 0.000000
Content percentage of class trafficsign in dataset: 0.000000
Content percentage of class sidewalk in dataset: 0.000000
Content percentage of class truck in dataset: 0.000000
Content percentage of class vegetation in dataset: 0.000000
Content percentage of class building in dataset: 0.000000
Content percentage of class bus in dataset: 0.000000
Content percentage of class terrain in dataset: 0.000000
Content percentage of class wall in dataset: 0.000000
Content percentage of class train in dataset: 0.000000
Content percentage of class sky in dataset: 0.000000
Content percentage of class fence in dataset: 0.000000
Content percentage of class motorcycle in dataset: 0.000000
Content percentage of class person in dataset: 0.000000
Content percentage of class pole in dataset: 0.000000
Content percentage of class bicycle in dataset: 0.000000
Content percentage of class rider in dataset: 0.000000
Content percentage of class trafficlight in dataset: 0.000000
Content percentage of class crap in dataset: 1.000000
Total amount of images: 1525
`Saving this iteration of training in ./log/lr_0.0001
Training model
Training network 100000 epochs (14875000 iterations at batch size 20)
Decaying learn rate by 1.010000 every 1 epochs (148 steps)
Traceback (most recent call last):
File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
terminate called without an active exception
terminate called recursively
Aborted (core dumped)
developer@linux:/shared/bonnet/train_py$
`
I think it is because a thread is break!
so I comment augment function as below:
`
if self.name == "ImgBufftrain":
# img, lbl = self.augment(img, lbl)
`
But! new error occurs!!!!!!!!!!! some information is below：
`Total number of parameters in network: 1871287
Reporting accuracy every 10 epochs
Saving this iteration of training in ./log/lr_0.0001
Training model
Training network 100000 epochs (14875000 iterations at batch size 20)
Decaying learn rate by 1.010000 every 1 epochs (148 steps)
Traceback (most recent call last):
File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
terminate called without an active exception
terminate called recursively
Aborted (core dumped)
`
a detail error infomation is:
`developer@linux:/shared/bonnet/train_py$ ./cnn_train.py -d cfg/cityscapes/data.yaml -n cfg/cityscapes/net_bonnet.yaml -t cfg/cityscapes/train_bonnet.yaml -l ./log
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
中文
INTERFACE:
data yaml: cfg/cityscapes/data.yaml
net yaml: cfg/cityscapes/net_bonnet.yaml
train yaml: cfg/cityscapes/train_bonnet.yaml
log dir ./log
model path None
model type iou
Commit hash (training version): b'7bf03b0'
Opening desired data file cfg/cityscapes/data.yaml
Opening desired net file cfg/cityscapes/net_bonnet.yaml
Opening desired train file cfg/cityscapes/train_bonnet.yaml
Copying files to ./log for further reference.
Training from scratch
Fetching dataset
Training with 1 GPU's
Training with batch size 20
DEVICE AVAIL: /device:CPU:0
DEVICE AVAIL: /device:GPU:0
Number of GPU's available is 1
This means 20 images per GPU
Data depth: 3
Parsing directory /shared/datasets/cityscapes/bonnet_data/train
Total number of pixels: 6239027200
Total number of pixels of class road: 2036049408
Total number of pixels of class train: 12863940
Total number of pixels of class car: 386502976
Total number of pixels of class sky: 221459584
Total number of pixels of class trafficsign: 30521320
Total number of pixels of class fence: 48487192
Total number of pixels of class sidewalk: 336031712
Total number of pixels of class truck: 14775009
Total number of pixels of class person: 67202392
Total number of pixels of class vegetation: 878734336
Total number of pixels of class pole: 67768880
Total number of pixels of class building: 1259774336
Total number of pixels of class bus: 12995807
Total number of pixels of class rider: 7444910
Total number of pixels of class bicycle: 22849672
Total number of pixels of class terrain: 63965240
Total number of pixels of class trafficlight: 11509943
Total number of pixels of class motorcycle: 5445904
Total number of pixels of class wall: 36211256
Total number of pixels of class crap: 718432320
Content percentage of class road in dataset: 0.326341
Content percentage of class car in dataset: 0.061949
Content percentage of class trafficsign in dataset: 0.004892
Content percentage of class sidewalk in dataset: 0.053860
Content percentage of class truck in dataset: 0.002368
Content percentage of class vegetation in dataset: 0.140845
Content percentage of class building in dataset: 0.201918
Content percentage of class bus in dataset: 0.002083
Content percentage of class terrain in dataset: 0.010252
Content percentage of class wall in dataset: 0.005804
Content percentage of class train in dataset: 0.002062
Content percentage of class sky in dataset: 0.035496
Content percentage of class fence in dataset: 0.007772
Content percentage of class motorcycle in dataset: 0.000873
Content percentage of class person in dataset: 0.010771
Content percentage of class pole in dataset: 0.010862
Content percentage of class bicycle in dataset: 0.003662
Content percentage of class rider in dataset: 0.001193
Content percentage of class trafficlight in dataset: 0.001845
Content percentage of class crap in dataset: 0.115151
Total amount of images: 2975
**** SPECIFIC TO CITYSCAPES ****
Don't weigh the 'crap' class (key 255)
Content percentage of class crap in dataset: inf
**** SPECIFIC TO CITYSCAPES ****
imhfetcher
Parsing directory /shared/datasets/cityscapes/bonnet_data/valid
Total number of pixels: 1048576000
Total number of pixels of class road: 345222080
Total number of pixels of class train: 1032100
Total number of pixels of class car: 59759312
Total number of pixels of class sky: 30708080
Total number of pixels of class trafficsign: 6110454
Total number of pixels of class fence: 7527026
Total number of pixels of class sidewalk: 49559052
Total number of pixels of class truck: 2760469
Total number of pixels of class person: 11890229
Total number of pixels of class vegetation: 158682896
Total number of pixels of class pole: 13564731
Total number of pixels of class building: 200895344
Total number of pixels of class bus: 3564221
Total number of pixels of class rider: 1970543
Total number of pixels of class bicycle: 6500852
Total number of pixels of class terrain: 7625936
Total number of pixels of class trafficlight: 1813749
Total number of pixels of class motorcycle: 728922
Total number of pixels of class wall: 6720678
Total number of pixels of class crap: 131939472
Content percentage of class road in dataset: 0.329229
Content percentage of class car in dataset: 0.056991
Content percentage of class trafficsign in dataset: 0.005827
Content percentage of class sidewalk in dataset: 0.047263
Content percentage of class truck in dataset: 0.002633
Content percentage of class vegetation in dataset: 0.151332
Content percentage of class building in dataset: 0.191589
Content percentage of class bus in dataset: 0.003399
Content percentage of class terrain in dataset: 0.007273
Content percentage of class wall in dataset: 0.006409
Content percentage of class train in dataset: 0.000984
Content percentage of class sky in dataset: 0.029286
Content percentage of class fence in dataset: 0.007178
Content percentage of class motorcycle in dataset: 0.000695
Content percentage of class person in dataset: 0.011339
Content percentage of class pole in dataset: 0.012936
Content percentage of class bicycle in dataset: 0.006200
Content percentage of class rider in dataset: 0.001879
Content percentage of class trafficlight in dataset: 0.001730
Content percentage of class crap in dataset: 0.125827
Total amount of images: 500
**** SPECIFIC TO CITYSCAPES ****
Don't weigh the 'crap' class (key 255)
Content percentage of class crap in dataset: inf
**** SPECIFIC TO CITYSCAPES ****
Parsing directory /shared/datasets/cityscapes/bonnet_data/test
Total number of pixels: 15250000
Total number of pixels of class road: 0
Total number of pixels of class train: 0
Total number of pixels of class car: 0
Total number of pixels of class sky: 0
Total number of pixels of class trafficsign: 0
Total number of pixels of class fence: 0
Total number of pixels of class sidewalk: 0
Total number of pixels of class truck: 0
Total number of pixels of class person: 0
Total number of pixels of class vegetation: 0
Total number of pixels of class pole: 0
Total number of pixels of class building: 0
Total number of pixels of class bus: 0
Total number of pixels of class rider: 0
Total number of pixels of class bicycle: 0
Total number of pixels of class terrain: 0
Total number of pixels of class trafficlight: 0
Total number of pixels of class motorcycle: 0
Total number of pixels of class wall: 0
Total number of pixels of class crap: 15250000
Content percentage of class road in dataset: 0.000000
Content percentage of class car in dataset: 0.000000
Content percentage of class trafficsign in dataset: 0.000000
Content percentage of class sidewalk in dataset: 0.000000
Content percentage of class truck in dataset: 0.000000
Content percentage of class vegetation in dataset: 0.000000
Content percentage of class building in dataset: 0.000000
Content percentage of class bus in dataset: 0.000000
Content percentage of class terrain in dataset: 0.000000
Content percentage of class wall in dataset: 0.000000
Content percentage of class train in dataset: 0.000000
Content percentage of class sky in dataset: 0.000000
Content percentage of class fence in dataset: 0.000000
Content percentage of class motorcycle in dataset: 0.000000
Content percentage of class person in dataset: 0.000000
Content percentage of class pole in dataset: 0.000000
Content percentage of class bicycle in dataset: 0.000000
Content percentage of class rider in dataset: 0.000000
Content percentage of class trafficlight in dataset: 0.000000
Content percentage of class crap in dataset: 1.000000
Total amount of images: 1525
Total number of parameters in network: 1871287
Reporting accuracy every 10 epochs
Saving this iteration of training in ./log/lr_0.0001
Training model
Training network 100000 epochs (14875000 iterations at batch size 20)
Decaying learn rate by 1.010000 every 1 epochs (148 steps)
Traceback (most recent call last):
File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
terminate called without an active exception
terminate called recursively
Aborted (core dumped)`
Hello,I want to train cityscapes datasets from scratch, But when I run:
developer@linux:/shared/bonnet/train_py$ ./cnn_train.py -d cfg/cityscapes/data.yaml -n cfg/cityscapes/net_bonnet.yaml -t cfg/cityscapes/train_bonnet.yaml -l ./logsome opencv errors occurs:
`developer@linux:/shared/bonnet/train_py$ ./cnn_train.py -d cfg/cityscapes/data.yaml -n cfg/cityscapes/net_bonnet.yaml -t cfg/cityscapes/train_bonnet.yaml -l ./log WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version. Instructions for updating: Use the retry module or similar alternatives.
INTERFACE: data yaml: cfg/cityscapes/data.yaml net yaml: cfg/cityscapes/net_bonnet.yaml train yaml: cfg/cityscapes/train_bonnet.yaml log dir ./log model path None model type iou
Commit hash (training version): b'7bf03b0'
Opening desired data file cfg/cityscapes/data.yaml Opening desired net file cfg/cityscapes/net_bonnet.yaml Opening desired train file cfg/cityscapes/train_bonnet.yaml Copying files to ./log for further reference. Training from scratch Fetching dataset Training with 1 GPU's Training with batch size 2 DEVICE AVAIL: /device:CPU:0 DEVICE AVAIL: /device:GPU:0 Number of GPU's available is 1 This means 2 images per GPU Data depth: 3 Parsing directory /shared/datasets/cityscapes/bonnet_data/train Total number of pixels: 6239027200 Total number of pixels of class road: 2036049408 Total number of pixels of class train: 12863940 Total number of pixels of class car: 386502976 Total number of pixels of class sky: 221459584 Total number of pixels of class trafficsign: 30521320 Total number of pixels of class fence: 48487192 Total number of pixels of class sidewalk: 336031712 Total number of pixels of class truck: 14775009 Total number of pixels of class person: 67202392 Total number of pixels of class vegetation: 878734336 Total number of pixels of class pole: 67768880 Total number of pixels of class building: 1259774336 Total number of pixels of class bus: 12995807 Total number of pixels of class rider: 7444910 Total number of pixels of class bicycle: 22849672 Total number of pixels of class terrain: 63965240 Total number of pixels of class trafficlight: 11509943 Total number of pixels of class motorcycle: 5445904 Total number of pixels of class wall: 36211256 Total number of pixels of class crap: 718432320 Content percentage of class road in dataset: 0.326341 Content percentage of class car in dataset: 0.061949 Content percentage of class trafficsign in dataset: 0.004892 Content percentage of class sidewalk in dataset: 0.053860 Content percentage of class truck in dataset: 0.002368 Content percentage of class vegetation in dataset: 0.140845 Content percentage of class building in dataset: 0.201918 Content percentage of class bus in dataset: 0.002083 Content percentage of class terrain in dataset: 0.010252 Content percentage of class wall in dataset: 0.005804 Content percentage of class train in dataset: 0.002062 Content percentage of class sky in dataset: 0.035496 Content percentage of class fence in dataset: 0.007772 Content percentage of class motorcycle in dataset: 0.000873 Content percentage of class person in dataset: 0.010771 Content percentage of class pole in dataset: 0.010862 Content percentage of class bicycle in dataset: 0.003662 Content percentage of class rider in dataset: 0.001193 Content percentage of class trafficlight in dataset: 0.001845 Content percentage of class crap in dataset: 0.115151 Total amount of images: 2975
OpenCV Error: Assertion failed (dims <= 2 && step[0] > 0) in locateROI, file /io/opencv/modules/core/src/matrix.cpp, line 991 Exception in thread ImgBufftrain: Traceback (most recent call last): File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/shared/bonnet/train_py/dataset/abstract_dataset.py", line 84, in run img, lbl = self.augment(img, lbl) File "/shared/bonnet/train_py/dataset/abstract_dataset.py", line 68, in augment img = cv2.blur(img,(ksize,ksize)) cv2.error: /io/opencv/modules/core/src/matrix.cpp:991: error: (-215) dims <= 2 && step[0] > 0 in function locateROI
Parsing directory /shared/datasets/cityscapes/bonnet_data/valid Total number of pixels: 1048576000 Total number of pixels of class road: 345222080 Total number of pixels of class train: 1032100 Total number of pixels of class car: 59759312 Total number of pixels of class sky: 30708080 Total number of pixels of class trafficsign: 6110454 Total number of pixels of class fence: 7527026 Total number of pixels of class sidewalk: 49559052 Total number of pixels of class truck: 2760469 Total number of pixels of class person: 11890229 Total number of pixels of class vegetation: 158682896 Total number of pixels of class pole: 13564731 Total number of pixels of class building: 200895344 Total number of pixels of class bus: 3564221 Total number of pixels of class rider: 1970543 Total number of pixels of class bicycle: 6500852 Total number of pixels of class terrain: 7625936 Total number of pixels of class trafficlight: 1813749 Total number of pixels of class motorcycle: 728922 Total number of pixels of class wall: 6720678 Total number of pixels of class crap: 131939472 Content percentage of class road in dataset: 0.329229 Content percentage of class car in dataset: 0.056991 Content percentage of class trafficsign in dataset: 0.005827 Content percentage of class sidewalk in dataset: 0.047263 Content percentage of class truck in dataset: 0.002633 Content percentage of class vegetation in dataset: 0.151332 Content percentage of class building in dataset: 0.191589 Content percentage of class bus in dataset: 0.003399 Content percentage of class terrain in dataset: 0.007273 Content percentage of class wall in dataset: 0.006409 Content percentage of class train in dataset: 0.000984 Content percentage of class sky in dataset: 0.029286 Content percentage of class fence in dataset: 0.007178 Content percentage of class motorcycle in dataset: 0.000695 Content percentage of class person in dataset: 0.011339 Content percentage of class pole in dataset: 0.012936 Content percentage of class bicycle in dataset: 0.006200 Content percentage of class rider in dataset: 0.001879 Content percentage of class trafficlight in dataset: 0.001730 Content percentage of class crap in dataset: 0.125827 Total amount of images: 500
SPECIFIC TO CITYSCAPES Don't weigh the 'crap' class (key 255) Content percentage of class crap in dataset: inf SPECIFIC TO CITYSCAPES Parsing directory /shared/datasets/cityscapes/bonnet_data/test Total number of pixels: 15250000 Total number of pixels of class road: 0 Total number of pixels of class train: 0 Total number of pixels of class car: 0 Total number of pixels of class sky: 0 Total number of pixels of class trafficsign: 0 Total number of pixels of class fence: 0 Total number of pixels of class sidewalk: 0 Total number of pixels of class truck: 0 Total number of pixels of class person: 0 Total number of pixels of class vegetation: 0 Total number of pixels of class pole: 0 Total number of pixels of class building: 0 Total number of pixels of class bus: 0 Total number of pixels of class rider: 0 Total number of pixels of class bicycle: 0 Total number of pixels of class terrain: 0 Total number of pixels of class trafficlight: 0 Total number of pixels of class motorcycle: 0 Total number of pixels of class wall: 0 Total number of pixels of class crap: 15250000 Content percentage of class road in dataset: 0.000000 Content percentage of class car in dataset: 0.000000 Content percentage of class trafficsign in dataset: 0.000000 Content percentage of class sidewalk in dataset: 0.000000 Content percentage of class truck in dataset: 0.000000 Content percentage of class vegetation in dataset: 0.000000 Content percentage of class building in dataset: 0.000000 Content percentage of class bus in dataset: 0.000000 Content percentage of class terrain in dataset: 0.000000 Content percentage of class wall in dataset: 0.000000 Content percentage of class train in dataset: 0.000000 Content percentage of class sky in dataset: 0.000000 Content percentage of class fence in dataset: 0.000000 Content percentage of class motorcycle in dataset: 0.000000 Content percentage of class person in dataset: 0.000000 Content percentage of class pole in dataset: 0.000000 Content percentage of class bicycle in dataset: 0.000000 Content percentage of class rider in dataset: 0.000000 Content percentage of class trafficlight in dataset: 0.000000 Content percentage of class crap in dataset: 1.000000 Total amount of images: 1525
SPECIFIC TO CITYSCAPES Don't weigh the 'crap' class (key 255) SPECIFIC TO CITYSCAPES Successfully imported datasets Train data samples: 2975 Validation data samples: 500 Test data samples: 1525 Initializing network TRAINING GRAPH Building graph encoder downsample1 W: [5, 5, 3, 13] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True downsample2 W: [5, 5, 16, 32] Train: True non-bt-1 W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True non-bt-2 W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True non-bt-3 W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True non-bt-4 W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True downsample3 W: [5, 5, 48, 32] Train: True non-bt-1 W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True non-bt-2 W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True non-bt-3 W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True non-bt-4 W: [7, 1, 80, 80] Train: True `
`Saving this iteration of training in ./log/lr_0.0001 Training model Training network 100000 epochs (14875000 iterations at batch size 20) Decaying learn rate by 1.010000 every 1 epochs (148 steps) Traceback (most recent call last):
File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
terminate called without an active exception
terminate called recursively
Aborted (core dumped)
developer@linux:/shared/bonnet/train_py$
`
I think it is because a thread is break! so I comment augment function as below:
`
if self.name == "ImgBufftrain":
`
But! new error occurs!!!!!!!!!!! some information is below：
`Total number of parameters in network: 1871287
Reporting accuracy every 10 epochs
Saving this iteration of training in ./log/lr_0.0001
Training model Training network 100000 epochs (14875000 iterations at batch size 20) Decaying learn rate by 1.010000 every 1 epochs (148 steps)
Traceback (most recent call last): File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
terminate called without an active exception
terminate called recursively
Aborted (core dumped) `
a detail error infomation is:
`developer@linux:/shared/bonnet/train_py$ ./cnn_train.py -d cfg/cityscapes/data.yaml -n cfg/cityscapes/net_bonnet.yaml -t cfg/cityscapes/train_bonnet.yaml -l ./log WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version. Instructions for updating: Use the retry module or similar alternatives. 中文
INTERFACE: data yaml: cfg/cityscapes/data.yaml net yaml: cfg/cityscapes/net_bonnet.yaml train yaml: cfg/cityscapes/train_bonnet.yaml log dir ./log model path None model type iou
Commit hash (training version): b'7bf03b0'
Opening desired data file cfg/cityscapes/data.yaml Opening desired net file cfg/cityscapes/net_bonnet.yaml Opening desired train file cfg/cityscapes/train_bonnet.yaml Copying files to ./log for further reference. Training from scratch Fetching dataset Training with 1 GPU's Training with batch size 20 DEVICE AVAIL: /device:CPU:0 DEVICE AVAIL: /device:GPU:0 Number of GPU's available is 1 This means 20 images per GPU Data depth: 3
Parsing directory /shared/datasets/cityscapes/bonnet_data/train Total number of pixels: 6239027200 Total number of pixels of class road: 2036049408 Total number of pixels of class train: 12863940 Total number of pixels of class car: 386502976 Total number of pixels of class sky: 221459584 Total number of pixels of class trafficsign: 30521320 Total number of pixels of class fence: 48487192 Total number of pixels of class sidewalk: 336031712 Total number of pixels of class truck: 14775009 Total number of pixels of class person: 67202392 Total number of pixels of class vegetation: 878734336 Total number of pixels of class pole: 67768880 Total number of pixels of class building: 1259774336 Total number of pixels of class bus: 12995807 Total number of pixels of class rider: 7444910 Total number of pixels of class bicycle: 22849672 Total number of pixels of class terrain: 63965240 Total number of pixels of class trafficlight: 11509943 Total number of pixels of class motorcycle: 5445904 Total number of pixels of class wall: 36211256 Total number of pixels of class crap: 718432320 Content percentage of class road in dataset: 0.326341 Content percentage of class car in dataset: 0.061949 Content percentage of class trafficsign in dataset: 0.004892 Content percentage of class sidewalk in dataset: 0.053860 Content percentage of class truck in dataset: 0.002368 Content percentage of class vegetation in dataset: 0.140845 Content percentage of class building in dataset: 0.201918 Content percentage of class bus in dataset: 0.002083 Content percentage of class terrain in dataset: 0.010252 Content percentage of class wall in dataset: 0.005804 Content percentage of class train in dataset: 0.002062 Content percentage of class sky in dataset: 0.035496 Content percentage of class fence in dataset: 0.007772 Content percentage of class motorcycle in dataset: 0.000873 Content percentage of class person in dataset: 0.010771 Content percentage of class pole in dataset: 0.010862 Content percentage of class bicycle in dataset: 0.003662 Content percentage of class rider in dataset: 0.001193 Content percentage of class trafficlight in dataset: 0.001845 Content percentage of class crap in dataset: 0.115151 Total amount of images: 2975
**** SPECIFIC TO CITYSCAPES **** Don't weigh the 'crap' class (key 255) Content percentage of class crap in dataset: inf **** SPECIFIC TO CITYSCAPES **** imhfetcher
Parsing directory /shared/datasets/cityscapes/bonnet_data/valid Total number of pixels: 1048576000 Total number of pixels of class road: 345222080 Total number of pixels of class train: 1032100 Total number of pixels of class car: 59759312 Total number of pixels of class sky: 30708080 Total number of pixels of class trafficsign: 6110454 Total number of pixels of class fence: 7527026 Total number of pixels of class sidewalk: 49559052 Total number of pixels of class truck: 2760469 Total number of pixels of class person: 11890229 Total number of pixels of class vegetation: 158682896 Total number of pixels of class pole: 13564731 Total number of pixels of class building: 200895344 Total number of pixels of class bus: 3564221 Total number of pixels of class rider: 1970543 Total number of pixels of class bicycle: 6500852 Total number of pixels of class terrain: 7625936 Total number of pixels of class trafficlight: 1813749 Total number of pixels of class motorcycle: 728922 Total number of pixels of class wall: 6720678 Total number of pixels of class crap: 131939472 Content percentage of class road in dataset: 0.329229 Content percentage of class car in dataset: 0.056991 Content percentage of class trafficsign in dataset: 0.005827 Content percentage of class sidewalk in dataset: 0.047263 Content percentage of class truck in dataset: 0.002633 Content percentage of class vegetation in dataset: 0.151332 Content percentage of class building in dataset: 0.191589 Content percentage of class bus in dataset: 0.003399 Content percentage of class terrain in dataset: 0.007273 Content percentage of class wall in dataset: 0.006409 Content percentage of class train in dataset: 0.000984 Content percentage of class sky in dataset: 0.029286 Content percentage of class fence in dataset: 0.007178 Content percentage of class motorcycle in dataset: 0.000695 Content percentage of class person in dataset: 0.011339 Content percentage of class pole in dataset: 0.012936 Content percentage of class bicycle in dataset: 0.006200 Content percentage of class rider in dataset: 0.001879 Content percentage of class trafficlight in dataset: 0.001730 Content percentage of class crap in dataset: 0.125827 Total amount of images: 500
**** SPECIFIC TO CITYSCAPES **** Don't weigh the 'crap' class (key 255) Content percentage of class crap in dataset: inf **** SPECIFIC TO CITYSCAPES ****
Parsing directory /shared/datasets/cityscapes/bonnet_data/test Total number of pixels: 15250000 Total number of pixels of class road: 0 Total number of pixels of class train: 0 Total number of pixels of class car: 0 Total number of pixels of class sky: 0 Total number of pixels of class trafficsign: 0 Total number of pixels of class fence: 0 Total number of pixels of class sidewalk: 0 Total number of pixels of class truck: 0 Total number of pixels of class person: 0 Total number of pixels of class vegetation: 0 Total number of pixels of class pole: 0 Total number of pixels of class building: 0 Total number of pixels of class bus: 0 Total number of pixels of class rider: 0 Total number of pixels of class bicycle: 0 Total number of pixels of class terrain: 0 Total number of pixels of class trafficlight: 0 Total number of pixels of class motorcycle: 0 Total number of pixels of class wall: 0 Total number of pixels of class crap: 15250000 Content percentage of class road in dataset: 0.000000 Content percentage of class car in dataset: 0.000000 Content percentage of class trafficsign in dataset: 0.000000 Content percentage of class sidewalk in dataset: 0.000000 Content percentage of class truck in dataset: 0.000000 Content percentage of class vegetation in dataset: 0.000000 Content percentage of class building in dataset: 0.000000 Content percentage of class bus in dataset: 0.000000 Content percentage of class terrain in dataset: 0.000000 Content percentage of class wall in dataset: 0.000000 Content percentage of class train in dataset: 0.000000 Content percentage of class sky in dataset: 0.000000 Content percentage of class fence in dataset: 0.000000 Content percentage of class motorcycle in dataset: 0.000000 Content percentage of class person in dataset: 0.000000 Content percentage of class pole in dataset: 0.000000 Content percentage of class bicycle in dataset: 0.000000 Content percentage of class rider in dataset: 0.000000 Content percentage of class trafficlight in dataset: 0.000000 Content percentage of class crap in dataset: 1.000000 Total amount of images: 1525
**** SPECIFIC TO CITYSCAPES **** Don't weigh the 'crap' class (key 255) Content percentage of class crap in dataset: inf **** SPECIFIC TO CITYSCAPES **** Successfully imported datasets Train data samples: 2975 Validation data samples: 500 Test data samples: 1525 Initializing network **** TRAINING GRAPH **** GRAPH GPU:0 Building graph encoder downsample1 W: [5, 5, 3, 13] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True downsample2 W: [5, 5, 16, 32] Train: True non-bt-1 W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True non-bt-2 W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True non-bt-3 W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True non-bt-4 W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True W: [5, 1, 48, 48] Train: True W: [1, 5, 48, 48] Train: True downsample3 W: [5, 5, 48, 32] Train: True non-bt-1 W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True non-bt-2 W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True non-bt-3 W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True non-bt-4 W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True godeep non-bt-1 W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True non-bt-2 W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True non-bt-3 W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True non-bt-4 W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True W: [7, 1, 80, 80] Train: True W: [1, 7, 80, 80] Train: True ============= End of encoder =============== size of code: [20, 80, 48, 96] =========== Beginning of decoder============ decoder upsample unpool1 W: [2, 2, 80, 48] Train: True W: [3, 1, 48, 48] Train: True W: [1, 3, 48, 48] Train: True W: [3, 1, 48, 48] Train: True W: [1, 3, 48, 48] Train: True W: [3, 1, 48, 48] Train: True W: [1, 3, 48, 48] Train: True W: [3, 1, 48, 48] Train: True W: [1, 3, 48, 48] Train: True W: [3, 1, 48, 48] Train: True W: [1, 3, 48, 48] Train: True W: [3, 1, 48, 48] Train: True W: [1, 3, 48, 48] Train: True W: [3, 1, 48, 48] Train: True W: [1, 3, 48, 48] Train: True W: [3, 1, 48, 48] Train: True W: [1, 3, 48, 48] Train: True unpool2 W: [2, 2, 48, 32] Train: True W: [3, 1, 32, 32] Train: True W: [1, 3, 32, 32] Train: True W: [3, 1, 32, 32] Train: True W: [1, 3, 32, 32] Train: True W: [3, 1, 32, 32] Train: True W: [1, 3, 32, 32] Train: True W: [3, 1, 32, 32] Train: True W: [1, 3, 32, 32] Train: True W: [3, 1, 32, 32] Train: True W: [1, 3, 32, 32] Train: True W: [3, 1, 32, 32] Train: True W: [1, 3, 32, 32] Train: True W: [3, 1, 32, 32] Train: True W: [1, 3, 32, 32] Train: True W: [3, 1, 32, 32] Train: True W: [1, 3, 32, 32] Train: True unpool3 W: [2, 2, 32, 16] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True W: [3, 1, 16, 16] Train: True W: [1, 3, 16, 16] Train: True W: [1, 1, 16, 20] Train: True b: [20] Train: True Defining loss function
Weights for loss function (1/log(frec(c)+e)): [ 3.36258268 14.03326797 4.98947144 39.25167084 36.5057373 32.89967728 46.27554703 40.67154694 6.70474911 33.55268478 18.51486206 32.99533463 47.68305206 12.69611359 45.20449448 45.78203583 45.8253479 48.40744781 42.75931931 0. ] Weight decay: 1e-06 Using tensorflow gradients
**** TESTING GRAPH ***** GRAPH GPU:0 Building graph encoder downsample1 W: [5, 5, 3, 13] Train: False W: [3, 1, 16, 16] Train: False W: [1, 3, 16, 16] Train: False W: [3, 1, 16, 16] Train: False W: [1, 3, 16, 16] Train: False W: [3, 1, 16, 16] Train: False W: [1, 3, 16, 16] Train: False W: [3, 1, 16, 16] Train: False W: [1, 3, 16, 16] Train: False downsample2 W: [5, 5, 16, 32] Train: False non-bt-1 W: [5, 1, 48, 48] Train: False W: [1, 5, 48, 48] Train: False W: [5, 1, 48, 48] Train: False W: [1, 5, 48, 48] Train: False non-bt-2 W: [5, 1, 48, 48] Train: False W: [1, 5, 48, 48] Train: False W: [5, 1, 48, 48] Train: False W: [1, 5, 48, 48] Train: False non-bt-3 W: [5, 1, 48, 48] Train: False W: [1, 5, 48, 48] Train: False W: [5, 1, 48, 48] Train: False W: [1, 5, 48, 48] Train: False non-bt-4 W: [5, 1, 48, 48] Train: False W: [1, 5, 48, 48] Train: False W: [5, 1, 48, 48] Train: False W: [1, 5, 48, 48] Train: False downsample3 W: [5, 5, 48, 32] Train: False non-bt-1 W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False non-bt-2 W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False non-bt-3 W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False non-bt-4 W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False godeep non-bt-1 W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False non-bt-2 W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False non-bt-3 W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False non-bt-4 W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False W: [7, 1, 80, 80] Train: False W: [1, 7, 80, 80] Train: False ============= End of encoder =============== size of code: [20, 80, 48, 96] =========== Beginning of decoder============ decoder upsample unpool1 W: [2, 2, 80, 48] Train: False W: [3, 1, 48, 48] Train: False W: [1, 3, 48, 48] Train: False W: [3, 1, 48, 48] Train: False W: [1, 3, 48, 48] Train: False W: [3, 1, 48, 48] Train: False W: [1, 3, 48, 48] Train: False W: [3, 1, 48, 48] Train: False W: [1, 3, 48, 48] Train: False W: [3, 1, 48, 48] Train: False W: [1, 3, 48, 48] Train: False W: [3, 1, 48, 48] Train: False W: [1, 3, 48, 48] Train: False W: [3, 1, 48, 48] Train: False W: [1, 3, 48, 48] Train: False W: [3, 1, 48, 48] Train: False W: [1, 3, 48, 48] Train: False unpool2 W: [2, 2, 48, 32] Train: False W: [3, 1, 32, 32] Train: False W: [1, 3, 32, 32] Train: False W: [3, 1, 32, 32] Train: False W: [1, 3, 32, 32] Train: False W: [3, 1, 32, 32] Train: False W: [1, 3, 32, 32] Train: False W: [3, 1, 32, 32] Train: False W: [1, 3, 32, 32] Train: False W: [3, 1, 32, 32] Train: False W: [1, 3, 32, 32] Train: False W: [3, 1, 32, 32] Train: False W: [1, 3, 32, 32] Train: False W: [3, 1, 32, 32] Train: False W: [1, 3, 32, 32] Train: False W: [3, 1, 32, 32] Train: False W: [1, 3, 32, 32] Train: False unpool3 W: [2, 2, 32, 16] Train: False W: [3, 1, 16, 16] Train: False W: [1, 3, 16, 16] Train: False W: [3, 1, 16, 16] Train: False W: [1, 3, 16, 16] Train: False W: [3, 1, 16, 16] Train: False W: [1, 3, 16, 16] Train: False W: [3, 1, 16, 16] Train: False W: [1, 3, 16, 16] Train: False W: [1, 1, 16, 20] Train: False b: [20] Train: False
Total number of parameters in network: 1871287 Reporting accuracy every 10 epochs Saving this iteration of training in ./log/lr_0.0001 Training model Training network 100000 epochs (14875000 iterations at batch size 20) Decaying learn rate by 1.010000 every 1 epochs (148 steps) Traceback (most recent call last): File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
terminate called without an active exception
terminate called recursively
Aborted (core dumped)`
I hope you can help me, sincerely！