xiaosongshine
commented
4 years ago I trained coco dataset is ok. When I train my Data used json label like below Program Pause befor train some batch:
[{ "gt_class": [ 1 ], "gt_bbox": [ [ 838.75, 282.5, 854.5, 300.5 ] ], "flipped": false, "h": 890, "w": 869, "image_url": "/home/yss/Documents/Cervical_Cancer_deal_data/images_val/pos_6_T2019_572_img_roi_min5_0.jpg", "im_id": 1 },,, ]train log:
10-17 10:54:28 Epoch[0] Batch [10] Speed: 9.54 samples/sec Train-RpnAcc=0.865128, RpnL1=1.197431, RcnnAcc=0.979368, RcnnL1=0.448382, 10-17 10:54:34 Epoch[0] Batch [20] Speed: 9.56 samples/sec Train-RpnAcc=0.852790, RpnL1=1.057644, RcnnAcc=0.980208, RcnnL1=0.283550, 10-17 10:54:39 Epoch[0] Batch [30] Speed: 9.79 samples/sec Train-RpnAcc=0.862954, RpnL1=0.997937, RcnnAcc=0.983203, RcnnL1=0.398492, 10-17 10:54:44 Epoch[0] Batch [40] Speed: 9.64 samples/sec Train-RpnAcc=0.862100, RpnL1=0.907419, RcnnAcc=0.984318, RcnnL1=0.413099, [10:54:46] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:97: Running performance tests to find the best convolution algorithm, this can take a while... (set the environment variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) Exception in thread Thread-7: Traceback (most recent call last): File "/home/yss/yes/envs/MX_DET/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/home/yss/yes/envs/MX_DET/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home/yss/simpledet/core/detection_input.py", line 732, in worker data_batch[name] = np.ascontiguousarray(np.stack([r[name] for r in records])) File "<__array_function__ internals>", line 6, in stack File "/home/yss/.local/lib/python3.6/site-packages/numpy/core/shape_base.py", line 425, in stack raise ValueError('all input arrays must have the same shape') ValueError: all input arrays must have the same shape Exception in thread Thread-9: Traceback (most recent call last): File "/home/yss/yes/envs/MX_DET/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/home/yss/yes/envs/MX_DET/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home/yss/simpledet/core/detection_input.py", line 732, in worker data_batch[name] = np.ascontiguousarray(np.stack([r[name] for r in records])) File "<__array_function__ internals>", line 6, in stack File "/home/yss/.local/lib/python3.6/site-packages/numpy/core/shape_base.py", line 425, in stack raise ValueError('all input arrays must have the same shape') ValueError: all input arrays must have the same shape 10-17 10:54:54 Epoch[0] Batch [50] Speed: 4.72 samples/sec Train-RpnAcc=0.864001, RpnL1=0.876073, RcnnAcc=0.985248, RcnnL1=0.436969,Then Program Pause Volatile GPU-Util into 0% I have to used "Ctrl + c" to stop it then i got below
^CTraceback (most recent call last): File "detection_train_json.py", line 296, in <module> train_net(parse_args()) File "detection_train_json.py", line 278, in train_net profile=profile File "/home/yss/simpledet/core/detection_module.py", line 1014, in fit next_data_batch = next(data_iter) File "/home/yss/.local/lib/python3.6/site-packages/mxnet/io/io.py", line 230, in __next__ return self.next() File "/home/yss/simpledet/core/detection_input.py", line 863, in next return self.__loader.next() File "/home/yss/simpledet/core/detection_input.py", line 770, in next result = first_non_empty_iter.next() File "/home/yss/simpledet/core/detection_input.py", line 712, in next result = self.load_batch() File "/home/yss/simpledet/core/detection_input.py", line 702, in load_batch result = self.result_queue.get() File "/home/yss/yes/envs/MX_DET/lib/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/home/yss/yes/envs/MX_DET/lib/python3.6/threading.py", line 295, in wait waiter.acquire() KeyboardInterruptI used cv2 test read my all imagas,have no error.
I also used test_loader.py read my db,Program Pause and got this:
Exception in thread Thread-8: Traceback (most recent call last): File "/home/yss/yes/envs/MX_DET/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/home/yss/yes/envs/MX_DET/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home/yss/simpledet/core/detection_input.py", line 732, in worker data_batch[name] = np.ascontiguousarray(np.stack([r[name] for r in records])) File "<__array_function__ internals>", line 6, in stack File "/home/yss/.local/lib/python3.6/site-packages/numpy/core/shape_base.py", line 425, in stack raise ValueError('all input arrays must have the same shape') ValueError: all input arrays must have the same shape Exception in thread Thread-1: Traceback (most recent call last): File "/home/yss/yes/envs/MX_DET/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/home/yss/yes/envs/MX_DET/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home/yss/simpledet/core/detection_input.py", line 728, in worker trans.apply(roi_record) File "/home/yss/simpledet/core/detection_input.py", line 41, in apply axis=1) File "<__array_function__ internals>", line 6, in concatenate numpy.AxisError: axis 1 is out of bounds for array of dimension 1
because I mixup w and h. when it change it,I solve this problem,
I trained coco dataset is ok. When I train my Data used json label like below Program Pause befor train some batch:
train log:
Then Program Pause Volatile GPU-Util into 0% I have to used "Ctrl + c" to stop it then i got below
I used cv2 test read my all imagas,have no error.
I also used test_loader.py read my db,Program Pause and got this: