Durdle
commented
4 years ago I have a similar problem. Have you found solution?
Mine is this:
2020-09-21 23:17:53 - Using CUDA...
2020-09-21 23:17:53 - Namespace(balance_data=False, base_net=None, base_net_lr=0.001, batch_size=4, checkpoint_folder='models/wood', dataset_type='voc', datasets=['data/wood'], debug_steps=10, extra_layers_lr=None, freeze_base_net=False, freeze_net=False, gamma=0.1, lr=0.01, mb2_width_mult=1.0, milestones='80,100', momentum=0.9, net='mb1-ssd', num_epochs=5, num_workers=2, pretrained_ssd='models/mobilenet-v1-ssd-mp-0_675.pth', resume=None, scheduler='cosine', t_max=100, use_cuda=True, validation_epochs=1, weight_decay=0.0005)
2020-09-21 23:17:53 - Prepare training datasets.
2020-09-21 23:17:53 - No labels file, using default VOC classes.
2020-09-21 23:17:53 - Stored labels into file models/wood/labels.txt.
2020-09-21 23:17:53 - Train dataset size: 50
2020-09-21 23:17:53 - Prepare Validation datasets.
2020-09-21 23:17:53 - No labels file, using default VOC classes.
2020-09-21 23:17:53 - Validation dataset size: 10
2020-09-21 23:17:53 - Build network.
2020-09-21 23:17:53 - Init from pretrained ssd models/mobilenet-v1-ssd-mp-0_675.pth
2020-09-21 23:17:53 - Took 0.15 seconds to load the model.
2020-09-21 23:17:56 - Learning rate: 0.01, Base net learning rate: 0.001, Extra Layers learning rate: 0.01.
2020-09-21 23:17:56 - Uses CosineAnnealingLR scheduler.
2020-09-21 23:17:56 - Start training from epoch 0.
/home/nvidia/.local/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Traceback (most recent call last):
File "train_ssd.py", line 343, in
dusty-nv
Johnson-Su
MohamedBenAli91
Li-camera-isp-test
eltonfernando
tmichals
I have followed the tutorial for training your own dateset for the SSD. I have been using this to recognise different plants. However once I run
python3 train_ssd.py --dataset-type=voc --data=(my dataset location) I get this what is below. I had edited the voc_dataset.py to allow captials in the names.
2020-09-07 16:24:11 - Using CUDA... 2020-09-07 16:24:11 - Namespace(balance_data=False, base_net=None, base_net_lr=0.001, batch_size=4, checkpoint_folder='models/', dataset_type='voc', datasets=['/home/farmbot/Desktop/Farmbot_Data'], debug_steps=10, extra_layers_lr=None, freeze_base_net=False, freeze_net=False, gamma=0.1, lr=0.01, mb2_width_mult=1.0, milestones='80,100', momentum=0.9, net='mb1-ssd', num_epochs=30, num_workers=2, pretrained_ssd='models/mobilenet-v1-ssd-mp-0_675.pth', resume=None, scheduler='cosine', t_max=100, use_cuda=True, validation_epochs=1, weight_decay=0.0005) 2020-09-07 16:24:11 - Prepare training datasets. 2020-09-07 16:24:11 - VOC Labels read from file: ('BACKGROUND', 'Young Beet', 'Young Khol Rabi', 'Young Radishe, Grown Beet', 'Grown Khol Rabi', 'Young Radishe') 2020-09-07 16:24:11 - Stored labels into file models/labels.txt. 2020-09-07 16:24:11 - Train dataset size: 82 2020-09-07 16:24:11 - Prepare Validation datasets. 2020-09-07 16:24:11 - VOC Labels read from file: ('BACKGROUND', 'Young Beet', 'Young Khol Rabi', 'Young Radishe, Grown Beet', 'Grown Khol Rabi', 'Young Radishe') 2020-09-07 16:24:11 - Validation dataset size: 0 2020-09-07 16:24:11 - Build network. 2020-09-07 16:24:11 - Init from pretrained ssd models/mobilenet-v1-ssd-mp-0_675.pth 2020-09-07 16:24:12 - Took 0.52 seconds to load the model. 2020-09-07 16:24:24 - Learning rate: 0.01, Base net learning rate: 0.001, Extra Layers learning rate: 0.01. 2020-09-07 16:24:24 - Uses CosineAnnealingLR scheduler. 2020-09-07 16:24:24 - Start training from epoch 0. /home/farmbot/.local/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning) /home/farmbot/jetson-inference/python/training/detection/ssd/vision/transforms/transforms.py:247: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray mode = random.choice(self.sample_options) /home/farmbot/jetson-inference/python/training/detection/ssd/vision/transforms/transforms.py:247: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray mode = random.choice(self.sample_options) Traceback (most recent call last): File "train_ssd.py", line 343, in <module> device=DEVICE, debug_steps=args.debug_steps, epoch=epoch) File "train_ssd.py", line 113, in train for i, data in enumerate(loader): File "/home/farmbot/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 354, in __next__ data = self._next_data() File "/home/farmbot/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 980, in _next_data return self._process_data(data) File "/home/farmbot/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1005, in _process_data data.reraise() File "/home/farmbot/.local/lib/python3.6/site-packages/torch/_utils.py", line 395, in reraise raise self.exc_type(msg) IndexError: Caught IndexError in DataLoader worker process 0. Original Traceback (most recent call last): File "/home/farmbot/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 185, in _worker_loop data = fetcher.fetch(index) File "/home/farmbot/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/farmbot/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp> data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/farmbot/.local/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 207, in __getitem__ return self.datasets[dataset_idx][sample_idx] File "/home/farmbot/jetson-inference/python/training/detection/ssd/vision/datasets/voc_dataset.py", line 64, in __getitem__ image, boxes, labels = self.transform(image, boxes, labels) File "/home/farmbot/jetson-inference/python/training/detection/ssd/vision/ssd/data_preprocessing.py", line 34, in __call__ return self.augment(img, boxes, labels) File "/home/farmbot/jetson-inference/python/training/detection/ssd/vision/transforms/transforms.py", line 55, in __call__ img, boxes, labels = t(img, boxes, labels) File "/home/farmbot/jetson-inference/python/training/detection/ssd/vision/transforms/transforms.py", line 275, in __call__ overlap = jaccard_numpy(boxes, rect) File "/home/farmbot/jetson-inference/python/training/detection/ssd/vision/transforms/transforms.py", line 30, in jaccard_numpy inter = intersect(box_a, box_b) File "/home/farmbot/jetson-inference/python/training/detection/ssd/vision/transforms/transforms.py", line 13, in intersect max_xy = np.minimum(box_a[:, 2:], box_b[2:]) IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed