search

idekazuki / diary

記録
0 stars 1 forks source link

error faster rcnn for epic-kitchen #89

Open idekazuki opened 4 years ago

idekazuki commented 4 years ago 
Traceback (most recent call last):
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 337, in reduce_storage
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/multiprocessing/reduction.py", line 194, in DupFd
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/multiprocessing/resource_sharer.py", line 48, in __init__
OSError: [Errno 24] Too many open files
^CTraceback (most recent call last):
  File "faster_rcnn.py", line 48, in <module>
    train_one_epoch(model, optimizer, train_data_loader, device, epoch, print_freq=10)
  File "/host/space0/ide-k/mygit/Detectron4epic/engine.py", line 26, in train_one_epoch
    for images, targets in metric_logger.log_every(data_loader, print_freq, header):
  File "/host/space0/ide-k/mygit/Detectron4epic/utils.py", line 209, in log_every
    for obj in iterable:
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/multiprocessing/queues.py", line 104, in get
    if not self._poll(timeout):
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/multiprocessing/connection.py", line 920, in wait
    ready = selector.select(timeout)
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt

thiis error is maybe solved by add blow code

import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')
idekazuki commented 4 years ago 
/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [13,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [17,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [20,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [21,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [22,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [28,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "faster_rcnn.py", line 50, in <module>
    train_one_epoch(model, optimizer, train_data_loader, device, epoch, print_freq=10)
  File "/host/space0/ide-k/mygit/Detectron4epic/engine.py", line 32, in train_one_epoch
    loss_dict = model(images, targets)
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py", line 71, in forward
    detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py", line 765, in forward
    class_logits, box_regression, labels, regression_targets)
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py", line 42, in fastrcnn_loss
    sampled_pos_inds_subset = torch.nonzero(labels > 0).squeeze(1)
RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered
idekazuki commented 4 years ago

this error is solved by chenge class num. 315 to 352.

idekazuki commented 4 years ago 
Traceback (most recent call last):
  File "faster_rcnn.py", line 50, in <module>
    train_one_epoch(model, optimizer, train_data_loader, device, epoch, print_freq=10)
  File "/host/space0/ide-k/mygit/Detectron4epic/engine.py", line 26, in train_one_epoch
    for images, targets in metric_logger.log_every(data_loader, print_freq, header):
  File "/host/space0/ide-k/mygit/Detectron4epic/utils.py", line 209, in log_every
    for obj in iterable:
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/yanai-lab/ide-k/ide-k/pyenv/detec2/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/host/space0/ide-k/mygit/Detectron4epic/epicdatasets.py", line 104, in __getitem__
    image = Image.fromarray(frames[frame_id])
IndexError: list index out of range
idekazuki commented 4 years ago

frames[frame_id]) -> frames[frame_id - 1])

idekazuki commented 4 years ago

Error loss is nan

lr = 0.005 -> lr = 0.0005

idekazuki commented 4 years ago

Error loss is nan . check dataset xmax, ymax > xmiin, ymin xmax, ymax and xmin, ymin in image range.

idekazuki commented 4 years ago

値のおかしいデータ合計4個を発見したので削除。