search

fastai / fastai2

Temporary home for fastai v2 while it's being developed
https://dev.fast.ai
Apache License 2.0
645 stars 235 forks source link

"IndexError: list index out of range" error. #24

Closed liwei46 closed 4 years ago

liwei46 commented 4 years ago

Hello,

I encountered the following error when trying to train unet using unet_learner, but the error happened when it trained at epoch 7, validation . The problem is in line "getattr(self.items,'iloc',self.items)[i]" is a length 1 "list", but index "i" is 1, which causes this error. Any work-around?

Traceback (most recent call last):
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 280, in fit
    self._do_epoch_train()
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 255, in _do_epoch_train
    self.all_batches()
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 233, in all_batches
    for o in enumerate(self.dl): self.one_batch(*o)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 247, in one_batch
    finally:                                             self('after_batch')
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 214, in __call__
    def __call__(self, event_name): L(event_name).map(self._call_one)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 362, in map
    return self._new(map(g, self))
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 315, in _new
    def _new(self, items, *args, **kwargs): return type(self)(items, *args, use_list=None, **kwargs)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 41, in __call__
    res = super().__call__(*((x,) + args), **kwargs)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 306, in __init__
    items = list(items) if use_list else _listify(items)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 242, in _listify
    if is_iter(o): return list(o)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 208, in __call__
    return self.fn(*fargs, **kwargs)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 217, in _call_one
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 217, in <listcomp>
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 20, in __call__
    if self.run: getattr(self, event_name, noop)()
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/callback/tracker.py", line 26, in after_batch
    if torch.isinf(self.loss) or torch.isnan(self.loss): raise CancelFitException
fastcore.utils.CancelFitException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 209, in added_cbs
    yield
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 283, in fit
    finally:                       self('after_epoch')
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 214, in __call__
    def __call__(self, event_name): L(event_name).map(self._call_one)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 362, in map
    return self._new(map(g, self))
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 315, in _new
    def _new(self, items, *args, **kwargs): return type(self)(items, *args, use_list=None, **kwargs)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 41, in __call__
    res = super().__call__(*((x,) + args), **kwargs)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 306, in __init__
    items = list(items) if use_list else _listify(items)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 242, in _listify
    if is_iter(o): return list(o)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 208, in __call__
    return self.fn(*fargs, **kwargs)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 217, in _call_one
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 217, in <listcomp>
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/learner.py", line 20, in __call__
    if self.run: getattr(self, event_name, noop)()
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/callback/tracker.py", line 104, in after_epoch
    super().after_epoch()
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastai2/callback/tracker.py", line 47, in after_epoch
    val = self.recorder.values[-1][self.idx]
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 316, in __getitem__
    def __getitem__(self, idx): return self._get(idx) if is_indexer(idx) else L(self._get(idx), use_list=None)
  File "/home/wei_li/anaconda3/envs/mainpy3.7/lib/python3.7/site-packages/fastcore/foundation.py", line 320, in _get
    if is_indexer(i) or isinstance(i,slice): return getattr(self.items,'iloc',self.items)[i]
IndexError: list index out of range
liwei46 commented 4 years ago

It seems caused by "CancelFitException" the first exception, I will try to disable TerminateOnNaNCallback, and train again.