search

pixelite1201 / BEDLAM

206 stars 19 forks source link

UnboundLocalError: local variable 'img' referenced before assignment #29

Closed GraceCheong closed 1 month ago

GraceCheong commented 9 months ago

I was training a model on the server. But i've got an error like this as shown below, but my image is perfectly normal. I have no idea how to deal with it. Do you have any similar experience of dealing with this kinds of error?

log : 

| INFO     | train.dataset.dataset:__getitem__:217 - @data/training_images/20221022_3_250_batch01handhair_static_bigOffice_30fps/png/seq_000118/seq_000118_0035.png from static-office-hair
could not broadcast input array from shape (0,243,3) into shape (233,243,3)
Traceback (most recent call last):
  File "train.py", line 130, in <module>
    train(hparams, fast_dev_run=args.fdr)
  File "train.py", line 73, in train
    trainer.fit(model, ckpt_path=hparams.TRAINING.RESUME)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 740, in fit
    self._call_and_handle_interrupt(
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1199, in _run
    self._dispatch()
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1279, in _dispatch
    self.training_type_plugin.start_training(self)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
    self._results = trainer.run_stage()
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1289, in run_stage
    return self._run_train()
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1319, in _run_train
    self.fit_loop.run()
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 145, in run
    self.advance(*args, **kwargs)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 234, in advance
    self.epoch_loop.run(data_fetcher)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 145, in run
    self.advance(*args, **kwargs)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 156, in advance
    batch_idx, (batch, self.batch_progress.is_last_batch) = next(self._dataloader_iter)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 203, in __next__
    return self.fetching_function()
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 270, in fetching_function
    self._fetch_next_batch()
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 300, in _fetch_next_batch
    batch = next(self.dataloader_iter)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 550, in __next__
    return self.request_next_batch(self.loader_iters)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 562, in request_next_batch
    return apply_to_collection(loader_iters, Iterator, next)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 95, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1325, in _next_data
    return self._process_data(data)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/torch/_utils.py", line 644, in reraise
    raise exception
UnboundLocalError: Caught UnboundLocalError in DataLoader worker process 5.
Original Traceback (most recent call last):
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/media/BEDLAM/bedlam/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 243, in __getitem__
    return self.datasets[dataset_idx][sample_idx]
  File "/home/media/BEDLAM/train/dataset/dataset.py", line 220, in __getitem__
    img = torch.from_numpy(img).float()
UnboundLocalError: local variable 'img' referenced before assignment

files under the directory : image

exact image copied from server : seq_000118_0035

pixelite1201 commented 4 months ago

No, I have not faced this issue. Is the error deterministic i.e. does it keep occurring for the same image?