search

NVlabs / VILA

VILA - a multi-image visual language model with training, inference and evaluation recipe, deployable from cloud to edge (Jetson Orin and laptops)
Apache License 2.0
1.98k stars 158 forks source link

AttributeError: 'Image' object has no attribute 'shape' #104

Open ZackBradshaw opened 3 months ago

ZackBradshaw commented 3 months ago

AttributeError: 'Image' object has no attribute 'shape'

ZackBradshaw commented 3 months ago

I keep getting this bug. Is the training expecting already converted tensor data?

ZackBradshaw commented 3 months ago

this will train for a moment up to 13% then throws these errors

Lyken17 commented 3 months ago

Can u attach a full log here?

ZackBradshaw commented 3 months ago 
Traceback (most recent call last):
  File "/home/user/.lib/.lib/VILA/llava/train/train_mem.py", line 36, in <module>
    train()
  File "/home/user/.lib/.lib/VILA/llava/train/train.py", line 436, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1537, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1821, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/usr/local/lib/python3.10/dist-packages/accelerate/data_loader.py", line 452, in __iter__
    current_batch = next(dataloader_iter)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.10/dist-packages/torch/_utils.py", line 644, in reraise      
    raise exception
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataset.py", line 243, in __getitem__
    return self.datasets[dataset_idx][sample_idx]
  File "/home/swarms/.lib/.lib/VILA/llava/data/dataset.py", line 837, in __getitem__        
    if len(image.shape) == 4:
AttributeError: 'Image' object has no attribute 'shape'

I'm able to push it along further by adding logic to just convert this to tensor but ultimately this just leads to more issues further on in the training.

Lyken17 commented 3 months ago

my suggestion would be

  1. disable the training logic and simpley iterate through the dataset
  2. check every data and make sure the return value if tensor

This looks like some of your images are grayscale or corrupted.

ZackBradshaw commented 3 months ago

Will this work work with other tools such as xtuner I cannot for the life of me figure this out