search

TinyLLaVA / TinyLLaVA_Factory

A Framework of Small-scale Large Multimodal Models
https://arxiv.org/abs/2402.14289
Apache License 2.0
570 stars 53 forks source link

Finetuning get: ValueError: not enough values to unpack (expected 2, got 0) in get_modality_length_grouped_indices #109

Open sc268 opened 1 month ago

sc268 commented 1 month ago

Hi I am encountering an issue during finetuning with the default script ValueError: not enough values to unpack (expected 2, got 0) in get_modality_length_grouped_indices

Parameter Offload: Total persistent parameters: 1324480 in 540 params
  0%|                                                                                                                          | 0/155 [00:00<?, ?it/s]Traceback (most recent call last):
  File "tinyllava/train/train.py", line 92, in <module>
    train()
  File "tinyllava/train/train.py", line 87, in train
    trainer.train()
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1624, in train
    return inner_training_loop(
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1928, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/opt/conda/lib/python3.8/site-packages/accelerate/data_loader.py", line 451, in __iter__
    dataloader_iter = super().__iter__()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 438, in __iter__
    return self._get_iterator()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 386, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1084, in __init__
    self._reset(loader, first_iter=True)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1117, in _reset
    self._try_put_index()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1351, in _try_put_index
    index = self._next_index()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 620, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 282, in __iter__
    for idx in self.sampler:
  File "/data/TinyLLaVA_Factory/tinyllava/train/tinyllava_trainer.py", line 114, in __iter__
    indices = get_modality_length_grouped_indices(self.lengths, self.batch_size, self.world_size, generator=self.generator)
  File "/data/TinyLLaVA_Factory/tinyllava/train/tinyllava_trainer.py", line 47, in get_modality_length_grouped_indices
    lang_indices, lang_lengths = zip(*[(i, -l) for i, l in enumerate(lengths) if l < 0])
ValueError: not enough values to unpack (expected 2, got 0)
  0%|                                                                                                                          | 0/155 [00:01<?, ?it/s]
[2024-08-22 21:44:17,772] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 9692

which occurs here in the code: lang_indices, lang_lengths = zip(*[(i, -l) for i, l in enumerate(lengths) if l < 0]) when all lengths > 0, the right side is [] hence unable to unpack.

Any suggestions? Thanks!

ggcr commented 1 month ago

Maybe this is related #89

Personally, I fixed it changing some code as seen in the PR 371ef287b32f78bb5075a437acc6ece731169ae9

sc268 commented 1 month ago

Thanks @ggcr ! The PR makes sense and resolves the issue.