A bug of the latest commit ` add v1.5`

[linhaojia13]

Rank: 0 partition count [1, 1] and sizes[(468725760, False), (4096, False)] 
  0%|          | 0/1043906 [00:00<?, ?it/s][rank0]: Traceback (most recent call last):
[rank0]:   File "/data1/lhj/codes/Chat-UniVi-1.5/ChatUniVi/train/train_mem.py", line 13, in <module>
[rank0]:     train()
[rank0]:   File "/data1/lhj/codes/Chat-UniVi-1.5/ChatUniVi/train/train.py", line 1211, in train
[rank0]:     trainer.train()
[rank0]:   File "/root/anaconda3/envs/chatunivi/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
[rank0]:     return inner_training_loop(
[rank0]:   File "/root/anaconda3/envs/chatunivi/lib/python3.10/site-packages/transformers/trainer.py", line 1787, in _inner_training_loop
[rank0]:     for step, inputs in enumerate(epoch_iterator):
[rank0]:   File "/root/anaconda3/envs/chatunivi/lib/python3.10/site-packages/accelerate/data_loader.py", line 384, in __iter__
[rank0]:     current_batch = next(dataloader_iter)
[rank0]:   File "/root/anaconda3/envs/chatunivi/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
[rank0]:     data = self._next_data()
[rank0]:   File "/root/anaconda3/envs/chatunivi/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
[rank0]:     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
[rank0]:   File "/root/anaconda3/envs/chatunivi/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
[rank0]:     data = [self.dataset[idx] for idx in possibly_batched_index]
[rank0]:   File "/root/anaconda3/envs/chatunivi/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
[rank0]:     data = [self.dataset[idx] for idx in possibly_batched_index]
[rank0]:   File "/data1/lhj/codes/Chat-UniVi-1.5/ChatUniVi/train/train.py", line 890, in __getitem__
[rank0]:     sources = preprocess_multimodal(
[rank0]:   File "/data1/lhj/codes/Chat-UniVi-1.5/ChatUniVi/train/train.py", line 333, in preprocess_multimodal
[rank0]:     replace_token, vid_replace_token = DEFAULT_IMAGE_TOKEN, DEFAULT_IMAGE_TOKEN * image_token_num
[rank0]: numpy.core._exceptions._UFuncNoLoopError: ufunc 'multiply' did not contain a loop with signature matching types (dtype('<U7'), dtype('int64')) -> None

I print image_token_num, it is a numpy.array but not an int number, which cause this error:

(Pdb) image_token_num
array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
(Pdb) image_token_num.shape 

[jpthu17]

I just updated dataloader.py. The new dataloader.py can avoid this error.

[linhaojia13]

Thank you very much!