LoRA training error - list index out of range

[JeffreyLuW]

I've been stuck on list index out of range error for lora fine-tuning. I would appreciate if anyone can give insights on how to solve this problem, Thank you. @GaussianObject

/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:424: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of thenum_workersargument to num_workers=191 in the DataLoader to improve performance. /root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py:298: The number of training batches (4) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch. Epoch 0: 0%| | 0/4 [00:00<?, ?it/s]Traceback (most recent call last): File "/root/autodl-tmp/GaussianObject/train_lora.py", line 117, in trainer.fit(model, dataloader) File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 538, in fit call._call_and_handle_interrupt( File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 47, in _call_and_handle_interrupt return trainer_fn(*args, *kwargs) File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 574, in _fit_impl self._run(model, ckpt_path=ckpt_path) File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 981, in _run results = self._run_stage() File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1025, in _run_stage self.fit_loop.run() File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 205, in run self.advance() File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 363, in advance self.epoch_loop.run(self._data_fetcher) File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 140, in run self.advance(data_fetcher) File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/loops/training_epochloop.py", line 212, in advance batch, , = next(data_fetcher) File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/loops/fetchers.py", line 133, in next batch = super().next() File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/loops/fetchers.py", line 60, in next batch = next(self.iterator) File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/utilities/combined_loader.py", line 341, in next out = next(self._iterator) File "/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/utilities/combined_loader.py", line 78, in next out[i] = next(self.iterators[i]) File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in next__ data = self._next_data() File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, **kwargs) File "/root/autodl-tmp/GaussianObject/dataset_lora.py", line 350, in getitem self.gaussian.add_statistics_noise(self.statistics_info, noise_dropout, noise_scale) File "/root/autodl-tmp/GaussianObject/scene/gaussian_model.py", line 494, in add_statistics_noise for key in statistics_info[0].keys(): IndexError: list index out of range`

[JeffreyLuW]

solved