Closed YounghwaJung closed 7 months ago
Hard to tell from just that error what is going wrong. Some shape mismatch when the pytorch dataloader is batching individual samples together. What was the command you used to start training? Were any changes made to the code?
I used the team_code/shell_train.sh and made following modifications because I encountered the same error https://github.com/autonomousvision/carla_garage/issues/12.
torchrun --nnodes=1 --nproc_per_node 8 --max_restarts=1 --rdzv_id=42353467 --rdzv_backend=c10d train.py --id train_id_000 --batch_size 8 --setting 02_05_withheld --root_dir /dataset/carla_garage --logdir logs --use_controller_input_prediction 1 --use_wp_gru 0 --use_discrete_command 1 --use_tp 1 --continue_epoch 1 --cpu_cores 0 --num_repetitions 3
The training runs smoothly for thousands of steps, but errors occur randomly. Is it possible that the datasets got corrupted during the download process?
I also encountered the same error #12, I have modified shell_train.sh
as suggested and now I'm obtaining as well:
stack expects each tensor to be equal size, but got [256, 256] at entry 0 and [1, 11] at entry 6
randomly during training.
I paste my shell_train.sh
in the following:
torchrun --nnodes=1 --nproc_per_node=1 --max_restarts=1 --rdzv_id=42353467 --rdzv_backend=c10d train.py --id train_id_000 --batch_size 12 --setting 02_05_withheld --root_dir /home/enrico/Projects/Carla/carla_garage/data --logdir /home/enrico/Projects/Carla/carla_garage/logs --use_controller_input_prediction 1 --use_wp_gru 0 --use_discrete_command 1 --use_tp 1 --continue_epoch 1 --cpu_cores 0 --num_repetitions 3
In principle its prossible that something got correpted during download or unzipping. some of the data loading libraries fail silently and return none when they can't load data.
You can check that by adding asserts for None or similar here.
Since you two also seem to have problems with the pytorch multiprocessing, which multiprocessing method is selected by your system (gets printed in the beginning with "Start method of multiprocessing:")?
If it is possible for you to debug and see which data it is that crashes the batching would also be helpful.
@Noce99 Unrelated to your problem but you are using a very small batch size (total of 12). Total batch size is: num gpu x batch size, if you want to train with 1 GPU you probably need to increase batch size or if not possible reduce the learning rate proportionally (to compensate for noisier gradient and more total gradient steps).
Start method of multiprocessing: fork
--lr 7.5e-5
for sure thanks!fork is the python default on linux. It works for me but you can also try the newer "spawn" or "forkserver" methods, since you seem to have problems with multithreading (setting --cpu_cores 0 might slow down training since data loading isn't parallelized). You can set the methods here.
I have solved the problem by checking the shape of the data before using them. It turns out that just one element was corrupted.
In the following how I modified data.py
.
...
class CARLA_Data(Dataset):
...
def __init__(self,
...
self.last_data = None
...
def __getitem__(self, index):
...
wanted_sizes = {'semantic': (256, 1024), 'bev_semantic': (256, 256), 'depth': (256, 1024), 'rgb': (3, 256, 1024), 'center_heatmap': (4, 64, 64), 'wh': (2, 64, 64), 'yaw_class': (64, 64), 'yaw_res': (1, 64, 64), 'offset': (2, 64, 64), 'velocity': (1, 64, 64), 'brake_target': (64, 64), 'pixel_weight': (2, 64, 64), 'lidar': (1, 256, 256), 'bounding_boxes': (30, 8), 'command': (6,), 'next_command': (6,), 'route': (20, 2), 'target_point': (2,), 'aim_wp': (2,)}
for wanted_key in wanted_sizes:
if tuple(data[wanted_key].shape) != wanted_sizes[wanted_key]:
print(f"Bad {wanted_key}")
return self.last_data
self.last_data = data
return data
...
It turns out that the not matching shape was a single bev_semantic
.
Great. Can you share which file was corrupt in case its the same for multiple people?
s1_dataset_2023_05_10/Routes_Town04_Scenario1_Repetition2/Town04_Scenario1_route49_05_11_16_31_34/bev_semantics_augmented/0039.png
Hi, I am trying to reproduce the results by training the model. However, while training, an error occurred, and I am uncertain about the cause. Could you please suggest any possible solutions?