HalfSummer11
commented
3 years ago We didn't use any special trick to improve GPU utilization. And in our case training on xia dataset usually take one day to converge. Specifically, we didn't try with num_workers>0 - so the dataset implementation may not be friendly for multi-processing.
You may consider setting for_try=False in config.py if you haven't done so already. for_try=True will output images/animations at a very high frequency.
Hi, it's me again! :） I want to use your code to retrain, but the utilization rate of GPU has not been raised. The speed of training is too slow. The GPU I use is RTX 2080ti and K80, both of which appear this situation.
So I adjusted the num_workers in data_loader.py
def get_dataloader(config, subset_name, shuffle=None, data_path=None, extra_data_dir=None): dataset = MotionNorm(config, subset_name, data_path=data_path, extra_data_dir=extra_data_dir) batch_size = config.batch_size if subset_name == 'train' else 1 return DataLoader(dataset, batch_size=batch_size, shuffle=(subset_name == 'train') if shuffle is None else shuffle, num_workers=0)I modifide num workers=0 to num workers=4 But there was a RuntimeError. May I ask how you use tricks to improve the utilization of GPU, so as to improve the training speed? @HalfSummer11 Thank you so much!!