I run in 4*3090(24G), but the data in 200-300 seem error
AssertionError: Caught AssertionError in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
data = fetcher.fetch(index)
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 219, in getitem
return self.datasets[dataset_idx][sample_idx]
File "/home/Newdisk/zhangzp/TubeDETR/TubeDETR/datasets/vidstg.py", line 116, in getitem
assert len(images_list) == len(frame_ids)
AssertionError
Killing subprocess 2844448
Killing subprocess 2844449
Killing subprocess 2844450
Killing subprocess 2844451
Traceback (most recent call last):
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in
main()
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/zhangzp/anaconda3/envs/tubedetr_env/bin/python', '-u', 'main.py', '--ema', '--load=pretrained_resnet101_checkpoint.pth', '--combine_datasets=vidstg', '--combine_datasets_val=vidstg', '--dataset_config', 'config/vidstg.json', '--output-dir=Vidstg_train']' returned non-zero exit status 1.
antoyang
commented
2 years ago
I run in 4*3090(24G), but the data in 200-300 seem error
AssertionError: Caught AssertionError in DataLoader worker process 1. Original Traceback (most recent call last): File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop data = fetcher.fetch(index) File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 219, in getitem
return self.datasets[dataset_idx][sample_idx]
File "/home/Newdisk/zhangzp/TubeDETR/TubeDETR/datasets/vidstg.py", line 116, in getitem
assert len(images_list) == len(frame_ids)
AssertionError
Killing subprocess 2844448 Killing subprocess 2844449 Killing subprocess 2844450 Killing subprocess 2844451 Traceback (most recent call last): File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in
main()
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/home/zhangzp/anaconda3/envs/tubedetr_env/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/zhangzp/anaconda3/envs/tubedetr_env/bin/python', '-u', 'main.py', '--ema', '--load=pretrained_resnet101_checkpoint.pth', '--combine_datasets=vidstg', '--combine_datasets_val=vidstg', '--dataset_config', 'config/vidstg.json', '--output-dir=Vidstg_train']' returned non-zero exit status 1.