search

antoyang / TubeDETR

[CVPR 2022 Oral] TubeDETR: Spatio-Temporal Video Grounding with Transformers
Apache License 2.0
171 stars 8 forks source link

Training error in main.py #27

Closed cya0102 closed 1 month ago

cya0102 commented 1 month ago

I unzipped all videos in the same folders, and I have processed annotations. When I trained the model to the Epoch: [0] [1000/80684] stage, the error occurred as follow: " Traceback (most recent call last): File "main.py", line 744, in main(args) File "main.py", line 661, in main train_stats = train_one_epoch( File "/data/chenyuan/videogrounding/TubeDETR-main/engine.py", line 51, in train_one_epoch for i, batch_dict in enumerate( File "/data/chenyuan/videogrounding/TubeDETR-main/util/metrics.py", line 146, in log_every for obj in iterable: File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in next data = self._next_data() File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1199, in _next_data return self._process_data(data) File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1225, in _process_data data.reraise() File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/site-packages/torch/_utils.py", line 429, in reraise raise self.exc_type(msg) AssertionError: Caught AssertionError in DataLoader worker process 0. Original Traceback (most recent call last): File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop data = fetcher.fetch(index) File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 219, in getitem return self.datasets[dataset_idx][sample_idx] File "/data/chenyuan/videogrounding/TubeDETR-main/datasets/vidstg.py", line 118, in getitem assert len(images_list) == len(frame_ids) AssertionError

Killing subprocess 841387 Traceback (most recent call last): File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in main() File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main sigkill_handler(signal.SIGTERM, None) # not coming back File "/home/chenyuan/miniconda3/envs/tubedetr/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd) subprocess.CalledProcessError: Command '['/home/chenyuan/miniconda3/envs/tubedetr/bin/python', '-u', 'main.py', '--ema', '--load=/home/chenyuan/.cache/torch/hub/checkpoints/pretrained_resnet101_checkpoint.pth', '--combine_datasets=vidstg', '--combine_datasets_val=vidstg', '--dataset_config', 'config/vidstg.json', '--output-dir=/data/chenyuan/videogrounding/TubeDETR-main/OUTPUT_DIR/']' returned non-zero exit status 1. "