data preprocessing and training error

[dvskabangira]

Thank you so much for the good work you're providing to the AI industry

However, Am training on customized dataset but Am getting the error below most probably related with my wrong pre-processing of pickle annotations files for training and testing. Am experiencing "index out of range" error Any help will be highly appreciated.

2023-12-27 16:36:19,717 - pyskl - INFO - workflow: [('train', 1)], max: 24 epochs 2023-12-27 16:36:19,717 - pyskl - INFO - Checkpoints will be saved to /home/user/work_dirs/posec3d/slowonly_r50_lab_data/joint by HardDiskBackend. 2023-12-27 16:36:24,903 - pyskl - INFO - Saving checkpoint at 1 epochs completed: 0, elapsed: 0s

Traceback (most recent call last): File "/home/user/pyskl/tools/train.py", line 164, in main() File "/home/user/pyskl/tools/train.py", line 156, in main train_model(model, datasets, cfg, validate=args.validate, test=test_option, timestamp=timestamp, meta=meta) File "/home/user/pyskl/pyskl/apis/train.py", line 144, in train_model runner.run(data_loaders, cfg.workflow, cfg.total_epochs) File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run epoch_runner(data_loaders[i], kwargs) File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 54, in train self.call_hook('after_train_epoch') File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/mmcv/runner/base_runner.py", line 309, in call_hook getattr(hook, fn_name)(self) File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/mmcv/runner/hooks/evaluation.py", line 267, in after_train_epoch self._do_evaluate(runner) File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/mmcv/runner/hooks/evaluation.py", line 507, in _do_evaluate key_score = self.evaluate(runner, results) File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/mmcv/runner/hooks/evaluation.py", line 364, in evaluate results, logger=runner.logger, self.eval_kwargs) File "/home/user/pyskl/pyskl/datasets/base.py", line 138, in evaluate if isinstance(results[0], list) or isinstance(results[0], tuple): IndexError: list index out of range WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 48162 closing signal SIGTERM WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 48163 closing signal SIGTERM ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 48161) of binary: /home/user/Downloads/nwiAnaconda/envs/pyskl/bin/python Traceback (most recent call last): File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in main() File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/torch/distributed/run.py", line 718, in run )(*cmd_args) File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/home/user/Downloads/nwiAnaconda/envs/pyskl/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent failures=result.failures, torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

/home/user/pyskl/tools/train.py FAILED

Failures:

------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2023-12-27_16:36:32 host : user-System-Product-Name rank : 0 (local_rank: 0) exitcode : 1 (pid: 48161) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

[dvskabangira]

solved!!!!

[followingcode]

solved!!!!

I'm also experiencing the same error as you, I would like to ask you how you solved it?

[followingcode]

The generated ann_file file is found to be processed with missing values.

[followingcode]

When generating the list file, you need to consider that the training set and the test set are included, and then generate the final ann_file file