Closed Xiyan-Xu closed 1 year ago
It must have been resolved with my recent commit, let me know if not. Thanks for the feedback :)
Thanks a lot!
But when I try to train the model, it still cast an error:
Traceback (most recent call last): File "/home/ajatar/TeachLocal/teach/train.py", line 47, in _train return train(cfg, ckpt_ft) File "/home/ajatar/TeachLocal/teach/train.py", line 129, in train trainer.fit(model, datamodule=data_module, ckpt_path=ckpt_ft) File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 740, in fit self._call_and_handle_interrupt( File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt return trainer_fn(*args, **kwargs) File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl self._run(model, ckpt_path=ckpt_path) File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1199, in _run self._dispatch() File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1279, in _dispatch self.training_type_plugin.start_training(self) File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training self._results = trainer.run_stage() File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1289, in run_stage return self._run_train() File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1311, in _run_train self._run_sanity_check(self.lightning_module) File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1375, in _run_sanity_check self._evaluation_loop.run() File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 145, in run self.advance(*args, **kwargs) File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 110, in advance dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders) File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 145, in run self.advance(*args, **kwargs) File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 122, in advance output = self._evaluation_step(batch, batch_idx, dataloader_idx) File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 217, in _evaluation_step output = self.trainer.accelerator.validation_step(step_kwargs) File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 239, in validation_step return self.training_type_plugin.validation_step(*step_kwargs.values()) File "/home/ajatar/anaconda3/envs/teach/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 219, in validation_step return self.model.validation_step(*args, **kwargs) File "/home/ajatar/TeachLocal/teach/teach/model/base.py", line 56, in validation_step return self.allsplit_step("val", batch, batch_idx) File "/home/ajatar/TeachLocal/teach/teach/model/teach.py", line 381, in allsplit_step length_0, length_1 = batch["length_0"], batch["length_1"] KeyError: 'length_0'
I have printed batch, it shows that batch indeed has no attribute "length_0", is data processed properly?
Yes, sorry I omitted to change the configuration. TEACH runs only with dtype=separate_pairs
. I fixed it. Check again and let me know.
Thanks a lot! BTW, I also found some of data processing codes in scripts need to be modified. Maybe you can try to make it better :)
I try to train the model according to the instruction. But it shows that there is a missing file:
File "/home/ajatar/TeachLocal/teach/teach/callback/__init__.py", line 1, in <module> from .render import RenderCallback File "/home/ajatar/TeachLocal/teach/teach/callback/render.py", line 27, in <module> labels_dict = read_json(f'{get_original_cwd()}/deps/inference/labels.json') File "/home/ajatar/TeachLocal/teach/teach/utils/file_io.py", line 79, in read_json with open(p, 'r') as fp: FileNotFoundError: [Errno 2] No such file or directory: '/home/ajatar/TeachLocal/teach/deps/inference/labels.json'
Where can I find the corresponding label.json? BTW, I have also encountered this error (avoided by commenting out the corresponding code):
ModuleNotFoundError: No module named 'pytorch_lightning.utilities.logger'
Thanks a lot for your time.