Key error: "probabilities"

[JFChi]

Hi there,

I encounter this error when I try to run the code. Could you please take a look at it?

File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1279, in _dispatch
    self.training_type_plugin.start_training(self)
  File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
    self._results = trainer.run_stage()
  File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1289, in run_stage
    return self._run_train()
  File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1319, in _run_train
    self.fit_loop.run()
  File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 145, in run
    self.advance(*args, **kwargs)
  File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 234, in advance
    self.epoch_loop.run(data_fetcher)
  File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 146, in run
    self.on_advance_end()
  File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 242, in on_advance_end
    self._run_validation()
  File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 337, in _run_validation
    self.val_loop.run()
  File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 151, in run
    output = self.on_run_end()
  File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 131, in on_run_end
    self._evaluation_epoch_end(outputs)
  File "/data/home/jianfengchi/miniconda/envs/llmtab/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 236, in _evaluation_epoch_end
    model.validation_epoch_end(outputs)
  File "/data/home/jianfengchi/code/TabLLM/t-few/src/models/EncoderDecoder.py", line 256, in validation_epoch_end
    metrics = self.dataset_reader.compute_metric(accumulated)
  File "/data/home/jianfengchi/code/TabLLM/t-few/src/data/dataset_readers.py", line 312, in compute_metric
    pos_probs = [p[1] for p in accumulated['probabilities']]

[stefanhgm]

Thanks for your interest in using TabLLM! Unfortunately, I currently don’t have access to my computer. I will come back to you on the 11th of April. Sorry for the delay.

Best, Stefan

[stefanhgm]

Hi @JFChi ,

seems like something is going wrong during the metric calculation. It seems to me that the error message is missing in your output. It only refers to the file that contains the error. Could you maybe provide the complete output?

[JFChi]

Hi @JFChi ,

seems like something is going wrong during the metric calculation. It seems to me that the error message is missing in your output. It only refers to the file that contains the error. Could you maybe provide the complete output?

Hi @stefanhgm,

Thanks for your reply. The error is in the output of predict function of EncoderDecoder.py. In order to calculate AUC score in dataset_readers.py, we should have the probabilities of the answer choices. Thus, I think you code add the probability calculation in the EncoderDecoder.py. Could you please upload the updated EncoderDecoder.py? Thanks.

[stefanhgm]

Hi @JFChi,

I added our EncoderDecoder.py implementation. We retrieve the probabilities to determine the AUC. We also added some functionality to keep track of the best model during validation, however, we did not use this for the final experiments where we choose the same number of epochs for all experiments.

Please let me know if there is anything else I could help you with!

Stefan