search

Lurunchik / NF-CATS

MIT License
15 stars 2 forks source link

TypeError: must be real number, not FBetaMeasure #1

Open seanmacavaney opened 2 years ago

seanmacavaney commented 2 years ago

Thanks for the nice work!

I get the following error, using a fresh environment. It appears to be something going wrong in AllenNLP, but I couldn't get to the bottom of it with a bit of searching and debugging.

Traceback (most recent call last):
  File "nfcats/train.py", line 54, in <module>
    main()
  File "nfcats/train.py", line 40, in main
    train_model(params=params, serialization_dir=str(serialization_dir))
  File "/home/sean/miniconda3/envs/nfcats3/lib/python3.7/site-packages/allennlp/commands/train.py", line 242, in train_model
    file_friendly_logging=file_friendly_logging,
  File "/home/sean/miniconda3/envs/nfcats3/lib/python3.7/site-packages/allennlp/commands/train.py", line 466, in _train_worker
    metrics = train_loop.run()
  File "/home/sean/miniconda3/envs/nfcats3/lib/python3.7/site-packages/allennlp/commands/train.py", line 528, in run
    return self.trainer.train()
  File "/home/sean/miniconda3/envs/nfcats3/lib/python3.7/site-packages/allennlp/training/trainer.py", line 930, in train
    metrics, epoch = self._try_train()
  File "/home/sean/miniconda3/envs/nfcats3/lib/python3.7/site-packages/allennlp/training/trainer.py", line 963, in _try_train
    train_metrics = self._train_epoch(epoch)
  File "/home/sean/miniconda3/envs/nfcats3/lib/python3.7/site-packages/allennlp/training/trainer.py", line 752, in _train_epoch
    description = training_util.description_from_metrics(metrics)
  File "/home/sean/miniconda3/envs/nfcats3/lib/python3.7/site-packages/allennlp/training/util.py", line 413, in description_from_metrics
    for name, value in metrics.items()
  File "/home/sean/miniconda3/envs/nfcats3/lib/python3.7/site-packages/allennlp/training/util.py", line 414, in <listcomp>
    if not name.startswith("_")
TypeError: must be real number, not FBetaMeasure

Maybe it's a library version incompatibility and worth freezing some versions?

Also, for my case, I don't really need to train -- I was just interested in model inference. Could you release the trained model?

Thanks!

Lurunchik commented 2 years ago

Thank you for pointing it out. I'll look into it shortly. In the meantime, you can use our hugginface model for inference (Readme updated). https://huggingface.co/spaces/Lurunchik/nf-cats spaces demo, https://huggingface.co/Lurunchik/nf-cats model

seanmacavaney commented 2 years ago

Thanks, the inference code is exactly what I needed!

It enabled us to extend our analysis in our SIGIR'22 paper Survivorship Bias in MS MARCO a bit -- see the results here: https://github.com/seanmacavaney/sigir2022-survivor/#appendix

In short, due to the MS MARCO annotation process, 77% of INSTRUCTION questions were tossed out (so they are highly under-represented in the final dataset). Meanwhile, only 39% of EVIDENCE-BASED questions were tossed out (so they are over-represented).