hparams missing in the longformer_large_science checkpoint?

[gnkitaa]

Hi David! Thanks a lot for sharing your great work!

I am currently struggling to use the longformer_large_science checkpoint for making predictions on the scifact dataset. It seems the model checkpoint doesn't have 'hparams' needed to instantiate the model. I am getting following error:

model = cls._load_model_state(checkpoint, strict=strict, **kwargs)

 File "/pytorch_lightning/core/saving.py", line 198, in _load_model_state
    model = cls(**_cls_kwargs)

TypeError: __init__() missing 1 required positional argument: 'hparams'

I wonder if there is an issue with saving the checkpoint or if additional modifications to the MultiVerSModel class can help run the checkpoint.

Thank you

[dwadden]

Hi, thanks for your interest!

I'm traveling for the next two weeks, but will take a look when I get back at the end of March. In the meantime, my only advice is to make sure that your setup matches the specification in the README.

Dave

[dwadden]

I'm taking a look at this now. I think it probably doesn't make sense to use longformer_large_science for prediction, since it wasn't trained on any fact-checking datasets. I think that's what's going on with the hparams. If you want to make predictions on scifact, you're probably better off using the dedicated scifact model, as done here.

Let me know if you have more questions.

[gnkitaa]

Thank you so much for the response. :) I wanted to try out a model that hasn't been specifically trained on fact-checking datasets but has science knowledge. Is there any way to use the longformer_large_science model to make predictions?

[dwadden]

The way that I made supports / refutes / nei decisions for MultiVerS was to train a three-way classification head on top of the <s> token from LongFormer. The base longformer_large_science model doesn't have this classification head, so you can't make fact-checking predictions in the same fashion. I think if you're interested in looking at zero-shot fact-checking abilities of scientific LLM's, your best bet is to try a prompting-based approach with an autoregressive language model like PubMedGPT.

[gnkitaa]

That makes sense, thanks alot for the pointer! :)

[dwadden]

No problem! I'm going to close this issue; feel free to reopen if you've got more questions.