Reproducing the behavior of the AI2 demo

[veronica320]

Hi,

Thanks for putting together the repo! I'm trying to make predictions with the model using the models/comet_atomic2020_bart/generation_example.py script. But the behavior is sometimes a little worse than that of the AI2 demo.

Could you please specify the model architecture/hyperparameter settings that are used by the demo? Thanks in advance!

[keisks]

Hi, you can download the model file from here and find the parameters in config.json.

But the behavior is sometimes a little worse than that of the AI2 demo.

hmm, interesting. The demo is using the same model file.

[veronica320]

Thanks! I was following these instructions exactly: downloading the model using download_model.sh and making predictions with generation_example.py. It should be the same as the process you described, right?

But the behavior is indeed sometimes different from the demo. For example, for the word boy and the relation CapableOf, here's the output of running generation_example.py:

['boy CapableOf [GEN]'] [[' kiss girl', ' dress up', ' dress nicely', ' play with', ' kiss']]

And here's the output from the demo:

How exactly should I reproduce the demo behavior?

[csbhagav]

Sorry for the delay in addressing the issues. We are looking at this and will respond soon.

[keisks]

Hi, @veronica320. I downloaded the model and made predictions from scratch, and got the same output as the demo.

model loading ...
model loaded
['boy CapableOf [GEN]']
[[' dance with girl', ' dress up doll', ' kiss girl', ' dress up', ' dance']]

Could you re-try downloading the model file and run prediction with the generation_example.py? (Make sure that you installed the same packages (see requirements.txt)

If it still behaves differently, could you tell us your running environment (OS, running the model on GPU or CPU, etc). I run it with a GPU by running e.g. ,CUDA_VISIBLE_DEVICES=0 python generation_example.py on Ubuntu 20.04.2 LTS.

Thank you.

[veronica320]

Thanks a lot @keisks! Sorry about the late reply as well. We ended up directly using the demo instead, but appreciate your help!

[keisks]

Great! As you may have already know, you can call the API directly by for example

curl "https://comet-atomic-2020.apps.allenai.org/comet?head=PersonX+answers+PersonY+question&rel=xReact&rel=xWant"

Thank you for using the COMET2020!