search

HKUST-KnowComp / atomic-conceptualization

Code and data for the paper Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization
23 stars 1 forks source link

Pretrained Models for generator no tokenizer files #1

Open ghosthamlet opened 1 year ago

ghosthamlet commented 1 year ago

Thanks for open source code and model for paper 'Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization', the paper is a wonderful read. There is just model and config file in download folder, where to download the tokenizer files?

mutiann commented 1 year ago

Ah, it is simply the standard GPT2 tokenizer on Huggingface transformers.

ghosthamlet commented 1 year ago

But you have added special tokens: https://github.com/HKUST-KnowComp/atomic-conceptualization/blob/main/generator/models/comet_atomic2020_gpt2/comet_gpt2.py#L190 , if you are convenient to upload the tokenizer files, it will be more convenient to use it.