Any way to run without an OpenAI key?

shmsw25 / FActScore

A package to evaluate factuality of long-form generation. Original implementation of our EMNLP 2023 paper "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation"

https://arxiv.org/abs/2305.14251

MIT License

238 stars 32 forks source link

Any way to run without an OpenAI key? #34

Open mohummedalee opened 7 months ago

mohummedalee commented 7 months ago

Hi, I have access to Llama through Huggingface and am trying to run FactScore through the CLI: python -m factscore.factscorer --input_path data/unlabeled/InstructGPT.jsonl --model_name retrieval+llama+npm

Shouldn't this use llama to do atomic fact generation? I am getting an error: AssertionError: Please place your OpenAI APT Key in api.key. Is there a way to use FactScore without an OpenAI key?

martiansideofthemoon commented 6 months ago

Hi @mohummedalee, OpenAI API is still needed for atomic fact generation. We welcome any contributions that replace it with LLAMA-based atomic fact generation.

Rosenberg37 commented 3 months ago

@martiansideofthemoon Hi, I was wondering if you have any experience in replacing text davinci with open-source models for atomic fact generation? Will llama2-65b be a viable option? Thanks a lot🙏

martiansideofthemoon commented 3 months ago

Hi @Rosenberg37, a collaborator of @shmsw25 has seen it work well enough with the LLAMA-2 models. However adding them to the codebase for atomic fact extraction may need some non-trivial refactoring. We welcome any contributions here.