search

chtmp223 / topicGPT

TopicGPT: A Prompt-Based Framework for Topic Modeling (NAACL'24)
https://chtmp223.github.io/topicGPT
228 stars 37 forks source link

Open Source LLM support #12

Closed julienOlivier3 closed 3 days ago

julienOlivier3 commented 3 days ago

Do you (plan) to support open source LLMs from Huggingface?

chtmp223 commented 3 days ago

Hi, we use vllm, which support inference with open-source LLMs from Huggingface. Simply specify api="vllm" and model=[huggingface repo id] to use this feature.

julienOlivier3 commented 3 days ago

Does your vllm integration use offline or online inference or is both possible?

chtmp223 commented 3 days ago

The integration only uses offline inference for now. Do you need online inference?

julienOlivier3 commented 3 days ago

Great, offline inference is what I am looking for. Thanks for your swift reply!