search

imelnyk / ArxivPapers

Code behind Arxiv Papers
445 stars 54 forks source link

Issues to expect when porting to free language model #3

Open SaschaHornauer opened 5 months ago

SaschaHornauer commented 5 months ago

I would be motivated to create a fork to swap GPT for an open language model, that's completely free. Looking at the main.py that seems easy.

Are you aware of forks who already do that? Can you think from the top of your head of hidden dependencies which would make this difficult?

imelnyk commented 5 months ago

Hi, that's a very good idea! I am not aware of other similar efforts.

I would think that one of the possible ways to do this is to use vLLM (which allows to use openai library and still swapping GPT with open LLM). That way the changes will be minimal.

One issue that might arise is probably "function calling" (this is used to create slides for short video, and also to extract questions for QA video). Open LLMs are not good at that, so here some updates are needed (maybe to get rid of function calling entirely, and replace with something else).

Anyway, if you get a change to do this - it will be really nice!