@jrichardson97 has been working on result graph + edge summarization using the OpenAI models. Ollama is a host / manager that can download, spin up, and host a REST endpoint to several LLM models. This could provide a more cost-effective alternative to using GPT3.5/4 for all queries if we self-host it in Sterling. Meta's LLAMA2 has a couple variants and the 70b param version would probably perform comparably to GPT3.5 (see recommended resources on the llama2 page). There are more specialized models like this medical dataset LLAMA2 that we could look into. We could also do a hybrid approach where the abstracts get converted to sentences using LLAMA2 and those sentences are fed into GPT4, or some other combination. Seems the main drawback right now is Ollama only supports sequential generation, so requests may be delayed as other generation finishes.
@jrichardson97 has been working on result graph + edge summarization using the OpenAI models. Ollama is a host / manager that can download, spin up, and host a REST endpoint to several LLM models. This could provide a more cost-effective alternative to using GPT3.5/4 for all queries if we self-host it in Sterling. Meta's LLAMA2 has a couple variants and the 70b param version would probably perform comparably to GPT3.5 (see recommended resources on the llama2 page). There are more specialized models like this medical dataset LLAMA2 that we could look into. We could also do a hybrid approach where the abstracts get converted to sentences using LLAMA2 and those sentences are fed into GPT4, or some other combination. Seems the main drawback right now is Ollama only supports sequential generation, so requests may be delayed as other generation finishes.