search

bklieger-groq / g1

g1: Using Llama-3.1 70b on Groq to create o1-like reasoning chains
MIT License
3.96k stars 357 forks source link

Fork for ollama #4

Closed tcsenpai closed 2 months ago

tcsenpai commented 2 months ago

Hello!

Not really an issue here, but just a note to inform you about a fork I made (https://github.com/tcsenpai/ol1) that uses local Ollama.

The fork is still in testing phase, but I thought it would just be fair to inform you as the original creator of the project.

If you think this can be merged in one project (maybe with configurable options in the frontend?) let me know, else we can continue as two separate projects : )

Compliments again for the wonderful idea!

av commented 2 months ago

Thanks for preparing the ollama version, you can significantly reduce the amount of misses by using "format": "json" for the chat completion API. Even 8b can handle the reasoning stream with that.

tcsenpai commented 2 months ago

Thanks for the comment. I also added a perplexity version of the app, in separate scripts so if @bklieger-groq wants to merge the projects he can do it without replacing the whole app.py.

I just tested the perplexity fork and it works quite good (except formatting errors) if you want to give it a try.

av commented 2 months ago

I've integrated your implementation with a few tweaks (accepting more relaxed generation, more parameters) in Harbor v0.1.24, thanks!

bklieger-groq commented 2 months ago

Happy to accept PRs here with alternative model hosting options! To keep the repo simple, I may move them into a dedicated folder, so the root has g1 as default and if you want to explore other options you can try them. Alternatively, happy to link to these forks in the README, but merging with this project would be ideal so that any improvements automatically sync across the options.

tcsenpai commented 2 months ago

Happy to accept PRs here with alternative model hosting options! To keep the repo simple, I may move them into a dedicated folder, so the root has g1 as default and if you want to explore other options you can try them. Alternatively, happy to link to these forks in the README, but merging with this project would be ideal so that any improvements automatically sync across the options.

For me too the best way is to merge!

I am preparing a PR, what do you think about having a launcher script that defaults to Grok but is able to launch Ollama and Perplexity too?

Ideally I would create three different modules and converge them in a main method, but I am really a newbie in starlit/gradio, so I am not able to do that.

Let me know, so I can prepare the pr accordingly : )

tcsenpai commented 2 months ago

Update: I tried to draft a launcher.py file that should be simple enough to allow multi modes. I changed my repo to multi1 to keep it simple. I am opening a PR so we can continue there (pr here)

jjafuller commented 2 months ago

We may want to consider using LiteLLM so that we can support just about everything without custom code for vendors.

bklieger-groq commented 2 months ago

Update: I tried to draft a launcher.py file that should be simple enough to allow multi modes. I changed my repo to multi1 to keep it simple. I am opening a PR so we can continue there (https://github.com/bklieger-groq/g1/pull/6)

I would prefer to keep this repository named "g1" and keep the quickstart process simple with one default app. I like the launcher and alternative options, so I am happy to merge into a folder called "mult1" that has these options. If you make that change, I will review and merge the PR.

tcsenpai commented 2 months ago

Update: I tried to draft a launcher.py file that should be simple enough to allow multi modes. I changed my repo to multi1 to keep it simple. I am opening a PR so we can continue there (#6)

I would prefer to keep this repository named "g1" and keep the quickstart process simple with one default app. I like the launcher and alternative options, so I am happy to merge into a folder called "mult1" that has these options. If you make that change, I will review and merge the PR.

Ok I am doing it. If that's ok with you, I will keep my repo as a standalone project with credits to your one, anyway

bklieger-groq commented 2 months ago

Sure! Actually, I am happy to link to your project in the README if thats preferable. I think its more intuitive to have mult1 in one place instead of two.

tcsenpai commented 2 months ago

Yep I agree! Feel free to link it, so that when I will update it, it will be in one place. To @jjafuller I will work to implement that in multi1, then

bklieger-groq commented 2 months ago

Great! I will close out the PR then.

bklieger-groq commented 2 months ago

Added to the README here: https://github.com/bklieger-groq/g1/commit/62735af1400fa1f8d821c5093c88326b668ec36a

Looking forward to seeing how Mult1 advances!

matbee-eth commented 2 months ago

multi1 doesnt allow for Issues though...

tcsenpai commented 2 months ago

multi1 doesnt allow for Issues though...

Oh my bad! Opening them