Open dfl opened 3 weeks ago
so happy to see this, i had the exact same idea and wanted to check beforehand.
i wonder if we're better off just making it an openai spec for the llm. reason being that i'd probably run the SDXL model on a CUDA machine and the LLM on mlx.
@fblissjr hi! yes, that's the path I ended up going down but didn't complete... will push what I have
sweet! i converted the models here already. only did 8 bit because of the small nature of the models:
@fblissjr thanks! are you able to take a crack at finishing up the integration?
Additionally, I see you have a fork of mlx-outlines... seems like outlines could be a useful addition to the project, no? https://github.com/lllyasviel/Omost/discussions/84#discussion-6794403
@fblissjr thanks! are you able to take a crack at finishing up the integration?
Additionally, I see you have a fork of mlx-outlines... seems like outlines could be a useful addition to the project, no? #84 (comment)
I didn't - got bogged down and now going to be out of pocket for a week. Most likely won't get to it unfortunately.
I did start outlines, forked from another fork. It was sorta working, but the project was just moving too fast and the fork too big of a drift that I abandoned it. Mostly was curious how far off it was from 'working' - which I discovered is still too far.
I think we need a proper mlx implementation of a constrained decoder & DSL rather than a retrofit of an older one.
@fblissjr thanks! are you able to take a crack at finishing up the integration?
Additionally, I see you have a fork of mlx-outlines... seems like outlines could be a useful addition to the project, no? #84 (comment)
@fblissjr thanks! are you able to take a crack at finishing up the integration?
Additionally, I see you have a fork of mlx-outlines... seems like outlines could be a useful addition to the project, no? #84 (comment)
@dfl just fyi, mlx adding logprobs, which will make stuff like outlines much easier to integrate: https://github.com/ml-explore/mlx-examples/discussions/826#discussioncomment-9739298
I began work on adding Mac / MPS support.
bitsandbytes doesn't yet support MPS, but until then the alternative is MLX. I think this PR is close to working, however streaming is not supported by MLX so I haven't finished integration. Maybe someone else can help?