search

EricLBuehler / mistral.rs

Blazingly fast LLM inference.
MIT License
4.4k stars 305 forks source link

Prebuilt binary for python bindings #744

Open mert-kurttutan opened 2 months ago

mert-kurttutan commented 2 months ago

What is the current status for providing prebuilt for providing python bindings?

If prebuilt binary is provided, this would be really beneficial in terms of download/compile time of python bindings. I also came across errors when downloading on my Windows machine due linking issues for intel mkl (its multi threading library, iomp). (I cannot provide further info on this issue at the moment). Providing prebuilt binary would also make these issues easier to solve imo.

Also, what do you think about using intel mkl pypi package instead of intel-mkl-src. When using mkl, in my experience, pypi package has been more smooth (especially on windows machines).

I am happy to put work into PR.

mert-kurttutan commented 1 month ago

Hi @EricLBuehler , I understand you must be busy. I would very much appreciate an answer as to whether pre built binary is something you are ok with since I can work on this if you are OK with this. The compilation step is very long, gives error on windows (especially when I share the code with my coworkers) and different from usual from python experience.

EricLBuehler commented 1 month ago

Hi @mert-kurttutan! Sorry for not getting back.

I would be happy to have prebuilt python binaries for the various packages. I think it would be very helpful for usability, but my only concern is that I do not have access to Mac or Accelerate hardware locally, so I would not be able to generate binaries for those targets. Perhaps we can use a cloud instance?