search

bigscience-workshop / petals

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
https://petals.dev
MIT License
8.89k stars 490 forks source link

compile to webassembly #566

Open Donovoi opened 3 months ago

Donovoi commented 3 months ago

Hi Team,

I love your work,

I am thinking at a very early time in the morning that we could reach a wider audience if we compiled to webassembly as well as your normal current release.

this could possibly allow users who are restricted by orwellion group policies to simply browse to a site privately or in the main swarm and take advantage of the hardware support that webassembly offers.

I am not familiar with all of the petals project and all of the complexities of successfully compiling this project to webassembly, but ill give it a go and hopefully offer a pull request if it works.

Thanks again and let me know if this has already been considered as a nono

JeremyMahieu commented 3 months ago

It seems so far out of scope of the possibilities for webassembly. To run it needs gigabytes of python packages and 4GB or ram. If you make a docker image out of the health app it's 7GB, the chat app 9GB, let's start there.

Priyanshupareek commented 3 months ago

Hey @Donovoi I'm working on trying to use webgpu to host the model blocks (MLC-inspired) Wanna chat more on discord?

Donovoi commented 3 months ago

Sounds good! I'm looking at possibly compiling with nuitka/py2wasm looks like they might have unpublished it so it may be unfinished - https://github.com/pypi-data/pypi-mirror-262/tree/code/packages/py2wasm/py2wasm-2.2rc1.tar.gz/py2wasm-2.2rc1