2bit llama70b with quip-sharp

the-crypt-keeper / can-ai-code

Self-evaluating interview for AI coders

MIT License

525 stars 30 forks source link

Closed the-crypt-keeper closed 3 months ago

the-crypt-keeper commented 10 months ago

the-crypt-keeper commented 9 months ago

Requires CUDA12.
Attempting to use nvcr.io/nvidia/pytorch:23.06-py3 as the base, but something is wrong with transformers-engine inside that image and it crashes on load
pip installing git+https://github.com/NVIDIA/TransformerEngine.git@main fixes the crash
the model loads, but generate call never returns

the-crypt-keeper commented 3 months ago

Closing out all old 2-bit quants.