castorini / rank_llm

Repository for prompt-decoding using LLMs (GPT3.5, GPT4, Vicuna, and Zephyr)

http://rankllm.ai

Apache License 2.0

273 stars 35 forks source link

Adds script for AWQ-quantizing model #101

Open ru5h16h opened 4 months ago

ru5h16h commented 4 months ago

Pull Request Checklist

Reference Issue

ref: https://github.com/castorini/ura-projects/issues/4

Checklist Items

Before submitting your pull request, please review these items:

[Yes] Have you followed the contributing guidelines?
[Yes] Have you verified that there are no existing Pull Requests for the same update/change?
[Yes] Have you updated any relevant documentation or added new tests where needed?

PR Type

What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other...
- Description: Adds model quantization using AWQ

ru5h16h commented 3 months ago

Here are the details outlining the insights gathered and other experimental information: https://docs.google.com/document/d/1BHpN9lDVGjtjIAFMxjUxNuu1K4KJOgIaOZXkWF1_K8c/edit