Open pprp opened 4 months ago
Abhis-123
commented
4 months ago HI @pprp Thanks for putting effort, can you share more about the requirements to build this pr, I want to use this branch for quantization..
Because in your pr it's still transformers==4.36.2
pprp
commented
4 months ago @Abhis-123 Here is the environment in jetson Orin:
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 51_gnu
absl-py 2.1.0 pypi_0 pypi
accelerate 0.28.0 pypi_0 pypi
aiofiles 23.2.1 pypi_0 pypi
aiohttp 3.9.3 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
altair 5.2.0 pypi_0 pypi
anyio 4.3.0 pypi_0 pypi
async-timeout 4.0.3 pypi_0 pypi
attributedict 0.3.0 pypi_0 pypi
attrs 23.2.0 pypi_0 pypi
awq 0.1.0 pypi_0 pypi
awq-inference-engine 0.0.0 pypi_0 pypi
blessings 1.7 pypi_0 pypi
bzip2 1.0.8 h998d150_5
ca-certificates 2024.3.11 hd43f75c_0
cachetools 5.3.3 pypi_0 pypi
certifi 2024.2.2 pypi_0 pypi
chardet 5.2.0 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
click 8.1.7 pypi_0 pypi
codecov 2.1.13 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
coloredlogs 15.0.1 pypi_0 pypi
colour-runner 0.1.1 pypi_0 pypi
contourpy 1.1.1 pypi_0 pypi
coverage 7.4.4 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
dataproperty 1.0.1 pypi_0 pypi
datasets 2.18.0 pypi_0 pypi
deepdiff 6.7.1 pypi_0 pypi
dill 0.3.8 pypi_0 pypi
distlib 0.3.8 pypi_0 pypi
distro 1.9.0 pypi_0 pypi
dockerhub-api 0.5 pypi_0 pypi
exceptiongroup 1.2.0 pypi_0 pypi
fastapi 0.110.0 pypi_0 pypi
ffmpy 0.3.2 pypi_0 pypi
filelock 3.13.1 pypi_0 pypi
fonttools 4.50.0 pypi_0 pypi
frozenlist 1.4.1 pypi_0 pypi
fsspec 2024.2.0 pypi_0 pypi
furl 2.1.3 pypi_0 pypi
gguf 0.6.0 pypi_0 pypi
gradio 3.35.2 pypi_0 pypi
gradio-client 0.2.9 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
hf-transfer 0.1.6 pypi_0 pypi
httpcore 1.0.4 pypi_0 pypi
httpx 0.27.0 pypi_0 pypi
huggingface-hub 0.23.0 pypi_0 pypi
humanfriendly 10.0 pypi_0 pypi
idna 3.6 pypi_0 pypi
importlib-resources 5.13.0 pypi_0 pypi
inspecta 0.1.3 pypi_0 pypi
jinja2 3.1.3 pypi_0 pypi
joblib 1.3.2 pypi_0 pypi
jsonlines 4.0.0 pypi_0 pypi
jsonschema 4.21.1 pypi_0 pypi
jsonschema-specifications 2023.12.1 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
ld_impl_linux-aarch64 2.38 h8131f2d_1
libffi 3.4.4 h419075a_0
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.41.5 h998d150_0
linkify-it-py 2.0.3 pypi_0 pypi
lm-eval 0.3.0 pypi_0 pypi
markdown-it-py 2.2.0 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
matplotlib 3.7.5 pypi_0 pypi
mbstrdecoder 1.1.3 pypi_0 pypi
mdit-py-plugins 0.3.3 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
multidict 6.0.5 pypi_0 pypi
multiprocess 0.70.16 pypi_0 pypi
ncurses 6.4 h419075a_0
networkx 3.1 pypi_0 pypi
nltk 3.8.1 pypi_0 pypi
numexpr 2.8.6 pypi_0 pypi
numpy 1.24.4 pypi_0 pypi
openai 1.14.2 pypi_0 pypi
openssl 3.0.13 h2f4d8fa_0
ordered-set 4.1.0 pypi_0 pypi
orderedmultidict 1.0.1 pypi_0 pypi
orjson 3.9.15 pypi_0 pypi
packaging 24.0 pypi_0 pypi
pandas 2.0.3 pypi_0 pypi
pathvalidate 3.2.0 pypi_0 pypi
pillow 10.2.0 pypi_0 pypi
pip 24.0 pypi_0 pypi
pkgutil-resolve-name 1.3.10 pypi_0 pypi
platformdirs 4.2.0 pypi_0 pypi
pluggy 1.4.0 pypi_0 pypi
portalocker 2.8.2 pypi_0 pypi
protobuf 5.26.0 pypi_0 pypi
psutil 5.9.8 pypi_0 pypi
pyarrow 15.0.2 pypi_0 pypi
pyarrow-hotfix 0.6 pypi_0 pypi
pybind11 2.11.1 pypi_0 pypi
pycountry 23.12.11 pypi_0 pypi
pydantic 1.10.14 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pygments 2.17.2 pypi_0 pypi
pyparsing 3.1.2 pypi_0 pypi
pyproject-api 1.6.1 pypi_0 pypi
pytablewriter 1.2.0 pypi_0 pypi
python 3.8.19 h4bb2201_0
python-dateutil 2.9.0.post0 pypi_0 pypi
python-multipart 0.0.9 pypi_0 pypi
pytz 2024.1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 8.2 h998d150_0
referencing 0.34.0 pypi_0 pypi
regex 2023.12.25 pypi_0 pypi
requests 2.31.0 pypi_0 pypi
rootpath 0.1.1 pypi_0 pypi
rouge-score 0.1.2 pypi_0 pypi
rpds-py 0.18.0 pypi_0 pypi
sacrebleu 1.5.0 pypi_0 pypi
safetensors 0.4.2 pypi_0 pypi
scikit-learn 1.3.2 pypi_0 pypi
scipy 1.10.1 pypi_0 pypi
semantic-version 2.10.0 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
setuptools 68.2.2 py38hd43f75c_0
six 1.16.0 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
sqlite 3.41.2 h998d150_0
sqlitedict 2.1.0 pypi_0 pypi
starlette 0.36.3 pypi_0 pypi
sympy 1.12 pypi_0 pypi
tabledata 1.3.3 pypi_0 pypi
tcolorpy 0.1.4 pypi_0 pypi
termcolor 2.4.0 pypi_0 pypi
texttable 1.7.0 pypi_0 pypi
threadpoolctl 3.4.0 pypi_0 pypi
tk 8.6.12 h241ca14_0
tokenizers 0.19.1 pypi_0 pypi
toml 0.10.2 pypi_0 pypi
tomli 2.0.1 pypi_0 pypi
toolz 0.12.1 pypi_0 pypi
torch 2.0.0+nv23.5 pypi_0 pypi
tox 4.14.1 pypi_0 pypi
tqdm 4.66.2 pypi_0 pypi
tqdm-multiprocess 0.0.11 pypi_0 pypi
transformers 4.41.0.dev0 pypi_0 pypi
typepy 1.3.2 pypi_0 pypi
typing-extensions 4.10.0 pypi_0 pypi
tzdata 2024.1 pypi_0 pypi
uc-micro-py 1.0.3 pypi_0 pypi
urllib3 2.2.1 pypi_0 pypi
uvicorn 0.29.0 pypi_0 pypi
virtualenv 20.25.1 pypi_0 pypi
websockets 12.0 pypi_0 pypi
wget 3.2 pypi_0 pypi
wheel 0.41.2 py38hd43f75c_0
xxhash 3.4.1 pypi_0 pypi
xz 5.4.6 h998d150_0
yarl 1.9.4 pypi_0 pypi
zipp 3.18.1 pypi_0 pypi
zlib 1.2.13 h998d150_0
zstandard 0.22.0 pypi_0 pypi
(awq) iota@iota-desktop:~$ conda list > env.txt
(awq) iota@iota-desktop:~$ cat env.txt
# packages in environment at /home/iota/anaconda3/envs/awq:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 51_gnu
absl-py 2.1.0 pypi_0 pypi
accelerate 0.28.0 pypi_0 pypi
aiofiles 23.2.1 pypi_0 pypi
aiohttp 3.9.3 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
altair 5.2.0 pypi_0 pypi
anyio 4.3.0 pypi_0 pypi
async-timeout 4.0.3 pypi_0 pypi
attributedict 0.3.0 pypi_0 pypi
attrs 23.2.0 pypi_0 pypi
awq 0.1.0 pypi_0 pypi
awq-inference-engine 0.0.0 pypi_0 pypi
blessings 1.7 pypi_0 pypi
bzip2 1.0.8 h998d150_5
ca-certificates 2024.3.11 hd43f75c_0
cachetools 5.3.3 pypi_0 pypi
certifi 2024.2.2 pypi_0 pypi
chardet 5.2.0 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
click 8.1.7 pypi_0 pypi
codecov 2.1.13 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
coloredlogs 15.0.1 pypi_0 pypi
colour-runner 0.1.1 pypi_0 pypi
contourpy 1.1.1 pypi_0 pypi
coverage 7.4.4 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
dataproperty 1.0.1 pypi_0 pypi
datasets 2.18.0 pypi_0 pypi
deepdiff 6.7.1 pypi_0 pypi
dill 0.3.8 pypi_0 pypi
distlib 0.3.8 pypi_0 pypi
distro 1.9.0 pypi_0 pypi
dockerhub-api 0.5 pypi_0 pypi
exceptiongroup 1.2.0 pypi_0 pypi
fastapi 0.110.0 pypi_0 pypi
ffmpy 0.3.2 pypi_0 pypi
filelock 3.13.1 pypi_0 pypi
fonttools 4.50.0 pypi_0 pypi
frozenlist 1.4.1 pypi_0 pypi
fsspec 2024.2.0 pypi_0 pypi
furl 2.1.3 pypi_0 pypi
gguf 0.6.0 pypi_0 pypi
gradio 3.35.2 pypi_0 pypi
gradio-client 0.2.9 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
hf-transfer 0.1.6 pypi_0 pypi
httpcore 1.0.4 pypi_0 pypi
httpx 0.27.0 pypi_0 pypi
huggingface-hub 0.23.0 pypi_0 pypi
humanfriendly 10.0 pypi_0 pypi
idna 3.6 pypi_0 pypi
importlib-resources 5.13.0 pypi_0 pypi
inspecta 0.1.3 pypi_0 pypi
jinja2 3.1.3 pypi_0 pypi
joblib 1.3.2 pypi_0 pypi
jsonlines 4.0.0 pypi_0 pypi
jsonschema 4.21.1 pypi_0 pypi
jsonschema-specifications 2023.12.1 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
ld_impl_linux-aarch64 2.38 h8131f2d_1
libffi 3.4.4 h419075a_0
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.41.5 h998d150_0
linkify-it-py 2.0.3 pypi_0 pypi
lm-eval 0.3.0 pypi_0 pypi
markdown-it-py 2.2.0 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
matplotlib 3.7.5 pypi_0 pypi
mbstrdecoder 1.1.3 pypi_0 pypi
mdit-py-plugins 0.3.3 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
multidict 6.0.5 pypi_0 pypi
multiprocess 0.70.16 pypi_0 pypi
ncurses 6.4 h419075a_0
networkx 3.1 pypi_0 pypi
nltk 3.8.1 pypi_0 pypi
numexpr 2.8.6 pypi_0 pypi
numpy 1.24.4 pypi_0 pypi
openai 1.14.2 pypi_0 pypi
openssl 3.0.13 h2f4d8fa_0
ordered-set 4.1.0 pypi_0 pypi
orderedmultidict 1.0.1 pypi_0 pypi
orjson 3.9.15 pypi_0 pypi
packaging 24.0 pypi_0 pypi
pandas 2.0.3 pypi_0 pypi
pathvalidate 3.2.0 pypi_0 pypi
pillow 10.2.0 pypi_0 pypi
pip 24.0 pypi_0 pypi
pkgutil-resolve-name 1.3.10 pypi_0 pypi
platformdirs 4.2.0 pypi_0 pypi
pluggy 1.4.0 pypi_0 pypi
portalocker 2.8.2 pypi_0 pypi
protobuf 5.26.0 pypi_0 pypi
psutil 5.9.8 pypi_0 pypi
pyarrow 15.0.2 pypi_0 pypi
pyarrow-hotfix 0.6 pypi_0 pypi
pybind11 2.11.1 pypi_0 pypi
pycountry 23.12.11 pypi_0 pypi
pydantic 1.10.14 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pygments 2.17.2 pypi_0 pypi
pyparsing 3.1.2 pypi_0 pypi
pyproject-api 1.6.1 pypi_0 pypi
pytablewriter 1.2.0 pypi_0 pypi
python 3.8.19 h4bb2201_0
python-dateutil 2.9.0.post0 pypi_0 pypi
python-multipart 0.0.9 pypi_0 pypi
pytz 2024.1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 8.2 h998d150_0
referencing 0.34.0 pypi_0 pypi
regex 2023.12.25 pypi_0 pypi
requests 2.31.0 pypi_0 pypi
rootpath 0.1.1 pypi_0 pypi
rouge-score 0.1.2 pypi_0 pypi
rpds-py 0.18.0 pypi_0 pypi
sacrebleu 1.5.0 pypi_0 pypi
safetensors 0.4.2 pypi_0 pypi
scikit-learn 1.3.2 pypi_0 pypi
scipy 1.10.1 pypi_0 pypi
semantic-version 2.10.0 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
setuptools 68.2.2 py38hd43f75c_0
six 1.16.0 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
sqlite 3.41.2 h998d150_0
sqlitedict 2.1.0 pypi_0 pypi
starlette 0.36.3 pypi_0 pypi
sympy 1.12 pypi_0 pypi
tabledata 1.3.3 pypi_0 pypi
tcolorpy 0.1.4 pypi_0 pypi
termcolor 2.4.0 pypi_0 pypi
texttable 1.7.0 pypi_0 pypi
threadpoolctl 3.4.0 pypi_0 pypi
tk 8.6.12 h241ca14_0
tokenizers 0.19.1 pypi_0 pypi
toml 0.10.2 pypi_0 pypi
tomli 2.0.1 pypi_0 pypi
toolz 0.12.1 pypi_0 pypi
torch 2.0.0+nv23.5 pypi_0 pypi
tox 4.14.1 pypi_0 pypi
tqdm 4.66.2 pypi_0 pypi
tqdm-multiprocess 0.0.11 pypi_0 pypi
transformers 4.41.0.dev0 pypi_0 pypi
typepy 1.3.2 pypi_0 pypi
typing-extensions 4.10.0 pypi_0 pypi
tzdata 2024.1 pypi_0 pypi
uc-micro-py 1.0.3 pypi_0 pypi
urllib3 2.2.1 pypi_0 pypi
uvicorn 0.29.0 pypi_0 pypi
virtualenv 20.25.1 pypi_0 pypi
websockets 12.0 pypi_0 pypi
wget 3.2 pypi_0 pypi
wheel 0.41.2 py38hd43f75c_0
xxhash 3.4.1 pypi_0 pypi
xz 5.4.6 h998d150_0
yarl 1.9.4 pypi_0 pypi
zipp 3.18.1 pypi_0 pypi
zlib 1.2.13 h998d150_0
zstandard 0.22.0 pypi_0 pypi@pprp experimented with it , the quantized phi3 comes in just 54mb. Is there something i missed
This pull request introduces support for the phi3 architecture within the llm-awq.
The transformer package for phi3 is not released. So I developed it using 4.41.0.dev0.
This PR works fine on RTX4090 and Jetson Orin 8G.