search

baaivision / EVA

EVA Series: Visual Representation Fantasies from BAAI
MIT License
2.29k stars 167 forks source link

Can EVA-CLIP-8B and EVA-CLIP-18B support quantization? #142

Closed Shimooth closed 7 months ago

Shimooth commented 8 months ago

Can EVA-CLIP-8B and EVA-CLIP-18B support quantization? My device doesn't have such high specifications, and I'm worried I won't be able to run these models. My device currently has only a little over 30GB of VRAM available.

Quan-Sun commented 8 months ago

Hi @Shimooth, Thank you for your interest in our work. Unfortunately, we currently do not have enough bandwidth to do quantization for EVA-CLIP-8B and EVA-CLIP-18B. We will update once quantization versions for these models become available in the future. Thank you for your understanding.

Shimooth commented 8 months ago

Hi @Shimooth, Thank you for your interest in our work. Unfortunately, we currently do not have enough bandwidth to do quantization for EVA-CLIP-8B and EVA-CLIP-18B. We will update once quantization versions for these models become available in the future. Thank you for your understanding.

Thank you for your reply. I am eagerly looking forward to the quantized 18B model. Given my current machine configuration, I might not be able to use the full model. Additionally, I encountered some errors while using PyTorch version of the code. Could you please help me take a look? The following is the error message and my pip list.

Error caught was: No module named 'triton' Please 'pip install apex' Traceback (most recent call last): File "ptctest.py", line 12, in model, , processor = create_model_and_transforms(model_name, pretrained, force_custom_clip=True) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/factory.py", line 493, in create_model_and_transforms model = create_model( File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/factory.py", line 386, in create_model model = CustomCLIP(**model_cfg, cast_dtype=cast_dtype) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/model.py", line 310, in init self.visual = _build_vision_tower(embed_dim, vision_cfg, quick_gelu, cast_dtype) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/model.py", line 139, in build_vision_tower visual = EVAVisionTransformer( File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 435, in init self.head.bias.data.mul(init_scale) AttributeError: 'NoneType' object has no attribute 'data'

pip list: annotated-types 0.6.0 anykeystore 0.2 apex 0.9.10.dev0 appdirs 1.4.4 blobfile 2.1.1 braceexpand 0.1.7 certifi 2024.2.2 charset-normalizer 3.3.2 click 8.1.7 cryptacular 1.6.2 deepspeed 0.8.3 defusedxml 0.7.1 docker-pycreds 0.4.0 einops 0.7.0 exceptiongroup 1.2.0 filelock 3.13.1 fsspec 2024.2.0 ftfy 6.1.3 gitdb 4.0.11 GitPython 3.1.42 greenlet 3.0.3 hjson 3.1.0 huggingface-hub 0.21.2 hupper 1.12.1 idna 3.6 iniconfig 2.0.0 iopath 0.1.10 lxml 4.9.4 MarkupSafe 2.1.5 mypy 1.8.0 mypy-extensions 1.0.0 ninja 1.11.1.1 numpy 1.24.4 oauthlib 3.2.2 packaging 23.2 pandas 2.0.3 PasteDeploy 3.1.0 pbkdf2 1.3 pillow 10.2.0 pip 23.3.1 plaster 1.1.2 plaster-pastedeploy 1.0.1 pluggy 1.4.0 portalocker 2.8.2 protobuf 4.25.3 psutil 5.9.8 py-cpuinfo 9.0.0 pycryptodomex 3.20.0 pydantic 2.6.3 pydantic_core 2.16.3 pyramid 2.0.2 pyramid-mailer 0.15.1 pyre-extensions 0.0.23 pytest 8.0.2 python-dateutil 2.8.2 python3-openid 3.2.0 pytz 2024.1 PyYAML 6.0.1 regex 2023.12.25 repoze.sendmail 4.4.1 requests 2.31.0 requests-oauthlib 1.3.1 scipy 1.10.1 sentry-sdk 1.40.6 setproctitle 1.3.3 setuptools 68.2.2 sh 2.0.6 six 1.16.0 smmap 5.0.1 SQLAlchemy 2.0.27 tensorboardX 2.6.2.2 timm 0.5.4 tokenizers 0.13.3 tomli 2.0.1 torch 1.12.1+cu116 torchvision 0.13.1+cu116 tqdm 4.66.2 transaction 4.0 transformers 4.28.1 translationstring 1.4 typing_extensions 4.10.0 typing-inspect 0.9.0 tzdata 2024.1 urllib3 2.2.1 velruse 1.1.1 venusian 3.1.0 wandb 0.16.3 wcwidth 0.2.13 webdataset 0.2.86 WebOb 1.8.7 wheel 0.41.2 WTForms 3.1.2 wtforms-recaptcha 0.3.2 xformers 0.0.13 zope.deprecation 5.0 zope.interface 6.2 zope.sqlalchemy 3.1

Quan-Sun commented 8 months ago

Hi @Shimooth, Thank you for your interest in our work. Unfortunately, we currently do not have enough bandwidth to do quantization for EVA-CLIP-8B and EVA-CLIP-18B. We will update once quantization versions for these models become available in the future. Thank you for your understanding.

Thank you for your reply. I am eagerly looking forward to the quantized 18B model. Given my current machine configuration, I might not be able to use the full model. Additionally, I encountered some errors while using PyTorch version of the code. Could you please help me take a look? The following is the error message and my pip list.

Error caught was: No module named 'triton' Please 'pip install apex' Traceback (most recent call last): File "ptctest.py", line 12, in model, , processor = create_model_and_transforms(model_name, pretrained, force_custom_clip=True) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/factory.py", line 493, in create_model_and_transforms model = create_model( File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/factory.py", line 386, in create_model model = CustomCLIP(**model_cfg, cast_dtype=cast_dtype) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/model.py", line 310, in init self.visual = _build_vision_tower(embed_dim, vision_cfg, quick_gelu, cast_dtype) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/model.py", line 139, in build_vision_tower visual = EVAVisionTransformer( File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 435, in init self.head.bias.data.mul(init_scale) AttributeError: 'NoneType' object has no attribute 'data'

pip list: annotated-types 0.6.0 anykeystore 0.2 apex 0.9.10.dev0 appdirs 1.4.4 blobfile 2.1.1 braceexpand 0.1.7 certifi 2024.2.2 charset-normalizer 3.3.2 click 8.1.7 cryptacular 1.6.2 deepspeed 0.8.3 defusedxml 0.7.1 docker-pycreds 0.4.0 einops 0.7.0 exceptiongroup 1.2.0 filelock 3.13.1 fsspec 2024.2.0 ftfy 6.1.3 gitdb 4.0.11 GitPython 3.1.42 greenlet 3.0.3 hjson 3.1.0 huggingface-hub 0.21.2 hupper 1.12.1 idna 3.6 iniconfig 2.0.0 iopath 0.1.10 lxml 4.9.4 MarkupSafe 2.1.5 mypy 1.8.0 mypy-extensions 1.0.0 ninja 1.11.1.1 numpy 1.24.4 oauthlib 3.2.2 packaging 23.2 pandas 2.0.3 PasteDeploy 3.1.0 pbkdf2 1.3 pillow 10.2.0 pip 23.3.1 plaster 1.1.2 plaster-pastedeploy 1.0.1 pluggy 1.4.0 portalocker 2.8.2 protobuf 4.25.3 psutil 5.9.8 py-cpuinfo 9.0.0 pycryptodomex 3.20.0 pydantic 2.6.3 pydantic_core 2.16.3 pyramid 2.0.2 pyramid-mailer 0.15.1 pyre-extensions 0.0.23 pytest 8.0.2 python-dateutil 2.8.2 python3-openid 3.2.0 pytz 2024.1 PyYAML 6.0.1 regex 2023.12.25 repoze.sendmail 4.4.1 requests 2.31.0 requests-oauthlib 1.3.1 scipy 1.10.1 sentry-sdk 1.40.6 setproctitle 1.3.3 setuptools 68.2.2 sh 2.0.6 six 1.16.0 smmap 5.0.1 SQLAlchemy 2.0.27 tensorboardX 2.6.2.2 timm 0.5.4 tokenizers 0.13.3 tomli 2.0.1 torch 1.12.1+cu116 torchvision 0.13.1+cu116 tqdm 4.66.2 transaction 4.0 transformers 4.28.1 translationstring 1.4 typing_extensions 4.10.0 typing-inspect 0.9.0 tzdata 2024.1 urllib3 2.2.1 velruse 1.1.1 venusian 3.1.0 wandb 0.16.3 wcwidth 0.2.13 webdataset 0.2.86 WebOb 1.8.7 wheel 0.41.2 WTForms 3.1.2 wtforms-recaptcha 0.3.2 xformers 0.0.13 zope.deprecation 5.0 zope.interface 6.2 zope.sqlalchemy 3.1

Thanks for pointing that out. The initialization of head.bias should be removed due to no head.bias in EVA-CLIP-8B and EVA-CLIP-18B. have fixed that in line

Shimooth commented 8 months ago

Hi @Shimooth, Thank you for your interest in our work. Unfortunately, we currently do not have enough bandwidth to do quantization for EVA-CLIP-8B and EVA-CLIP-18B. We will update once quantization versions for these models become available in the future. Thank you for your understanding.

Thank you for your reply. I am eagerly looking forward to the quantized 18B model. Given my current machine configuration, I might not be able to use the full model. Additionally, I encountered some errors while using PyTorch version of the code. Could you please help me take a look? The following is the error message and my pip list. Error caught was: No module named 'triton' Please 'pip install apex' Traceback (most recent call last): File "ptctest.py", line 12, in model, , processor = create_model_and_transforms(model_name, pretrained, force_custom_clip=True) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/factory.py", line 493, in create_model_and_transforms model = create_model( File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/factory.py", line 386, in create_model model = CustomCLIP(**model_cfg, cast_dtype=cast_dtype) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/model.py", line 310, in init self.visual = _build_vision_tower(embed_dim, vision_cfg, quick_gelu, cast_dtype) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/model.py", line 139, in build_vision_tower visual = EVAVisionTransformer( File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 435, in init self.head.bias.data.mul(init_scale) AttributeError: 'NoneType' object has no attribute 'data' pip list: annotated-types 0.6.0 anykeystore 0.2 apex 0.9.10.dev0 appdirs 1.4.4 blobfile 2.1.1 braceexpand 0.1.7 certifi 2024.2.2 charset-normalizer 3.3.2 click 8.1.7 cryptacular 1.6.2 deepspeed 0.8.3 defusedxml 0.7.1 docker-pycreds 0.4.0 einops 0.7.0 exceptiongroup 1.2.0 filelock 3.13.1 fsspec 2024.2.0 ftfy 6.1.3 gitdb 4.0.11 GitPython 3.1.42 greenlet 3.0.3 hjson 3.1.0 huggingface-hub 0.21.2 hupper 1.12.1 idna 3.6 iniconfig 2.0.0 iopath 0.1.10 lxml 4.9.4 MarkupSafe 2.1.5 mypy 1.8.0 mypy-extensions 1.0.0 ninja 1.11.1.1 numpy 1.24.4 oauthlib 3.2.2 packaging 23.2 pandas 2.0.3 PasteDeploy 3.1.0 pbkdf2 1.3 pillow 10.2.0 pip 23.3.1 plaster 1.1.2 plaster-pastedeploy 1.0.1 pluggy 1.4.0 portalocker 2.8.2 protobuf 4.25.3 psutil 5.9.8 py-cpuinfo 9.0.0 pycryptodomex 3.20.0 pydantic 2.6.3 pydantic_core 2.16.3 pyramid 2.0.2 pyramid-mailer 0.15.1 pyre-extensions 0.0.23 pytest 8.0.2 python-dateutil 2.8.2 python3-openid 3.2.0 pytz 2024.1 PyYAML 6.0.1 regex 2023.12.25 repoze.sendmail 4.4.1 requests 2.31.0 requests-oauthlib 1.3.1 scipy 1.10.1 sentry-sdk 1.40.6 setproctitle 1.3.3 setuptools 68.2.2 sh 2.0.6 six 1.16.0 smmap 5.0.1 SQLAlchemy 2.0.27 tensorboardX 2.6.2.2 timm 0.5.4 tokenizers 0.13.3 tomli 2.0.1 torch 1.12.1+cu116 torchvision 0.13.1+cu116 tqdm 4.66.2 transaction 4.0 transformers 4.28.1 translationstring 1.4 typing_extensions 4.10.0 typing-inspect 0.9.0 tzdata 2024.1 urllib3 2.2.1 velruse 1.1.1 venusian 3.1.0 wandb 0.16.3 wcwidth 0.2.13 webdataset 0.2.86 WebOb 1.8.7 wheel 0.41.2 WTForms 3.1.2 wtforms-recaptcha 0.3.2 xformers 0.0.13 zope.deprecation 5.0 zope.interface 6.2 zope.sqlalchemy 3.1

Thanks for pointing that out. The initialization of head.bias should be removed due to no head.bias in EVA-CLIP-8B and EVA-CLIP-18B. have fixed that in line

Thank you for your response. This issue has been resolved, but a new error has occurred. Can you help me see how to solve this? The code is on a remote server, not locally, making it inconvenient to pinpoint the cause of the error. The error message is as follows: Traceback (most recent call last): File "ptc_test.py", line 22, in image_features = model.encode_image(image) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/model.py", line 331, in encode_image features = self.visual(image) File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, kwargs) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 529, in forward x = self.forward_features(x) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 516, in forward_features x = blk(x, rel_pos_bias=rel_pos_bias) File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, *kwargs) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 293, in forward x = x + self.drop_path(self.attn(self.norm1(x), rel_pos_bias=rel_pos_bias, attn_mask=attn_mask)) File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(input, kwargs) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 208, in forward x = xops.memory_efficient_attention(

Shimooth commented 8 months ago

I removed the scale parameter according to the error path, and now it reports a tensor dimension mismatch error, details are as follows: Traceback (most recent call last): File "ptc_test.py", line 22, in image_features = model.encode_image(image) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/model.py", line 331, in encode_image features = self.visual(image) File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, kwargs) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 535, in forward x = self.forward_features(x) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 522, in forward_features x = blk(x, rel_pos_bias=rel_pos_bias) File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, *kwargs) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 299, in forward x = x + self.drop_path(self.attn(self.norm1(x), rel_pos_bias=rel_pos_bias, attn_mask=attn_mask)) File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(input, kwargs) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 214, in forward x = xops.memory_efficient_attention( File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/xformers/ops.py", line 574, in memory_efficient_attention return op.forward_no_grad( File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/xformers/ops.py", line 315, in forward_no_grad return cls.forward( File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/xformers/ops.py", line 353, in forward query = query.reshape([batch * seqlen_q, 1, head_dim_q]) RuntimeError: shape '[257, 1, 32]' is invalid for input of size 1052672

What should I do to resolve this error?

Quan-Sun commented 8 months ago

I removed the scale parameter according to the error path, and now it reports a tensor dimension mismatch error, details are as follows: Traceback (most recent call last): File "ptc_test.py", line 22, in image_features = model.encode_image(image) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/model.py", line 331, in encode_image features = self.visual(image) File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, kwargs) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 535, in forward x = self.forward_features(x) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 522, in forward_features x = blk(x, rel_pos_bias=rel_pos_bias) File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, *kwargs) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 299, in forward x = x + self.drop_path(self.attn(self.norm1(x), rel_pos_bias=rel_pos_bias, attn_mask=attn_mask)) File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(input, kwargs) File "/data/MLLM/EVA-CLIP/EVA/EVA-CLIP-18B/shinji/eva_clip/eva_vit_model.py", line 214, in forward x = xops.memory_efficient_attention( File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/xformers/ops.py", line 574, in memory_efficient_attention return op.forward_no_grad( File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/xformers/ops.py", line 315, in forward_no_grad return cls.forward( File "/home/feihu/miniconda3/envs/eva_clip/lib/python3.8/site-packages/xformers/ops.py", line 353, in forward query = query.reshape([batch * seqlen_q, 1, head_dim_q]) RuntimeError: shape '[257, 1, 32]' is invalid for input of size 1052672

What should I do to resolve this error?

head_dim_q should be 128, but got 32 here. the head dim is set here