Fail to run video_audio_demo.py with ValueError

[superobk]

As titled, I installed all the packages as well as flash_attn following "Training" instructions.

However, when trying to run video_audio_demo, it will trigger the error to specify the ViT model is missing. It is any configuration I could specify the path of ViT model? Please kindly advise. Thanks!

root@bo-deployment-work-68c488599-r4qrz:/Code/VITA# CUDA_VISIBLE_DEVICES=0,1 python video_audio_demo.py --model_path /Model/Weight/VITA-MLLM/VITA/models--VITA-MLLM--VITA/VITA_ckpt/ --image_path asset/vita_log2.png --model_type mixtral-8x7b --conv_mode mixtral_two --question "请描述这张图片。" Please build and install Nvidia apex package with option '--cuda_ext' according to https://github.com/NVIDIA/apex#from-source . Please install mamba_ssm to use MambaSSM component. /usr/local/lib/python3.10/dist-packages/torch/_jit_internal.py:739: FutureWarning: ignore(True) has been deprecated. TorchScript will now drop the function call on compilation. Use torch.jit.unused now. {} warnings.warn( Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn conn = connection.create_connection( File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1017, in _validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 353, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 181, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f35ab4841c0>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 756, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /InternViT-300M-448px/resolve/main/config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f35ab4841c0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py", line 1746, in _get_metadata_or_catch_error metadata = get_hf_file_metadata( File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn return fn(args, kwargs) File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py", line 1666, in get_hf_file_metadata r = _request_wrapper( File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py", line 364, in _request_wrapper response = _request_wrapper( File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py", line 387, in _request_wrapper response = get_session().request(method=method, url=url, params) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, kwargs) File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_http.py", line 93, in send return super().send(request, args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 700, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /InternViT-300M-448px/resolve/main/config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f35ab4841c0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), '(Request ID: 24aaa941-a7d7-4a20-8e28-2a3758260c0f)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py", line 399, in cached_file resolved_file = hf_hub_download( File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f return f(*args, *kwargs) File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn return fn(args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py", line 1232, in hf_hub_download return _hf_hub_download_to_cache_dir( File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py", line 1339, in _hf_hub_download_to_cache_dir _raise_on_head_call_error(head_call_error, force_download, local_files_only) File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py", line 1857, in _raise_on_head_call_error raise LocalEntryNotFoundError( huggingface_hub.errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/Code/VITA/video_audio_demo.py", line 163, in tokenizer, model, image_processor, context_len = load_pretrained_model( File "/Code/VITA/vita/model/builder.py", line 220, in load_pretrained_model model = VITAMixtralForCausalLM.from_pretrained( File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3626, in from_pretrained model = cls(config, *model_args, model_kwargs) File "/Code/VITA/vita/model/language_model/vita_mixtral.py", line 237, in init self.model = VITAMixtralModel(config) File "/Code/VITA/vita/model/language_model/vita_mixtral.py", line 229, in init super(VITAMixtralModel, self).init(config) File "/Code/VITA/vita/model/vita_arch.py", line 18, in init self.vision_tower = build_vision_tower( File "/Code/VITA/vita/model/multimodal_encoder/builder.py", line 38, in build_vision_tower return InternViTVisionTower(vision_tower, args=vision_tower_cfg, kwargs) File "/Code/VITA/vita/model/multimodal_encoder/internvit/internvit_encoder.py", line 21, in init self.cfg_only = AutoConfig.from_pretrained( File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 934, in from_pretrained config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 632, in get_config_dict config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 689, in _get_config_dict resolved_config_file = cached_file( File "/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py", line 442, in cached_file raise EnvironmentError( OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like InternViT-300M-448px is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.

[linhaojia13]

Hi @superobk , modify themm_vision_tower in the config.json.

[learn01one]

Hi @superobk , modify themm_vision_tower in the config.json.

hello,how can we modify the mm_vision_tower?local path?thanks~

[superobk]

Hi @superobk , modify themm_vision_tower in the config.json.

hello,how can we modify the mm_vision_tower?local path?thanks~

The .config.json is in the folder VITA_ckpt, you need to change both "mm_audio_encoder" and "mm_vision_tower" to your local path