Problems about launching demo

[zhtstar]

Thank you for your excellent work. I encountered some problems when trying to use the geochat_demo.py file. Although I tried changing the conda environment many times and changing computers and servers, my problem still did not solve. I'm looking for your help here. My problem is:

1. In the "Install" section you wrote "Clone this repository and navigate to LLaVA folder". What is "LLaVA folder"? Do I need to create a new folder myself? If so, what is the directory structure of this folder compared to other files?

2. When I configure the environment according to your instructions and execute the code "python geochat_demo.py --model-path ./models", I will encounter the following error: (geochat) root@Y9000K:/mnt/d/GeoChat# python geochat_demo.py --model-path ./models /root/anaconda3/envs/geochat/lib/python3.8/site-packages/gradio_client/documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Parallel'>: No known documentation group for module 'gradio.mix' warnings.warn(f"Could not get documentation group for {cls}: {exc}") /root/anaconda3/envs/geochat/lib/python3.8/site-packages/gradio_client/documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Series'>: No known documentation group for module 'gradio.mix' warnings.warn(f"Could not get documentation group for {cls}: {exc}") Initializing Chat Loading checkpoint shards: 50%|████████████████████████████████████████████████████ | 1/2 [02:06<02:06, 126.92s/it] Traceback (most recent call last): File "geochat_demo.py", line 53, in tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, args.load_8bit, args.load_4bit, device=args.device) File "/mnt/d/GeoChat/geochat/model/builder.py", line 125, in load_pretrained_model model = AutoModelForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, kwargs) File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 493, in from_pretrained return model_class.from_pretrained( File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained ) = cls._load_pretrained_model( File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3260, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/modeling_utils.py", line 717, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, set_module_kwargs) File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 348, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect.

3. If I do not use the model from GeoChat-7B, but directly use "facebook/opt-350m", the interface can be loaded, but nothing happens when clicking the "send" button

Sorry to bother you, I haven't been able to solve this problem even though I tried many times. I hope I can get your help.

[zhtstar]

Sorry to bother you, there are some other problems when trying to use the geochat_demo.py file: even I try to use the example you give, it will only prompt "Error"

In the terminal, the outputs are follows: 1. Traceback (most recent call last): File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/routes.py", line 437, in run_predict output = await app.get_blocks().process_api( File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/blocks.py", line 1352, in process_api result = await self.call_function( File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/blocks.py", line 1093, in call_function prediction = await utils.async_iteration(iterator) File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/utils.py", line 341, in async_iteration return await iterator.anext() File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/utils.py", line 334, in anext return await anyio.to_thread.run_sync( File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread return await future File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run result = context.run(func, *args) File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/utils.py", line 317, in run_sync_iterator_async return next(iterator) File "geochat_demo.py", line 532, in gradio_stream_answer streamer = chat.stream_answer(conv=chat_state, File "/root/GeoChat/geochat/conversation.py", line 458, in stream_answer generation_kwargs = self.answer_prepare(conv, img_list, **kargs) File "/root/GeoChat/geochat/conversation.py", line 432, in answer_prepare images=img_list[0], IndexError: list index out of range

2. Traceback (most recent call last): File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/routes.py", line 437, in run_predict output = await app.get_blocks().process_api( File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/blocks.py", line 1352, in process_api result = await self.call_function( File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/blocks.py", line 1093, in call_function prediction = await utils.async_iteration(iterator) File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/utils.py", line 341, in async_iteration return await iterator.anext() File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/utils.py", line 334, in anext return await anyio.to_thread.run_sync( File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread return await future File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run result = context.run(func, args) File "/root/miniconda3/envs/geochat/lib/python3.8/site-packages/gradio/utils.py", line 317, in run_sync_iterator_async return next(iterator) File "geochat_demo.py", line 531, in gradio_stream_answer chat.encode_img(img_list) File "/root/GeoChat/geochat/conversation.py", line 498, in encode_img image = process_images_demo([raw_image], self.vis_processor ) File "/root/GeoChat/geochat/mm_utils.py", line 48, in process_images_demo image = expand2square(image, tuple(int(x255) for x in image_processor.image_mean)) AttributeError: 'NoneType' object has no attribute 'image_mean'

[KjAeRsTuIsK]

Hi @zhtstar, can you please update the code from the repo once and rerun it.

[xianjiedai]

Hi @KjAeRsTuIsK ,

I got the same error and the code is already is the latest. I have followed the instructions on ReadMe to set up environment and update the code.

Lokking forward to hearing from you soon!

[xianjiedai]

Hi @KjAeRsTuIsK ,

I have fixed the above problem but I got a new one with the example provided. Could you help me with this error? I used the example image and sentence as inputs.

[tkajihara-nec]

Hi @xianjiedai

I got the same error as below. Could you let me know how you fixed?

'AttributeError: 'NoneType' object has no attribute 'image_mean'

[xianjiedai]

Hi @xianjiedai

I got the same error as below. Could you let me know how you fixed?

'AttributeError: 'NoneType' object has no attribute 'image_mean'

By properly setting the model path. If you check builder.py, you will notice that

if 'geochat' in model_name.lower():

requires geochat in the model path and you may need to change the path or the name of the folder that contains model files cloned from hugging face.

If you can fix it and encounter the same problem as I do:

Hi @KjAeRsTuIsK ,

I have fixed the above problem but I got a new one with the example provided. Could you help me with this error? I used the example image and sentence as inputs.

please let me know

[tkajihara-nec]

@xianjiedai thank you!

[ZFU123456]

Thank you for your excellent work. I encountered some problems when trying to use the geochat_demo.py file. Although I tried changing the conda environment many times and changing computers and servers, my problem still did not solve. I'm looking for your help here. My problem is:

1. In the "Install" section you wrote "Clone this repository and navigate to LLaVA folder". What is "LLaVA folder"? Do I need to create a new folder myself? If so, what is the directory structure of this folder compared to other files?
2. When I configure the environment according to your instructions and execute the code "python geochat_demo.py --model-path ./models", I will encounter the following error: (geochat) root@Y9000K:/mnt/d/GeoChat# python geochat_demo.py --model-path ./models /root/anaconda3/envs/geochat/lib/python3.8/site-packages/gradio_client/documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Parallel'>: No known documentation group for module 'gradio.mix' warnings.warn(f"Could not get documentation group for {cls}: {exc}") /root/anaconda3/envs/geochat/lib/python3.8/site-packages/gradio_client/documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Series'>: No known documentation group for module 'gradio.mix' warnings.warn(f"Could not get documentation group for {cls}: {exc}") Initializing Chat Loading checkpoint shards: 50%|████████████████████████████████████████████████████ | 1/2 [02:06<02:06, 126.92s/it] Traceback (most recent call last): File "geochat_demo.py", line 53, in tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, args.load_8bit, args.load_4bit, device=args.device) File "/mnt/d/GeoChat/geochat/model/builder.py", line 125, in load_pretrained_model model = AutoModelForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, kwargs) File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 493, in from_pretrained return model_class.from_pretrained( File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained ) = cls._load_pretrained_model( File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3260, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/modeling_utils.py", line 717, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, set_module_kwargs) File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 348, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect.
3. If I do not use the model from GeoChat-7B, but directly use "facebook/opt-350m", the interface can be loaded, but nothing happens when clicking the "send" button

Sorry to bother you, I haven't been able to solve this problem even though I tried many times. I hope I can get your help.

请问你解决了吗？

[fyan1024]

Thank you for your excellent work. I encountered some problems when trying to use the geochat_demo.py file. Although I tried changing the conda environment many times and changing computers and servers, my problem still did not solve. I'm looking for your help here. My problem is:

1. In the "Install" section you wrote "Clone this repository and navigate to LLaVA folder". What is "LLaVA folder"? Do I need to create a new folder myself? If so, what is the directory structure of this folder compared to other files?

2. When I configure the environment according to your instructions and execute the code "python geochat_demo.py --model-path ./models", I will encounter the following error:
   (geochat) root@Y9000K:/mnt/d/GeoChat# python geochat_demo.py --model-path ./models
   /root/anaconda3/envs/geochat/lib/python3.8/site-packages/gradio_client/documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Parallel'>: No known documentation group for module 'gradio.mix'
   warnings.warn(f"Could not get documentation group for {cls}: {exc}")
   /root/anaconda3/envs/geochat/lib/python3.8/site-packages/gradio_client/documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Series'>: No known documentation group for module 'gradio.mix'
   warnings.warn(f"Could not get documentation group for {cls}: {exc}")
   Initializing Chat
   Loading checkpoint shards:  50%|████████████████████████████████████████████████████                                                    | 1/2 [02:06<02:06, 126.92s/it]
   Traceback (most recent call last):
   File "geochat_demo.py", line 53, in 
   tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, args.load_8bit, args.load_4bit, device=args.device)
   File "/mnt/d/GeoChat/geochat/model/builder.py", line 125, in load_pretrained_model
   model = AutoModelForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **kwargs)
   File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 493, in from_pretrained
   return model_class.from_pretrained(
   File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained
   ) = cls._load_pretrained_model(
   File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3260, in _load_pretrained_model
   new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
   File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/transformers/modeling_utils.py", line 717, in _load_state_dict_into_meta_model
   set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
   File "/root/anaconda3/envs/geochat/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 348, in set_module_tensor_to_device
   raise ValueError(
   ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect.

3. If I do not use the model from GeoChat-7B, but directly use "facebook/opt-350m", the interface can be loaded, but nothing happens when clicking the "send" button

Sorry to bother you, I haven't been able to solve this problem even though I tried many times. I hope I can get your help.

I also get the error ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect. Anyone know whats going on with this?

[JoyLinWQ]

hi, i have renamed the folder containing model weights to contain geochat and it solved the error 'AttributeError: 'NoneType' object has no attribute 'image_mean' and the demo images run fine.

However, when i upload my own image, it does not appear. An Error appears in the demo UI too. The following error in terminal appears: AttributeError: 'NoneType' object has no attribute 'rsplit'. Did anyone encounter this and how can it be resolved? Thanks!

Traceback (most recent call last):
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/routes.py", line 437, in run_predict
    output = await app.get_blocks().process_api(
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/blocks.py", line 1350, in process_api
    inputs = self.preprocess_data(fn_index, inputs, state)
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/blocks.py", line 1200, in preprocess_data
    processed_input.append(block.preprocess(inputs[i]))
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/components/image.py", line 260, in preprocess
    mask_im = processing_utils.decode_base64_to_image(mask)
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/processing_utils.py", line 51, in decode_base64_to_image
    image_encoded = extract_base64_data(encoding)
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/processing_utils.py", line 42, in extract_base64_data
    return x.rsplit(",", 1)[-1]
AttributeError: 'NoneType' object has no attribute 'rsplit'

[JoyLinWQ]

hi, i have renamed the folder containing model weights to contain geochat and it solved the error 'AttributeError: 'NoneType' object has no attribute 'image_mean' and the demo images run fine.

However, when i upload my own image, it does not appear. An Error appears in the demo UI too. The following error in terminal appears: AttributeError: 'NoneType' object has no attribute 'rsplit'. Did anyone encounter this and how can it be resolved? Thanks!

Traceback (most recent call last):
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/routes.py", line 437, in run_predict
    output = await app.get_blocks().process_api(
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/blocks.py", line 1350, in process_api
    inputs = self.preprocess_data(fn_index, inputs, state)
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/blocks.py", line 1200, in preprocess_data
    processed_input.append(block.preprocess(inputs[i]))
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/components/image.py", line 260, in preprocess
    mask_im = processing_utils.decode_base64_to_image(mask)
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/processing_utils.py", line 51, in decode_base64_to_image
    image_encoded = extract_base64_data(encoding)
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/processing_utils.py", line 42, in extract_base64_data
    return x.rsplit(",", 1)[-1]
AttributeError: 'NoneType' object has no attribute 'rsplit'

hi, i have renamed the folder containing model weights to contain geochat and it solved the error 'AttributeError: 'NoneType' object has no attribute 'image_mean' and the demo images run fine.

However, when i upload my own image, it does not appear. An Error appears in the demo UI too. The following error in terminal appears: AttributeError: 'NoneType' object has no attribute 'rsplit'. Did anyone encounter this and how can it be resolved? Thanks!

Traceback (most recent call last):
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/routes.py", line 437, in run_predict
    output = await app.get_blocks().process_api(
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/blocks.py", line 1350, in process_api
    inputs = self.preprocess_data(fn_index, inputs, state)
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/blocks.py", line 1200, in preprocess_data
    processed_input.append(block.preprocess(inputs[i]))
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/components/image.py", line 260, in preprocess
    mask_im = processing_utils.decode_base64_to_image(mask)
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/processing_utils.py", line 51, in decode_base64_to_image
    image_encoded = extract_base64_data(encoding)
  File "/opt/conda/envs/geochat2/lib/python3.10/site-packages/gradio/processing_utils.py", line 42, in extract_base64_data
    return x.rsplit(",", 1)[-1]
AttributeError: 'NoneType' object has no attribute 'rsplit'

i realised it can work for images of size 1024 x 768, but for other squarish images like 1185 x 1189, or 1169 x 1140, the error above keeps appearing. One current workaround is to resize the images.