Awesome work! Some issues when trying to re-implement inferencing on local/cloud machines.

[Mattz-CE]

Awesome work!

I had some issues when trying to test it with custom data via gradio interface. Could you provide some insights on how the apis are called? Or perhaps any insights on implementing with inference endpoint or local inference setup?

result = client.predict(
    "x",    # str  in 'parameter_3' Textbox component
    "https://cdn.discordapp.com/attachments/879419021060112394/1294109267518947418/image.png?ex=6709d0ac&is=67087f2c&hm=6519550764c5e505553329e3896069b3d4733ab9a547ca020a65f274756c3b35&", # filepath  in 'parameter_11' Image component
    "Default",  # Literal['Crop', 'Resize', 'Pad', 'Default']  in 'Preprocess for non-square image' Radio component
    api_name="/add_text"
)
print(result)  # Should print the updated image and other details

[1494](file:///C:/tmp/self-operating-computer/.conda/Lib/site-packages/gradio_client/client.py:1494) def result(self, timeout: float | None = None) -> Any:
...
   [1246](file:///C:/tmp/self-operating-computer/.conda/Lib/site-packages/gradhttps://docs.github.com/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntaxio_client/client.py:1246)             "The upstream Gradio app has raised an exception: "
   [1247](file:///C:/tmp/self-operating-computer/.conda/Lib/site-packages/gradio_client/client.py:1247)             + result["error"]
   [1248](file:///C:/tmp/self-operating-computer/.conda/Lib/site-packages/gradio_client/client.py:1248)         )

AppError: The upstream Gradio app has raised an exception but has not enabled verbose error reporting. To enable, set show_error=True in launch().

Inference Endpoints:

[Server message]Endpoint failed to start
See details
Exit code: 3. Reason: st):
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 993, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 695, in __getitem__
    raise KeyError(key)
KeyError: 'llava_llama'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 693, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 569, in __aenter__
    await self._router.startup()
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 670, in startup
    await handler()
  File "/app/webservice_starlette.py", line 62, in prepare_model_artifacts
    inference_handler = get_inference_handler_either_custom_or_default_handler(
  File "/app/huggingface_inference_toolkit/handler.py", line 96, in get_inference_handler_either_custom_or_default_handler
    return HuggingFaceHandler(model_dir=model_dir, task=task)
  File "/app/huggingface_inference_toolkit/handler.py", line 19, in __init__
    self.pipeline = get_pipeline(
  File "/app/huggingface_inference_toolkit/utils.py", line 261, in get_pipeline
    hf_pipeline = pipeline(task=task, model=model_dir, device=device, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/pipelines/__init__.py", line 805, in pipeline
    config = AutoConfig.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 995, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `llava_llama` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

Application startup failed. Exiting.

[boyugou]

Hi Matthew! Thanks for your interest of our work. We will organize a very clean version of code for inference very soon. You don't really need to make one from the demo code. It will actually be mostly from llava 1.5 & NeXT codebase, with slight changes to the image processing.

[boyugou]

I know many people want to try our model locally, this is our first priority. Sorry for the relatively slow pace of code organization due to autumn break and other projects.

[Mattz-CE]

Glad to hear! Thanks for the update on the progress, and I'm looking forward to it!