LLaVa Broken

[reddyn12]

With DEBUG=2 error:

Traceback (most recent call last):
  File "/Users/schlime/Code/exo/exo/api/chatgpt_api.py", line 273, in handle_post_chat_completions
    await asyncio.wait_for(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/exo/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/Users/schlime/Code/exo/exo/orchestration/standard_node.py", line 98, in process_prompt
    resp = await self._process_prompt(base_shard, prompt, image_str, request_id, inference_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/schlime/Code/exo/exo/orchestration/standard_node.py", line 134, in _process_prompt
    result, inference_state, is_finished = await self.inference_engine.infer_prompt(request_id, shard, prompt, 
image_str, inference_state=inference_state)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/schlime/Code/exo/exo/inference/mlx/sharded_inference_engine.py", line 23, in infer_prompt
    inputs = await loop.run_in_executor(self.executor, self.tokenizer, prompt, image, return_tensors="np")
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: BaseEventLoop.run_in_executor() got an unexpected keyword argument 'return_tensors'

[AlexCheema]

@varshith15 could you take a look?

[reddyn12]

Poking through code rn, and I think it has to do with self.tokenizer needing return_tensors and in a refactor, the arg got moved. Maybe a functools.partial?

[varshith15]

@AlexCheema this is fixed https://github.com/exo-explore/exo/pull/329

[reddyn12]

Works!!