Issue with LM_Studio or Local app option advance prompt enhancer

[sleppyrobot]

Hi, i noticed that the LM_Studio ask for an OpenAI key when it should not.

"Invalid or missing OpenAI API key. Keys must be stored in an environment variable (see: ReadMe). ChatGPT request aborted"

The local app option all does this, and triggers the following error in the terminal as well.

ComfyUI/custom_nodes/Plush-for-ComfyUI/api_requests.py", line 200, in request_completion self.j_mngr.log_events(f"Server STATUS error {e.status_code}: {e.body['message'] if e.body else ''}. File may be too large.", TypeError: string indices must be integer

[glibsonoran]

OK, thanks for the head's up, looking at it now.

[glibsonoran]

OK update your Plush for comfyUI and see if that fixed it. The fixed file is: style_prompt.py

[sleppyrobot]

Hey, thanks for the quick reply, the Open AI key message is gone, and both options now work with text only.

However, the other error about string indices must be integer occurs when passing an image

The error from the server side reads {"title" : Invalid 'content': 'content' objects must have a 'type' field that is either 'text' or 'image_url' Got 'object'"}

[glibsonoran]

Ok, I'm in the middle of something right now. I'll look at it in about 2 hrs.

[glibsonoran]

OK, I uploaded a new "api_requests.py" file that should fix the image input problem. Refresh your Plush suite and let me know if it works. Thx.

[sleppyrobot]

Hey, the bad news is that both LM_Studio and Local app options still trigger the "TypeError: string indices must be integers" error. I did not mention it before but am using LM Studio as the endpoint. The good news is that Oobabooga works with LM Studio endpoint, am not sure if it worked prior to these last few updates since I only recall using LM_Studio and Local app options.

Also the LM studio server side error has changed to
Error loading image. Error Data :n/a Additional Data: n/a

[glibsonoran]

Are you processing an image or image + text, or just text when you get the error?

[sleppyrobot]

Image + prompt + instruction triggers the error

[glibsonoran]

If you just provide instruction + prompt does it work with the "LM_studio" menu item?

[glibsonoran]

Also one other question. Did using an image with instruction and prompt work before in LM studio?

[glibsonoran]

Because it seems to be choking on the presentation of the image. I do that in standard OpenAI style and that hasn't changed since before this last update.

[sleppyrobot]

Yes everything worked before, even the examples worked to a degree but didnt use it much.

Doing Text only works right now, with local app, but not Lm studio option

[sleppyrobot]

Also, if i change the url from http://192.168.169.132:1234/v1 to http://192.168.169.132:1234/v1/chat/completions, text works now in lm studio but image still does not work.

[glibsonoran]

OK the "LM_Studio" menu item uses an http: POST, so you will always need to use /v1/chat/completions with this menu item. This is the way it should appear in the LM Studio local server screen when you start the local server up.

[glibsonoran]

What version of LM Studio are you using? And which vision model? Also it looks like you're using a LAN address rather than a local address.

[sleppyrobot]

The newest version 0.3.2 ive been using it stably for a while now, with the LAN address, since it on another PC.

Also i was wrong about Oobabooga working with image, it returns a response, but it is wrong.

[sleppyrobot]

I've used many different llama vision models with it, without issues.

https://huggingface.co/BAAI/Bunny-Llama-3-8B-V-gguf https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-gguf https://huggingface.co/FiditeNemini/Llama-3.1-Unhinged-Vision-8B-GGUF

[glibsonoran]

OK, yes Oobabooga has a pretty screwed up API and I can't find a way to pass it images. So it just tosses the image, but not many people use it for vision anyway... if any. So let me see if I can reproduce the problem using your models here.

[sleppyrobot]

I just reverted back to an old commit with git reset --hard 559f06bdfedbf2520d5622f85ed32ff0c93ebde8

The working config, i have been using is below http://192.168.169.132:1234/v1/chat/completions - lm_studio/local app

[glibsonoran]

Does it work with the old commit? Because I see the same error: Server was unable to process the request. Status: 400: {"error":" Error loading image. Error Data: n/a, Additional Data: n/a"},

But when I look at how it's being submitted it's exactly like the python example in LM Studio (other than minor line break differenes:

My completion message:

"messages": [ { "role": "system", "content": "You are an image evaluator" }, { "role": "user", "content": [ { "type": "text", "text": "Please create a caption for this image" }, { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,blah blah image data)" } } ] } ], "temperature": 0.7, "max_tokens": 1000

---

The example from LM Studio:

messages=[ { "role": "system", "content": "This is a chat between a user and an assistant. The assistant is helping the user to describe an image.", }, { "role": "user", "content": [ {"type": "text", "text": "What’s in this image?"}, { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,blah blah image data" }, }, ], } ], max_tokens=1000,

So I'm stumped. Anthropic and OpenAI can read my converted images just fine but not LM Studio.

[sleppyrobot]

Yes the old commit works

[glibsonoran]

what version of style_prompt? (It should be in your ComfyUI cmd output)

[glibsonoran]

Sorry not style_prompt... what version of Plush?

[sleppyrobot]

Am not sure what your referencing, I do not see anything in the terminal.

I used the command git reset --hard 559f06bdfedbf2520d5622f85ed32ff0c93ebde8 in the plush install folder to roll it back

oh this right?

[glibsonoran]

Would you do one more favor? Run that image through again (using the older version 1.21.13 of Plush) making sure your LM Studio log is set to verbose.

At the top of your LM Studio log should be the incoming message from Plush/ComfyUI, it will have your system and user prompts with the image included. If you could copy that whole message and put it in a comment here, I could see if there's something different about how it's being presented to LM Studio than the current version.

[glibsonoran]

OK, never mind, I'm pretty sure I found the issue, a space after a comma that was OK with chatGPT and Anthropic, but LM Studio didn't like it. :( Anyway the new version 1.21.14 should work with your LM Studio Vision images now.

[sleppyrobot]

oh I was just about to send it and my internet died.

Did a git pull it works again, thanks