search

Adriankhl / godot-llm

LLM in Godot
MIT License
100 stars 3 forks source link

How to get it to exclude the prompt in the output #4

Closed spiderbyte87 closed 2 months ago

spiderbyte87 commented 2 months ago

I've been banging my head against the wall trying to pull only the generated text without the prompt. I can get a debug print with only the prompt by being tricky with strings, but that's messy.

Adriankhl commented 2 months ago

Do you have the patience to wait a few more hours to a day 😄 A new release is coming which adds an option to disable prompt output (and a lot more functionalities!) I just need to write up the documentation and do a bit of testing.

spiderbyte87 commented 2 months ago

Dang dude you're on it! That's freaking awesome!!

Adriankhl commented 2 months ago

The new version is out, waiting for the asset library to approve the release, or you may just get it from the github release page. Now, you can simply turn off the Should Output Prompt property (in GDScript should_output_prompt = false) to disable outputting the input prompt. You can also turn off Should Output Bos and Should Output Eos to disable those beginning and ending token.

Note that there is a small breaking change, I renamed generate_text to generate_text_simple, or you may simply add string argument there - generate_text(prompt, "", "").

spiderbyte87 commented 2 months ago

The new version is out, waiting for the asset library to approve the release, or you may just get it from the github release page. Now, you can simply turn off the Should Output Prompt property (in GDScript should_output_prompt = false) to disable outputting the input prompt. You can also turn off Should Output Bos and Should Output Eos to disable those beginning and ending token.

I literally just downloaded it and I'm testing inow. Will let you know if I run into any issues.

spiderbyte87 commented 2 months ago

Only issue was I was running a script to do multithreading manually, but it looks like you included proper threading in the extension for this version, causing it to freeze when trying to generate text while a thread was already running. Deleted my script and set it to run directly using the class functions and everything works great. Here's a clip of how I'm using your plugin: https://youtu.be/iW7Bom5qbPI

Thanks again for all your hard work! I'll be including (with your permission) your info in the credits when my game releases!!

Adriankhl commented 2 months ago

Only issue was I was running a script to do multithreading manually, but it looks like you included proper threading in the extension for this version, causing it to freeze when trying to generate text while a thread was already running. Deleted my script and set it to run directly using the class functions and everything works great. Here's a clip of how I'm using your plugin: https://youtu.be/iW7Bom5qbPI

Thanks again for all your hard work! I'll be including (with your permission) your info in the credits when my game releases!!

Very cool! Wish you have a successful release.

Adriankhl commented 2 months ago

Only issue was I was running a script to do multithreading manually, but it looks like you included proper threading in the extension for this version, causing it to freeze when trying to generate text while a thread was already running

Yes, threading is hard, especially with signals and big ram usage for LLM. Instead of using things like callback to handle parallel execution, I decided to keep it simple - only one execution per node, and it freezes easily to ensure that people notice the mistake when they call a function while the background thread is already running. You can always duplicate the node to execute things in parallel.