Action Server does not handle output size

[kariharju]

Issue by mkorpela Sunday Jan 14, 2024 at 11:53 GMT Originally opened as https://github.com/robocorp/robocorp/issues/152

---

What happened?

• Current Status *

Current Action Server @action has limitless return value size. OpenAI GPT and Large Language Models in general work with limited size tokenized texts. OpenAI GPT allowed return value size is around 3500 ASCII characters. After that it will just error on Action use.

@action
def kaboom():
    return "A"*10000

This means that actions that return too big outputs will always fail from the callers (clients) point of view.

• How should it work *

I propose that Action Server internally truncates the outputs. The exact max size number could be configured in the Web UI.

Reference implementation:

def truncate_output_with_beginning_clue(output: str, max_chars: int = 2000) -> str:
    beginning_clue = "[Cut] "  # A very short clue at the beginning to indicate possible truncation

    if len(output) > max_chars:
        truncated_output = output[:max_chars - len(beginning_clue)]
        chars_missed = len(output) - len(truncated_output)
        truncated_message = f"[+{chars_missed}]"
        return beginning_clue + truncated_output + truncated_message
    else:
        return output

System Info

• 0.0.13 Python 3.11.5 OSX *

[kariharju]

Comment by tonnitommi Friday Mar 22, 2024 at 05:42 GMT

---

OpenAI GPT allowed return value size is around 3500 ASCII characters

Doesn't look like GPTs would restrict this anymore (at least there is not an error), but the GPT will not have access to the entire output if it's long. When asking a GPT what it can handle, the response is:

The maximum length of text I can effectively handle in a single interaction depends on the context of the request. For processing and analyzing text within a single response, I work best with segments up to around 4096 tokens (words and punctuation marks). This limit helps ensure accuracy in my responses and analyses.

Note: Action Server Toolkit for langchain has a truncation in the connector. Here.