Action Server does not handle timeouts

[kariharju]

Issue by mkorpela Sunday Jan 14, 2024 at 11:46 GMT Originally opened as https://github.com/robocorp/robocorp/issues/151

---

What happened?

Current Status

Current Action Server @action has limitless time to be executed. AI clients and internet services in general have timeouts for http requests. These range from 30 seconds up. OpenAI GPT seems to currently have Action timeout around 45 seconds.

This means that actions that take long enough will always fail from the callers (clients) point of view.

Based on experiments LLM clients such as OpenAI GPT easily can work with actions that give a separate callback url to check the results later on.

How should it work

I propose that Action Server internally switches to callback mode if action takes long enough. For example 20 seconds.

This would mean the following:

1. Action that takes 20 seconds or less => return result directly
2. Action that takes over 20 seconds => return callback url and status
   • calling callback url before action has finished => return callback url and status
   • calling callback url after action has finished => return action result

This method has been tested against OpenAI GPT and it works.

System Info

• 0.0.13 Python 3.11.5 OSX *

[kariharju]

Comment by fabioz Wednesday Jan 17, 2024 at 11:10 GMT

---

Seems reasonable.

Can you give a concrete example of what'd be the callback message? Would an LLM make sense of something as:

The action has not completed and in processing in the background. Please call "/get-async-result/<generated-uuid>" later to see if it completed. If it has not completed this same message will be repeated, otherwise the actual result will be provided.

And then /get-async-result/309uhsoenu]3stnp2-ue3 would be called at a later time and would return either the same message or the result.

[kariharju]

Comment by fabioz Wednesday Jan 17, 2024 at 11:12 GMT

---

Also, is it a problem if the return in this case is a string even if the original schema could've been marked to return an integer?

[kariharju]

Comment by mkorpela Wednesday Jan 17, 2024 at 17:48 GMT

---

Can you give a concrete example of what'd be the callback message?

Something in line with: "Execution ongoing; use callback 'ABC123' for results post-completion."

The OpenAI action spec does not allow variating url paths, so the id (ABC123 here) would have to be in the HTTP POST payload.

is it a problem if the return in this case is a string even if the original schema could've been marked to return an integer?

I would first separate @action decorated functions signature from the matching http endpoint signatures. So they can differ.

For the ongoing result a differing http return code (from 200) could be used to clearly separate that the process is still ongoing 202 (https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/202). So this is kind of "positive" exception, where the payload signature can be a bit different from the normal.

[kariharju]

Comment by fabioz Wednesday Jan 17, 2024 at 18:25 GMT

---

Ok, so, we'd have a /callback url path which would accept an id for the callback as a post. And then we'd return something with a 202 status asking to call that callback later.

Is that it?

Now, I'm not sure I follow:

I would first separate @action decorated functions signature from the matching http endpoint signatures. So they can differ.

What exactly do you mean by that?

As a note, internally the action-server manages everything, the @action is already executed in a different process and is managed by the action-server (so, in theory it's kind of decoupled already -- although there are definitely some internal reorganizations needed to return early while still keeping on executing the action).

[kariharju]

Comment by mkorpela Wednesday Jan 17, 2024 at 21:20 GMT

---

What exactly do you mean by that?

I think what you have written is what I was after. Nothing else.

[kariharju]

Comment by kariharju Thursday Jan 18, 2024 at 12:37 GMT

---

Is there anyway where the GPT side would provide the callback url? Would this be essentially make it so that the GPT side becomes a poller that needs to figure out all kinds of error cases. GPT side is developing fast, is there anything in the pipelines for this problem?

[kariharju]

Comment by tonnitommi Tuesday Jan 23, 2024 at 12:24 GMT

---

This realates also to async Actions discussed in #177. The AI app frameworks are developing fast, so this should be not a top priority just yet.

For now, we should have good design principles indicating that there are limits like request timeout and max response size that the developer needs to consider, and they vary per AI App framework.