Increase the HTTP timeout

[skorokithakis]

I have a custom API for Willow, and it's working pretty well in general, but sometimes if OpenAI takes too long to respond (mostly because of multiple roundtrips with actions like adding appointments), Willow times out. Is there a way to increase the timeout? I'm fine with a 30 sec timeout when it has to do things like read me my calendar, as I know that will take a while.

[skorokithakis]

I just flashed 0.3.0-rc.2, but the timeout doesn't seem to have increased to 10 seconds (it seems to time out after 5 seconds for me). Also, what's the rationale behind such a short timeout? If I'm asking something, I'd probably prefer to wait for the response, rather than get an error, especially when the response does something somewhere (like add a calendar event).

Maybe 30 sec would be preferable? The best would be a configurable timeout via WAS, if possible.

[stintel]

I suspect you're confusing the timeout to WIS with the timeout to the HTTP REST endpoint. That would explain why you're seeing 5s: https://github.com/espressif/esp-idf/blob/v5.1.1/components/esp_http_client/esp_http_client.c#L145. Can you confirm this?

As for the rationale for keeping timeouts short: we feel that having to wait too long for a voice assistant respond is terrible UX. We therefore reduced the timeout on the HTTP stream to WIS to 2s initially (af99de418c1853cb5f25f2b19fb0a011bac946d9) but as the ESP HTTP Client only has a single timeout setting that also resulted in aborting the connection to WIS is inference took longer than 2s, so we increased that slightly (67ff0351b7413e80636fa4cb277bc952d41e2e9f) to allow users for testing self-hosted WIS on CPU before deciding to buy a GPU.

For the same reason of UX, we decided not to make this variable configurable at all, as it has the potential to kill the UX.

[skorokithakis]

Yes, sorry, I didn't realize that code wasn't for the API server. Is it possible to set the API server timeout to something long, 30 or 60 sec? Anything above that, I'd probably write something async and use the new (and fantastic) notifications feature.

I don't think the UX would suffer, especially if the request is abortable (ie I can say the wake word while waiting and it will cancel the old request and begin a new cycle).

[stintel]

Please try enabling WAS Command Endpoint mode in WAS Advanced settings. The WAS REST Command Endpoint should have a timeout of 30s (toverainc/willow-application-server@ff86dd97). WAS Command Endpoint mode will eventually become default, and endpoint support will eventually be removed from Willow, so it's unlikely we will make any new changes to the endpoint code in Willow.

[skorokithakis]

I can certainly do that, thank you. Is there any documentation for how to use the command endpoint? I haven't managed to find anything on the Willow site or Google.

[skorokithakis]

Ah, is the difference that the WAS connects to my REST API, rather than Willow directly? I'm going to have some Docker rearranging, if so. Out of curiosity, doesn't that increase the latency?

[stintel]

There is brief info about command endpoint support in WAS here.

Out of curiosity, doesn't that increase the latency?

We actually experience lower latency with WAS Command Endpoint mode. The WebSocket connection to WAS is always open, so there is no overhead to setup a new connection when you make a request. This now happens on the machine running WAS, which is almost always going to be more powerful than the ESP32-S3 in the Willow device.

[skorokithakis]

This worked great, thanks! There's only one bug: The screen turns off on the timeout, regardless of whether there's a request in flight. Instead, the screen should turn off timeout seconds after the end of activity.

[stintel]

Fixing that might be very tricky to get right. In any case, that belongs in a separate issue.

[skorokithakis]

Filed #349, thank you.