search

h2oai / wave

Realtime Web Apps and Dashboards for Python and R
https://wave.h2o.ai
Apache License 2.0
3.9k stars 324 forks source link

Wave app being dropped when there's a request failure between Waved and the app #2043

Closed senalw closed 1 year ago

senalw commented 1 year ago

Wave SDK Version, OS

Wave SDK: 0.25.3 , OS: MacOs/ Ubuntu

Actual behavior

Wave daemon drops Wave apps when there is a http request failure between Waved and apps without even retrying it for a configurable time. This request failure could occur when there's a resource starvation for a small amount of time period. Therefore, dropping app is not acceptable here.

More info: https://h2oai.slack.com/archives/C0561218TBM/p1685689177638099

Expected behavior

Wave daemon should have a request-retry mechanism with user configurable time period. By default this retry-mechanism is turned off and we can enable it by setting env variable.

Steps To Reproduce

  1. Start the Wave app.
  2. Simulate sending RST signal for the connection between Waved and the app. iptables -A INPUT -p tcp --dport 8000 -j REJECT --reject-with tcp-reset
  3. Wave app will be dropped with connection reset by peer or connection refused error with the current behaviour.
mturoci commented 1 year ago

As discussed over Slack, this behavior is by-design. Can be sidestepped via setting H2O_WAVE_KEEP_APP_LIVE that prevents the app from being dropped in case of failure.

Keep in mind that the Wave team's suggestion is to make sure the app is running all the time, avoiding resource starvation by following proper deployment strategies.