Revisit Retry Edge Cases

[nthallen]

Changes to support optional Clients:

• Add final parameter to Socket::set_retries(). Return that value from Socket::not_reconnecting();
• To ensure backwards compatibility, I will set the client default retries to 0 with final = true.
• Need to review all the potentially terminating methods:
  • Client::app_negotiation_failed() Called due to:
    • Receiving something other than "OK\n" in response to handshake
    • Socket closed by remote during negotiation
  • Client::app_process_eof() Called due to:
    • Socket closed by remote after successful negotiation (implicit quit?)
  • Client::connected()
    • returns true on iwrite() failure, but iwrite() is returning iwrite_check() which on error returns iwrite_error() which Socket overrides to return reset(), so it will retry
  • Socket::connected() never returns true

Notes:

• Interface::iwrite_error() always returns true whereas Socket::iwrite_error() calls reset()
• General question is why support retries on Sockets and not on Interfaces in general?
  • The short answer is that it seemed more obvious that Sockets might be more flaky than local hardware, but USB devices can easily come and go. Should the retry mechanism be promoted to the Interface level?
    • In most cases, no such file or device is a fatal error, but it doesn't have to be
• Who counts on implicit quit? Is there any reason we should not require an explicit quit instruction in most cases?
  • Review all interconnect protocols, starting at the wiki
  • Should continue to have [app_]?process_eof() return true, but could override where not recognized and return reset() instead.
• We generally do not do retries on server sockets, but we know we can run into transient failures (TIME_WAIT)
  • Socket::read_error()
  • Socket::iwrite_error()
  • It looks as though this should just work. The question to address is how to structure servers so they don't mess up the Launch logic. That probably isn't actually an issue, since Launch uses unix domain sockets where this should not be a problem -- unless we have permission problems in /var/run/monarch
• I am dithering over the best default behavior, but these are all virtual methods for a reason. Different choices can be made in special cases.

[nthallen]

Best practice is to consider retry strategy whenever a class is derived from Socket.

[nthallen]

Changing behavior for server sockets:

• Defaults to no retries with final_response true
• Socket::read_error() and Socket::iwrite_error() will return reset() instead of true for server sockets

That should mean no immediate change, but on a case-by-case basis, we can enable retries with set_retries().

[nthallen]

playbacksrvr running as a session-specific local server without responsibilities needs option not to connect to tm_gen and/or it should not terminate if tm_gen is not present

[nthallen]

Tried making Clients default to set_retries(0,0,0,true). Encountered problem with ci_server() within playbacksrvr dying because tm_gen was not found. Updated cmdif_wr_clt to use set_retries(0,0,0,false), and playbacksrvr was OK

Encountered additional problem with subbuspp when trying to run Moudisrvr to test Moudialgo.

Then tried going back to change the Client default to set_retries(0,0,0,false) and ran into trouble with rdr/bfr when trying to quit out of playback. Both were running ready, may be counting on the implicit quit functionality. (debugging with SCoPEx 221110.2)