search

tango-controls / pytango

This project was moved to gitlab.com
http://pytango.rtfd.io
54 stars 44 forks source link

[question] reconnection delay #375

Closed woutdenolf closed 4 years ago

woutdenolf commented 4 years ago

When I see the exception The connection request was delayed. Last connection request was done less than 1000 ms ago

  1. is my only option to sleep 1 second and try again? It doesn't seem like creating a new DeviceProxyhelps.
  2. Is the reconnection delay process wide, device specific or proxy specific? In the cpp code it seems to be related to a Connection but I don't know what that means.
  3. Is the delay client side or server side?
  4. Why does it exist in the first place? Can it be disabled?
ajoubertza commented 4 years ago

Here's some speculation from a non-cppTango developer:

1 and 2. Looking at cppTango code, DeviceProxy inherits from Connection. The timeout variables look like they are per Connection instance, so I would expect a new DeviceProxy to have its own timeout lifecycle. If the device still isn't up when a new DeviceProxy is created, it will likely experience a similar issue.

3. There is definitely a delay implemented on the client side. Not sure if there is a delay on the server side too. With some basic testing I can connect to new server less than 1 second after it has started (this is using nodb mode, and starting the client after the server). That suggests that it is only client side.

4. I'm not sure why it exists. My guess is to reduce excessive resource usage. If clients were to retry connections with a very short delay that could spin the client's CPU, maybe hit the DatabaseDS repeatedly for IOR lookups, and cause unnecessary network traffic. If a server has gone offline for a while, there is no need for clients to try and reconnect more than once per second. From the git blame on the cppTango code, this delay has been around for at least 10 years. It cannot be disabled.

If you can ensure that the devices are running before the clients that use them, then you would be able to avoid this problem.

woutdenolf commented 4 years ago

Hard to reproduce but it seems I can sometimes instantiate a DeviceProxy and get a The connection request was delayed exception when I ping immediately after. So this means DeviceProxy tries to connect on construction but does not raise an exception when this connection attempt failed. So DeviceProxy gets instantiated with its prev_failed_t0 time set.

PS: this situation occurs when the device is in the process of starting up.

bourtemb commented 4 years ago

So this means DeviceProxy tries to connect on construction but does not raise an exception when this connection attempt failed.

Indeed the DeviceProxy construction will not raise an exception if the connection attempt occurring during the construction failed. You could get an exception if the device is not defined in the Database though.

And you actually get an error when really trying to use the DeviceProxy object to execute a command, read/write an attribute or a pipe on this device since there was already a failed attempt to connect less than 1 second ago (when the DeviceProxy constructor was invoked).

woutdenolf commented 4 years ago

Ok that explains the behavior I'm observing. Thanks!