Timeout in ReconnectingStreamHttp.outgoingConnection loves to fail

[hseeberger]

Caused by: java.lang.IllegalArgumentException: Task scheduled with [1073741823] seconds delay, which is too far in future, maximum delay is [21474835] seconds

[hseeberger]

Actually this seems to be an Akka issue: The LightArrayRevolverScheduler doesn't accept arbitrarily large Timeouts. Sigh!

conductrHaproxy Caused by: java.lang.IllegalArgumentException: Task scheduled with [1073741823] seconds delay, which is too far in future, maximum delay is [21474835] seconds
conductrHaproxy     at akka.actor.LightArrayRevolverScheduler.checkMaxDelay(Scheduler.scala:322) ~[akka-actor_2.11-2.3.9.jar:na]

[hseeberger]

@2m, @ktoso, please check if we should raise an issue against Akka.

[ktoso]

Hm interesting... as far as ReconnectingStreamHttp is concerned (current impl is timeout = retries * interval) we just need a "bigger timeout" than retry interval - strictly speaking it is "connection timeout". I initially just used timeout = reconnection interval but felt this may be too little - we can use a heuristic here, how about https://github.com/typesafehub/akka-contrib-extra/commit/2d5a960c7098c6897f8d2b8f27ecd2ce6a4dc0ad#diff-68724db5dd39db39802984d7660268caR68 simply being interval - sounds good for your use cases?

[ktoso]

I'd expect LightArrayRevolverScheduler having a good reason for this limit if it's explicit about it. Would be interesting to understand why such upper bound (except from "dont do big timeouts, could cause blocking")

// cc @patriknw @rkuhn

[hseeberger]

Ah, I see. My first impression was that maxRetries * interval is there for a particular purpose, like an overall timeout.

If that's not the case, I think we don't need to ask: Why not tell the Reconnector and let this guy forward to the sender()?

[rkuhn]

The data structure for managing the timeouts is heavily optimized, and the limit is 2^31 ticks (which at 10ms per tick yields the result you quote). Why would you ever need a timeout that is larger than 8 months? The original error message quoted above says that you wanted to schedule something more than 30 years into the future ;-)

[ktoso]

I'll fix both this and the http issue today, thanks for reporting Heiko!

[ktoso]

Resolving this by using the right value for connecting timeout:

implicit val timeout = Timeout(sys.settings.config.getDuration("akka.http.client.connecting-timeout", TimeUnit.MILLISECONDS).millis)