search

seqeralabs / nf-tower

Nextflow Tower system
https://tower.nf
Mozilla Public License 2.0
146 stars 52 forks source link

Gateway timeout (504) #98

Closed pditommaso closed 5 years ago

pditommaso commented 5 years ago

When running in the prod environment I'm getting gateway timeout error (504). I guess it's caused by the AWS load balancer and it may be related to #48.

Any suggestion?

image

pditommaso commented 5 years ago

@evanfloden I've increased the load balancer timeout to 90 secs.

Report here if it happens again.

evanfloden commented 5 years ago

I'm seeing this error still with long running pipelines.

My speculation: is it possible if there is no update from the workflow, ie no change in tasks in 90 sec, there is a gateway timeout?

Steps to reproduce:

nextflow run nf-core/mhcquant -r 1.3 -profile docker,test -with-tower

pditommaso commented 5 years ago

I think, there should be a heartbeat every sec to keep alive the connection, this is way I've set the timeout to 90 secs

tcrespog commented 5 years ago

How long it takes for the error to appear since the connection to trace/live/workflow starts? Is there no heartbeat in between? A heartbeat must be generated every 60 seconds.

tcrespog commented 5 years ago

I haven't been able to reproduce this issue. The hearbeats are being correctly generated and received every minute.

Maybe your last change in the load balancer solved the problem Paolo. Could you recheck it Paolo and Evan?

pditommaso commented 5 years ago

This is still happening. Now the load balancer timeout is set to 5min. In fact, the problem arises when between two tasks there a gap longer than 5 mins.

See the log below: 

08:47:42.551 [pool-1-thread-6] INFO  i.s.tower.controller.TraceController - Receiving workflow trace for workflows ID=null
08:47:42.909 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Creating flowable: workflow-aJ8AzrjX
08:47:42.910 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: user-1
08:47:42.910 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Data published for flowable: user-1
08:47:44.091 [pool-1-thread-9] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:47:44.158 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:47:44.159 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Data published for flowable: workflow-aJ8AzrjX
08:47:45.285 [pool-1-thread-9] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:47:45.340 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:47:45.340 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Data published for flowable: workflow-aJ8AzrjX
08:47:46.303 [pool-1-thread-9] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:47:46.348 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:47:46.348 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Data published for flowable: workflow-aJ8AzrjX
08:47:46.614 [nioEventLoopGroup-1-2] INFO  i.s.tower.controller.TraceController - Subscribing to live events of workflow: workflow-aJ8AzrjX
08:47:56.201 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:48:02.881 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 15 for user-1
08:48:21.123 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 9 for user-1
08:48:26.253 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:48:34.632 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 15 for user-1
08:48:56.333 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:49:02.881 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 16 for user-1
08:49:21.123 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 10 for user-1
08:49:26.407 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:49:34.632 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 16 for user-1
08:49:56.497 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:50:02.881 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 17 for user-1
08:50:05.249 [pool-1-thread-9] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:50:05.302 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:50:05.303 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Data published for flowable: workflow-aJ8AzrjX
08:50:06.181 [pool-1-thread-9] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:50:06.225 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:50:06.225 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Data published for flowable: workflow-aJ8AzrjX
08:50:10.286 [pool-1-thread-9] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:50:10.359 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:50:10.359 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Data published for flowable: workflow-aJ8AzrjX
08:50:13.166 [pool-1-thread-6] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:50:13.238 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:50:13.239 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Data published for flowable: workflow-aJ8AzrjX
08:50:14.222 [pool-1-thread-9] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:50:14.310 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:50:14.311 [pool-1-thread-9] INFO  i.s.t.s.ServerSentEventsServiceImpl - Data published for flowable: workflow-aJ8AzrjX
08:50:15.353 [pool-1-thread-6] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:50:15.422 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:50:15.422 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Data published for flowable: workflow-aJ8AzrjX
*** [last `Receiving task trace` before gap] ***
08:50:16.354 [pool-1-thread-6] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:50:16.414 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:50:16.415 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Data published for flowable: workflow-aJ8AzrjX
08:50:21.123 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 11 for user-1
08:50:26.542 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:50:34.632 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 17 for user-1
08:50:56.629 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:51:02.881 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 18 for user-1
08:51:21.123 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 12 for user-1
08:51:26.693 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:51:34.632 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 18 for user-1
08:51:56.772 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:52:02.881 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 19 for user-1
08:52:21.123 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 13 for user-1
08:52:26.823 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:52:34.632 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 19 for user-1
08:52:56.909 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:53:02.881 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 20 for user-1
08:53:21.123 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 14 for user-1
08:53:26.993 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:53:34.632 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 20 for user-1
08:53:57.061 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:54:02.881 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 21 for user-1
08:54:21.123 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 15 for user-1
08:54:27.138 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:54:34.632 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 21 for user-1
08:54:57.221 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:55:02.881 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 22 for user-1
08:55:16.415 [RxComputationThreadPool-1] INFO  i.s.t.s.ServerSentEventsServiceImpl - Idle timeout reached for flowable: workflow-aJ8AzrjX
*** `Completing flowable` message  
08:55:16.415 [RxComputationThreadPool-1] INFO  i.s.t.s.ServerSentEventsServiceImpl - Completing flowable: workflow-aJ8AzrjX
08:55:21.123 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 16 for user-1
08:55:27.314 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:55:32.320 [nioEventLoopGroup-1-2] INFO  i.s.tower.controller.TraceController - Subscribing to live events of workflow: workflow-aJ8AzrjX
*** first `No live events emitter` message *** 
08:55:32.320 [nioEventLoopGroup-1-2] INFO  i.s.tower.controller.TraceController - No live events emitter: workflow-aJ8AzrjX
08:55:34.632 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 22 for user-1
08:55:49.199 [nioEventLoopGroup-1-2] INFO  i.s.tower.controller.TraceController - Subscribing to live events of workflow: workflow-aJ8AzrjX
08:55:49.199 [nioEventLoopGroup-1-2] INFO  i.s.tower.controller.TraceController - No live events emitter: workflow-aJ8AzrjX
08:55:57.377 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:56:02.881 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 23 for user-1
08:56:10.354 [nioEventLoopGroup-1-2] INFO  i.s.tower.controller.TraceController - Subscribing to live events of workflow: workflow-aJ8AzrjX
08:56:10.354 [nioEventLoopGroup-1-2] INFO  i.s.tower.controller.TraceController - No live events emitter: workflow-aJ8AzrjX
08:56:16.869 [nioEventLoopGroup-1-2] INFO  i.s.tower.controller.TraceController - Subscribing to live events of user: user-1
08:56:17.080 [nioEventLoopGroup-1-3] INFO  i.s.tower.controller.TraceController - Subscribing to live events of workflow: workflow-aJ8AzrjX
08:56:17.080 [nioEventLoopGroup-1-3] INFO  i.s.tower.controller.TraceController - No live events emitter: workflow-aJ8AzrjX
08:56:21.123 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 17 for user-1
08:56:27.436 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:56:34.632 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 23 for user-1
08:56:57.513 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:57:02.881 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 24 for user-1
08:57:16.870 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 0 for user-1
08:57:21.123 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 18 for user-1
08:57:27.557 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:57:33.917 [nioEventLoopGroup-1-3] INFO  i.s.tower.controller.TraceController - Subscribing to live events of workflow: workflow-aJ8AzrjX
08:57:33.917 [nioEventLoopGroup-1-3] INFO  i.s.tower.controller.TraceController - No live events emitter: workflow-aJ8AzrjX
08:57:34.632 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 24 for user-1
08:57:41.616 [nioEventLoopGroup-1-3] INFO  i.s.tower.controller.TraceController - Subscribing to live events of workflow: workflow-aJ8AzrjX
08:57:41.617 [nioEventLoopGroup-1-3] INFO  i.s.tower.controller.TraceController - No live events emitter: workflow-aJ8AzrjX
08:57:57.629 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:58:02.881 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 25 for user-1
08:58:04.349 [pool-1-thread-6] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:58:04.399 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
*** first ERROR `No flowable found` message *** 
08:58:04.399 [pool-1-thread-6] ERROR i.s.tower.controller.TraceController - No flowable found while trying to publish task data: workflow-aJ8AzrjX
08:58:05.257 [pool-1-thread-6] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:58:05.314 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:58:05.318 [pool-1-thread-6] ERROR i.s.tower.controller.TraceController - No flowable found while trying to publish task data: workflow-aJ8AzrjX
08:58:08.333 [pool-1-thread-6] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:58:08.378 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:58:08.378 [pool-1-thread-6] ERROR i.s.tower.controller.TraceController - No flowable found while trying to publish task data: workflow-aJ8AzrjX
08:58:09.348 [pool-1-thread-6] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:58:09.407 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:58:09.407 [pool-1-thread-6] ERROR i.s.tower.controller.TraceController - No flowable found while trying to publish task data: workflow-aJ8AzrjX
08:58:11.735 [pool-1-thread-6] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:58:11.779 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:58:11.779 [pool-1-thread-6] ERROR i.s.tower.controller.TraceController - No flowable found while trying to publish task data: workflow-aJ8AzrjX
08:58:12.778 [pool-1-thread-11] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:58:12.855 [pool-1-thread-11] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:58:12.855 [pool-1-thread-11] ERROR i.s.tower.controller.TraceController - No flowable found while trying to publish task data: workflow-aJ8AzrjX
08:58:15.150 [pool-1-thread-6] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:58:15.207 [pool-1-thread-6] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:58:15.207 [pool-1-thread-6] ERROR i.s.tower.controller.TraceController - No flowable found while trying to publish task data: workflow-aJ8AzrjX
08:58:16.870 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 1 for user-1
08:58:21.123 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 19 for user-1
08:58:27.676 [Mail spooler] DEBUG io.seqera.mail.MailSpoolerImpl - Pending mail to send: 0
08:58:34.632 [RxComputationThreadPool-1] INFO  i.s.tower.controller.TraceController - Generating heartbeat 25 for user-1
08:58:38.769 [pool-1-thread-11] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:58:38.841 [pool-1-thread-11] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:58:38.842 [pool-1-thread-11] ERROR i.s.tower.controller.TraceController - No flowable found while trying to publish task data: workflow-aJ8AzrjX
08:58:39.665 [pool-1-thread-11] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:58:39.746 [pool-1-thread-11] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:58:39.746 [pool-1-thread-11] ERROR i.s.tower.controller.TraceController - No flowable found while trying to publish task data: workflow-aJ8AzrjX
08:58:43.292 [pool-1-thread-11] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:58:43.321 [pool-1-thread-11] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:58:43.322 [pool-1-thread-11] ERROR i.s.tower.controller.TraceController - No flowable found while trying to publish task data: workflow-aJ8AzrjX
08:58:44.371 [pool-1-thread-11] INFO  i.s.tower.controller.TraceController - Receiving task trace for workflow ID=aJ8AzrjX
08:58:44.440 [pool-1-thread-11] INFO  i.s.t.s.ServerSentEventsServiceImpl - Publishing event for flowable: workflow-aJ8AzrjX
08:58:44.440 [pool-1-thread-11] ERROR i.s.tower.controller.TraceController - No flowable found while trying to publish task data: workflow-aJ8AzrjX
pditommaso commented 5 years ago

Oops

https://github.com/seqeralabs/nf-tower/blob/552016070bca05f1a9e5caae2592c8b1085d8652/tower-backend/src/main/groovy/io/seqera/tower/controller/TraceController.groovy#L57-L58

tcrespog commented 5 years ago

Ok, now I see that the problem is in the workflow detail events.

That's right, the idle timeouts are set as a safety mechanism to cleanup the idle flowables. At the 5 minute mark without receiving progress events it's assumed that some problem happened so the workflow completed event wasn't received and the flowable is deleted. If a long time without receiving progress events is possible, we should set the timeout to a longer value and add a heartbeat emitter as in the workflow list emitter.

Nevertheless, I think the current event emission treatment should be refactored to adopt a new strategy:

This way there will be no need to keep an arbitrary timeout to cleanup resources.

pditommaso commented 5 years ago

I agree that the timeout cannot be used to delete the flowable because there can be any time between two different task executions.

Not sure about the counter. When do you decrement it?

Let's recap the problem, there are two different events:

I think we track for the same user only one workflow-flow that represents the current user selection.

This could be done using the /live/workflowDetail/{workflowId} request when the request is received the previous workflow-flow (for the same use) should be replaced by a new one.

Then remain the problem for the user-flow termination. because we don't know when the user close the browser/end the session. For this, I would move the hearth beat from the server to the client-side. The client should make a request every minute to the server, we use this to update a timeout for the user-flow, when the browser closes the browser the flow can be discarded.

The limit of this approach is that we cannot have the same user on two different browsers, for this we should use a more complex unique in place of the user-id, maybe a UUID, generated the very first time on the client and stored in a cookie and used for all the following requests .

Thoughts ?

tcrespog commented 5 years ago

Not sure about the counter. When do you decrement it?

Flowables have a doOnCancel method; I tested it and the callback is called when the user disconnects from the endpoint (e. g. closes the browser). We should wrap the flowables in a class that has the very own flowable and the number of subscriptions as its properties and store instances of this class in the flowables map instead of the plain flowables.

This could be done using the /live/workflowDetail/{workflowId} request when the request is received the previous workflow-flow (for the same use) should be replaced by a new one.

I think we should keep the flows separated. In fact, several workflows running at the same time might be possible. Is that right?

I would move the hearth beat from the server to the client-side. The client should make a request every minute to the server, we use this to update a timeout for the user-flow, when the browser closes the browser the flow can be discarded. [...] The limit of this approach is that we cannot have the same user on two different browsers, for this we should use a more complex unique in place of the user-id

I think this makes things quite complex 🤔. If we can avoid it managing the number of subscribers, the better.

pditommaso commented 5 years ago

Flowables have a doOnCancel method; I tested it and the callback is called when the user disconnects from the endpoint (e. g. closes the browser).

This is great, it solves most of the problems.

We should wrap the flowables in a class that has the very own flowable and the number of subscriptions as its properties and store instances of this class in the flowables map instead of the plain flowables.

Not understanding this. Can the doOnCancel method be used to remove itself from the map?

I think we should keep the flows separated. In fact, several workflows running at the same time might be possible. Is that right?

Yes, a user can run more than one, but only one can be selected in the browser at time. Then wh we should keep updating workflow-run not showed?

I think this makes things quite complex 🤔. If we can avoid it managing the number of subscribers, the better.

This is not needed anymore if there's a doOnCancel event.

tcrespog commented 5 years ago

Not understanding this. Can the doOnCancel method be used to remove itself from the map?

Yes I think so. The doOnCancel callback will be called when the subscription get cancelled, in this callback we should control if the number of subscriptions for the cancelled flowable goes to 0, if it does, we complete the flowable and also we remove the map entry from the map.

pditommaso commented 5 years ago

I don't agree on the flowable count to zero. We should have only one per run (and per browser session) representing the current view.

tcrespog commented 5 years ago

I think it's not necessary to have a different flowable for each browser session, I think we can achieve the behaviour perfectly with just one flowable. Why do you think differently? Could you elaborate?

pditommaso commented 5 years ago

Because, a user can open more than one browser sessions intentionally or by mistake. Just think you leave the browser open in your office and then connect from home. Therefore this event should be handled correctly.

pditommaso commented 5 years ago

Fixed by f4d8913.