search

cylc / cylc-uiserver

A Jupyter Server extension that serves the cylc-ui web application for monitoring and controlling Cylc workflows.
https://cylc.org
GNU General Public License v3.0
15 stars 18 forks source link

artificial workflow limit #569

Closed oliver-sanders closed 6 months ago

oliver-sanders commented 6 months ago

There's an artificial limit to the number of workflows that will update in the GUI based on the number of cores on the system.

Instructions to reproduce:

  1. Install and start a workflow.
  2. Navigate to it in the tree view.
  3. Leave this workflow running and goto (1).

Repeat the above instructions enough times and you will eventually end up with a workflow that does not update. The tree view will show the initial tasks in the workflow, but the workflow itself could be many cycles ahead of this state.

The number of repetitions of the above instructions required to replicate the bug is the number of virtual CUP cores on your system plus 5.

Cause

We are using a TheadPoolExecutor to manage workflow subscriptions in threads:

https://github.com/cylc/cylc-uiserver/blob/49fa3ef27186b7dcf86aef0aacfe09048f620d5a/cylc/uiserver/data_store_mgr.py#L90

https://github.com/cylc/cylc-uiserver/blob/49fa3ef27186b7dcf86aef0aacfe09048f620d5a/cylc/uiserver/data_store_mgr.py#L150-L156

But, we don't specify the maximum number of threads so we get the default which is the number of virtual cores on the machine plus 4:

https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor

So on a dual core machine (2 physical cores = 4 virtual cores) the maximum number of workflows that will update is 8! Ouch!

Solutions

  1. Configure a larger number.
    • What's the overheads of lots of threads?
    • Is this a problem?
  2. Stop using threads.
wxtim commented 6 months ago

Replicated on my ol' home laptop. (Ancient processor)

oliver-sanders commented 6 months ago

Closed by https://github.com/cylc/cylc-uiserver/pull/570