search

elsa-workflows / elsa-core

A .NET workflows library
https://v3.elsaworkflows.io/
MIT License
6.25k stars 1.14k forks source link

Need node affinity setting #3330

Open johnwc opened 1 year ago

johnwc commented 1 year ago

When running several workers on different nodes, we need to be able to apply key/value properties on nodes that we can set a workflow's constraint to. We also need a way to put a constraint on workflows to make sure two or more with constraint value of X don't execute on same node.

Ex 1. Node 1-4 have much faster CPUs, so we set property on the nodes of SPEED=High. Workflow X that needs to always run only on one of those four nodes should have a node affinity rule placed on it where only a node with property SPEED is set to HIGH.

Ex 2. Workflow A, B, and C have activities that are disk or memory intensive and never should run at the same time on the same node.

If no node is available, the workflow can be configured to sit in queue for next available or to cancel/not run.

sfmskywalker commented 1 year ago

@jdevillard gave an awesome demo during yesterday's community meeting that I think will be able to handle this scenario perfectly. I think we can expect a PR soon ;)

johnwc commented 1 year ago

@sfmskywalker how do we get invited to these community meetings? I wouldn't mind listening in on some if I am available at the same time.

johnwc commented 1 year ago

@sfmskywalker it would also be nice to be able to see in the table in the dashboard, what node a workflow executed/excuting on.

sfmskywalker commented 1 year ago

how do we get invited to these community meetings? I wouldn't mind listening in on some if I am available at the same time.

Everyone is invited 🤗 The meeting occurs every Thursday at 18:00 UTC via our Elsa workflows server on Discord in the voice channel.

Next week, however, we are going to try Teams to see if it offers a better screen-sharing experience. Link will follow.

sfmskywalker commented 1 year ago

it would also be nice to be able to see in the table in the dashboard, what node a workflow executed/excuting on.

💯

johnwc commented 1 year ago

how do we get invited to these community meetings? I wouldn't mind listening in on some if I am available at the same time.

Everyone is invited 🤗 The meeting occurs every Thursday at 18:00 UTC via our Elsa workflows server on Discord in the voice channel.

Next week, however, we are going to try Teams to see if it offers a better screen-sharing experience. Link will follow.

I much prefer teams, since we use it here at the office already, over having to create a new account for yet another service. ;-) Will the teams link only be shared in discord?

jdevillard commented 1 year ago

This issue refers to discussion #2932 where I try to bring a start of reflexion!

Indeed the idea is to allow fine granularity to execute activity on specific node .

As you mention idea will be to get a list of node available and maybe possibility to set tag or anything else , then choose what node execute activity (and so entire workflow )