[HELIX-817] Support for multiple instance group tags for Job config

[narendly]

Currently we can set only one instance group tag for a job.

jobCfg.setInstanceGroupTag("INSTANCEGROUPTAG");

It will be really helpful to support support multiple instance group tags for one job so that the job can be run either in group A or group B. Through this, we can support more advanced scheduling use cases like in Node Affinity in Kubernetes [1]

[1] https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

[jiajunwang]

What's the difference between using multiple instance group tags and defining a new tag that includes the desired scope? The second solution is supported right now and maybe easier to use.

Best Regards, Jiajun

On Mon, Jun 24, 2019 at 10:10 AM Hunter Lee notifications@github.com wrote:

Currently we can set only one instance group tag for a job.

jobCfg.setInstanceGroupTag("INSTANCEGROUPTAG");

It will be really helpful to support support multiple instance group tags for one job so that the job can be run either in group A or group B. Through this, we can support more advanced scheduling use cases like in Node Affinity in Kubernetes [1]

[1] https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/apache/helix/issues/323?email_source=notifications&email_token=AANYM2ATRXFDLXXUN6BLWV3P4D5YFA5CNFSM4H3ATMLKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G3LCLWA, or mute the thread https://github.com/notifications/unsubscribe-auth/AANYM2DJZK4IOD7BITHDEWDP4D5YFANCNFSM4H3ATMLA .

[DImuthuUpe]

@jiajunwang Good question. We can add tags to instances as much as we want. But the problem is, is it scalable? I'm not sure whether there is a straightforward way to update the instance config tag dynamically with out restarting participants. I usually set the tag by calling InstanceConfig.addTag(tag). In my use case, I need to dynamically set the target instance / group for a job based on some other external metrics. In that case it would be easy to manipulate the tags at JobConfig level rather than InstanceConfig level.

[jiajunwang]

@DImuthuUpe, by supporting both sides configuration, we definitely cover more use cases. I think it's very nice to have feature.

To clarify, the instance config update does not require restarting participants.

In addition, I want to better understand your needs. By "dynamically set the target instance / group", can you give us some examples of the external metrics that you might use?

[desaikomal]

Trying to clean up the bug backlog in open-helix. Is this feature still required? If not, can we please close this? @jiajunwang - what do you suggest?

[jiajunwang]

Can we evaluate if we can dynamically update the instance tag as Dimuthu mentioned in 2019? If this is doable (which I belive so), then I think we can close it.

Best Regards, Jiajun

On Sat, Apr 8, 2023 at 12:49 PM Komal Desai @.***> wrote:

Trying to clean up the bug backlog in open-helix. Is this feature still required? If not, can we please close this? @jiajunwang https://github.com/jiajunwang - what do you suggest?

— Reply to this email directly, view it on GitHub https://github.com/apache/helix/issues/323#issuecomment-1500965241, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANYM2EZXADN4RTJODBTBT3XAG6ODANCNFSM4H3ATMLA . You are receiving this because you were mentioned.Message ID: @.***>

[AlecHenx]

Did the issue be solved? I encounter the same issue with @DImuthuUpe . In my case, the relation between task and resource(or data, instance) is controlled by our logics. So we only use task framework, NO resource management framework.

The only way can we dispatch tasks to instances under our logic is instance tag. But we cannot set instance tag of a job in runtime.