Create Multiple TaskDef's on One `LHTaskWorker`

[coltmcnealy-lh]

Context

Right now, you need to create one LHTaskWorker object for each TaskDef that you want to execute. That LHTaskWorker gets its own:

• ThreadPool
• Set of connections to LH Server

This was designed with Enterprise use-cases in mind where there is a common TaskDef used by multiple WfSpec's, and the Task Worker is owned by a separate team from the WfSpec. In that case, it would be on its own "microservice" (i.e. K8s Deployment).

However, as we've built some applications using LittleHorse, not all TaskDef's fit this pattern. Some perform logic that is highly-coupled to the WfSpec, and as such is better to be deployed alongside the Gateway API for the WfSpec. One internal application we are working on will have dozens of such Task Worker's deployed on one machine. Having each of them create connections to the LH Cluster is wasteful.

Description

Currently, the LHTaskWorker constructor takes 3 args:

• the object with the Task Method
• The name of the Task Method (in case there are multiple @LHTaskMethod annotations)
• a Config

After this ticket, there should be another constructor with two arguments:

• Object with the task method(s)
• a config

It should prepare to execute all of them. It should also use only one connection pool and threadpool.

Technical Context

Right now the rpc PollTask has a TaskDefId task_def_id.

We will make that field repeated TaskDefId task_def_ids;. This change will be backwards-compatible (bless you, protobuf!).

[coltmcnealy-lh]

We want to be able to do this:

new LHTaskWorker(new DummyWorker(), config);

Instead of:

        DummyWorker executableWorker = new DummyWorker(this.client);

        List<String> tasks = List.of(
                "notify-org-deactivated",
                "send-welcome-email",
                "send-goodbye-email",
                "add-user-to-org-upon-creation",
                "welcome-user-to-org",
                "notify-user-removed-from-org",
                "post-event-adding-org-to-user",
                "post-event-removing-org-from-user",
                "notify-org-not-enough-seats",
                "send-invoice",
                "run-invoice-workflow",
                "notify-of-invitation",
                "notify-invitation-rejected");
        for (String task : tasks) {
            workers.add(new LHTaskWorker(executableWorker, task, lhConfig));
        }

   for (LHTaskWorker worker : workers) {
     worker.start()
   }

[coltmcnealy-lh]

Also needs to cover GO + Python

[coltmcnealy-lh]

Implementation: Task Worker Group Id will be the hash of the concatenated and sorted list of TaskDefid's.

[coltmcnealy-lh]

How will we ensure fair scheduling between multiple TaskDef's? I think the server should return the oldest TaskRun which has been waiting the longets.

[coltmcnealy-lh]

We could make our users' lives easier with just a pure client-side optimization. That would just be some syntactic sugar and overloaded constructors for the LHTaskWorker class in Java and the equivalent in Go/Python.

However, we could also go further and optimize it properly on the server side. That would mean that a PollTaskRequest would include multiple task def names:

repeated TaskDefId task_def_ids = ?;

instead of:

TaskDefId task_def_id = ?;

The "full optimization" would require significant refactoring on the server side. However it would reduce the number of Task Worker Heartbeats.

Currently, we send a heartbeat once every 5 seconds for a worker. So a hypothetical application that has 30 workers means we're sending 6 heartbeats per second, which is actually kind of expensive and somewhat silly. So doing the server-side optimization does indeed have value.

Perhaps we should just start with the client-side optimization for now. This would also simplify the healthcheck logic for users of the SDK.