How to add multiple ComputeNodes of the same type in ComputeKernel?

[wugren]

Currently, whether tasks are supported is determined by comparing ComputeTaskType, and ComputeTaskType defines a broad category of tasks, such as adding a local llm node. How to do this? There is no obvious specification in the code implementation.

[streetycat]

maybe, we should add a param for the model-type, it should be match with the capacity of the node.

[glen0125]

I propose that for the same type of tasks, different ComputeNodes may have varying proficiency in answering certain types of questions (QuestionType, such as travel, science, life, etc.), with the same prompt. The scheduler should have a scoring system in place. If a ComputeNode's response to a certain type of question is accepted, then the weight of this ComputeNode being called for that type of question should increase. This would mean that the probability of this ComputeNode being asked to answer this type of question in the future would be higher.

[waterflier]

We consider the local LLM as a type of LLM node, which does not necessitate the expansion of TaskType. Instead, we introduce a new computation node.

We can deterministically use a class of LLM by specifying different mode-names (parameters specified each time LLM is called). If the upper layer does not specify the LLM model name, it implies that this LLM call can run on any LLM model.

[streetycat]

I have commit a Pull Request, we can define the rules in the is_support method for a ComputeNode to filter the tasks. And config a weight for the nodes.

When multiple nodes support one task, we select a random node to complete it in the schedule, and it's positive correlation between hit rate and weight.