Open shwstppr opened 7 months ago
Moving to 4.19.1.0 milestone for possible work on a generic solution for externalizing command timeouts
Hey @DaanHoogland @vishesh92
I did not have time to elaborate on the proposal until now, but here is the draft.
For some edge cases, we have "wait"/"timeout" hardcoded values or values that come from global settings to set the commands timeout, like for GetAutoScaleMetricsCommand
and for MigrateCommand
. Currently, we have 385 classes that extend com.cloud.agent.api.Command
(I skipped classes that extend com.cloud.agent.api.Answer
); therefore, externalizing global settings for each command to provide flexibility is not feasible; however, we can reach flexibility by other means.
Currently, if we do not set a wait
to the command, AgentManagerImpl
will fallback to the wait
global setting:
I propose to create a table of command_timeout
in which we would store the classpath of the command and its wait/timeout. Then, instead of fall backing directly to the wait
global setting, we would check if there is an entry of the command in this table. If it exists, then we use the entry's value; otherwise, we fall back to the wait
global setting. Thus, we can change the commands wait/timeout on demand and in a flexible way.
The first step of the proposal would be creating the table and changing AgentManagerImpl
to check the table. We could already create some entries for hardcoded values, like for the GetAutoScaleMetricsCommand
.
Then, as a second step, we could create APIs to allow operators to list, persist, and remove the wait/timeouts. We could also create an annotation to describe the commands and create an API to allow operators to list the annotated commands; thus, making it clear to them which processes they can manage the timeout.
As a third and final step, we could extend this feature to be managed via GUI.
I will create a POC of this proposal; meanwhile, let me know your thoughts about the proposal.
Note: this proposal cover the common cases; eventually we will discuss the edge cases.
Following there is the code I used to retrieve the classes that extend com.cloud.agent.api.Command
:
import com.cloud.agent.api.Answer;
import com.cloud.agent.api.Command;
import org.reflections.Reflections;
import java.util.HashSet;
import java.util.Set;
import java.util.stream.Collectors;
class ClassMapper {
public static void main(String[] args) {
Set<Class<?>> classesToIterateThrough = new HashSet<>(new Reflections("org.apache", "com.cloud").getSubTypesOf(Command.class));
classesToIterateThrough = classesToIterateThrough.stream().filter(klazz -> !Answer.class.isAssignableFrom(klazz)).collect(Collectors.toSet());
System.out.println("Count: " + classesToIterateThrough.size());
for (Class<?> cmd: classesToIterateThrough) {
System.out.println(cmdClass);
}
}
}
To execute it, run java -cp <path-to-cloud-client.jar> <path-to-file-with-the-above-code>
.
Following there are the classes retrieved by the script:
@DaanHoogland @vishesh92 I've created PR #8605 to address the POC I'd mentioned .
As discussed in https://github.com/apache/cloudstack/pull/8502#discussion_r1450410628 it could be useful to externalize the wait timeouts for hypervisor/agent commands like ModifyStoragePoolCommand. The downside of introducing new global configurations for each of these commands is having too many configurations so maybe a generic solution can be implemented.