shwstppr
commented
7 months ago Moving to 4.19.1.0 milestone for possible work on a generic solution for externalizing command timeouts
Open shwstppr opened 7 months ago
shwstppr
commented
7 months ago Moving to 4.19.1.0 milestone for possible work on a generic solution for externalizing command timeouts
GutoVeronezi
commented
7 months ago Hey @DaanHoogland @vishesh92
I did not have time to elaborate on the proposal until now, but here is the draft.
For some edge cases, we have "wait"/"timeout" hardcoded values or values that come from global settings to set the commands timeout, like for GetAutoScaleMetricsCommand and for MigrateCommand. Currently, we have 385 classes that extend com.cloud.agent.api.Command (I skipped classes that extend com.cloud.agent.api.Answer); therefore, externalizing global settings for each command to provide flexibility is not feasible; however, we can reach flexibility by other means.
Currently, if we do not set a wait to the command, AgentManagerImpl will fallback to the wait global setting:
I propose to create a table of command_timeout in which we would store the classpath of the command and its wait/timeout. Then, instead of fall backing directly to the wait global setting, we would check if there is an entry of the command in this table. If it exists, then we use the entry's value; otherwise, we fall back to the wait global setting. Thus, we can change the commands wait/timeout on demand and in a flexible way.
The first step of the proposal would be creating the table and changing AgentManagerImpl to check the table. We could already create some entries for hardcoded values, like for the GetAutoScaleMetricsCommand.
Then, as a second step, we could create APIs to allow operators to list, persist, and remove the wait/timeouts. We could also create an annotation to describe the commands and create an API to allow operators to list the annotated commands; thus, making it clear to them which processes they can manage the timeout.
As a third and final step, we could extend this feature to be managed via GUI.
I will create a POC of this proposal; meanwhile, let me know your thoughts about the proposal.
Note: this proposal cover the common cases; eventually we will discuss the edge cases.
Following there is the code I used to retrieve the classes that extend com.cloud.agent.api.Command:
import com.cloud.agent.api.Answer;
import com.cloud.agent.api.Command;
import org.reflections.Reflections;
import java.util.HashSet;
import java.util.Set;
import java.util.stream.Collectors;
class ClassMapper {
public static void main(String[] args) {
Set<Class<?>> classesToIterateThrough = new HashSet<>(new Reflections("org.apache", "com.cloud").getSubTypesOf(Command.class));
classesToIterateThrough = classesToIterateThrough.stream().filter(klazz -> !Answer.class.isAssignableFrom(klazz)).collect(Collectors.toSet());
System.out.println("Count: " + classesToIterateThrough.size());
for (Class<?> cmd: classesToIterateThrough) {
System.out.println(cmdClass);
}
}
}To execute it, run java -cp <path-to-cloud-client.jar> <path-to-file-with-the-above-code>.
Following there are the classes retrieved by the script:
GutoVeronezi
commented
7 months ago @DaanHoogland @vishesh92 I've created PR #8605 to address the POC I'd mentioned .
As discussed in https://github.com/apache/cloudstack/pull/8502#discussion_r1450410628 it could be useful to externalize the wait timeouts for hypervisor/agent commands like ModifyStoragePoolCommand. The downside of introducing new global configurations for each of these commands is having too many configurations so maybe a generic solution can be implemented.