Add timeout for handling of Command on agent's implementation

[vishesh92]

ISSUE TYPE

• Improvement Request

COMPONENT NAME

Agent, ServerResource

CLOUDSTACK VERSION

All versions

CONFIGURATION

N/A

OS / ENVIRONMENT

N/A

SUMMARY

The value for wait set in Command is not honored by agents (SSVM, host agent, etc.) while processing the request. These requests can also pile up if the same request is sent multiple times because of retries by user or the management server itself. This can potentially cause issues on SSVM and agents while executing long tasks leading to high wait times for commands to get executed on host agents, ssvm, etc.

STEPS TO REPRODUCE

1. Add a sleep in a ssvm command for 60 seconds and a log line before and after the sleep to check number of executions for that command.
2. Send 100 requests in parallel which will execute the above ssvm command with cmd.setWait(5).
3. Check logs on ssvm to check the actual number of requests received.

Note: The above example is for a Command for which executeInSequence is false.

EXPECTED RESULTS

If the request on management server times out in 5 seconds, the processing of command on SSVM should stop after 5 seconds as well.

ACTUAL RESULTS

The number of requests received on the SSVM would be executing these requests in batches of 10 requests or so. The requests would still get processed even if the server itself has timed out and no longer waiting for the answer. You will the log for both before sleep and after sleep.

[vishesh92]

cc: @DaanHoogland

[DaanHoogland]

@vishesh92 I think you are right in your expectations. There are some more cases to define/consider;

• will responses be sent of timed out but completed commands?
• will the time-out be from receiving of the command? I think you want the commands to not be executed if these have timeout before being on top of the queue, right?
• are time-out responses being sent?
• ?