BB tests: forceful shutdown

[VakarisZ]

Is your feature request related to a problem? Please describe. BB test take up to a minute to kill monkeys:

07:18:59 [INFO] exploitation.log_success.59: SSH_exploiter_and_keys test passed, time taken: 204.0 seconds.
07:19:03 [INFO] monkey_island_client.kill_all_monkeys.52: Killing all monkeys after the test.
07:20:00 [INFO] exploitation.parse_logs.94: Parsing test logs:

07:32:03 [INFO] exploitation.log_success.59: SMB_exploiter_mimikatz test passed, time taken: 59.2 seconds.
07:32:07 [INFO] monkey_island_client.kill_all_monkeys.52: Killing all monkeys after the test.
07:34:26 [INFO] exploitation.parse_logs.94: Parsing test logs:
07:34:26 [INFO] monkey_logs_downloader.download_monkey_logs.15: Downloading each monkey log.

07:34:56 [INFO] exploitation.log_success.59: SMB_PTH test passed, time taken: 0.0 seconds.
07:35:00 [INFO] monkey_island_client.kill_all_monkeys.52: Killing all monkeys after the test.
07:36:11 [INFO] exploitation.parse_logs.94: Parsing test logs:
07:36:11 [INFO] monkey_logs_downloader.download_monkey_logs.15: Downloading each monkey log.

The maximum timeout for killing monkeys are 5min 40s (5 mins for monkeys dying and 40s for monkey process to die). Until the island has a proper way to kill monkeys, we can shut down the machines themselves.

Describe the solution you'd like Since BB tests always run in the same order, what we should do:

• On starting the test, send a signal to gcp to start machines of the next test.

• On finish of the test, kill all machines of the current test, except for the island.

  This solves 3 problems:

  • Performance. No longer need to wait for monkey processes to shut down.
  • Quota. BB tests become infinitely scalable.
  • Costs. No need to keep all of the machines running at the same time.

This would introduce the following issues/caveats:

• Keeping tests ordered in a way where test machines don't overlap
• Fetching logs before monkeys are done

Describe alternatives you've considered Run monkeys with a custom command line, similar to --setup-only that would skip all actions of the monkey

[mssalvatore]

Agents can now be reliably stopped. ATM, cost isn't a concern. Most of our machines are in a region with a very small CPU quota. Other regions have quotas that are 2 orders of magnitude larger. We've worked round the quota issues by 1) requesting quota increases and 2) Creating new machines in different regions.