Run drush commands in parallel

[jigarius]

Requirements

• Say, there are 200 sites.
• The user wants to run 5 instances of drush, thereby running commands on 5 sites at a time.
  • User defines number of parallel processes, e.g. --drall-processes=5.
• Ensure intelligible output.
  • If we run a command like foo &; bar & the lines of output of foo and bar get mixed up.
  • We need to ensure that the output of each drush command appears together and is readable.

Solution

• If --drall-processes, say n, is greater than 1, then continue, otherwise, there's nothing special to do.
• Create a unique ID for the command, e.g. a UUID.
• Get a list of all the targeted sites.
• Create a JSON file in the tmp directory to track command progress.
  • Let's call this the task file.
• Launch n drall commands (let's call these workers) with a reference to the task file.
  • Let's call these Drall workers.
  • E.g. drall --drall-task=/path/to/task.json ... &;.
• Each Drall worker picks up a site,
  • Takes a task from the list (if it's not empty).
  • Does the task.
  • Obtains a lock on the list.
  • Updates the list.
  • Prints output.
  • Releases lock on the list.

Tasks

• [x] Decide task file specifications.
  • [x] Decide task file name and location.
  • [x] Decide task file structure.
• [x] Create an internal command exec:task.
  • [x] Takes a task path as an argument.
  • [x] Implement task file locking mechanism, i.e. only one process can write to the task file.
  • [x] Implement task execution mechanism, i.e. execute a task/command from the task file and mark it as done.
• Update the exec:drush command.
  • Support a --drall-processes=n parameter.
  • If n > 1, then generate a task file and launch Drall workers to execute the tasks.
• [x] Update tests
• [x] Update documentation

[jigarius]

Some more thoughts for making this commonly-requested feature a reality:

• When a command is being executed,
  • Collect the resulting output.
  • Ensure that no other Drall sub-instance is printing output at the moment.
  • Print the collected output.
  • Preferably, this should only be used when running in parallel mode 🤔
• When a command is issued in parallel mode,
  • Determine the number of parallel processes requested, say, p.
  • Determine the sites on which the commands have to be executed.
  • Break down the list of sites into p number of sets.
  • Issue one drall command for each of these sets in non-parallel mode.

Progress tracking

This can probably be dealt with in a separate issue where Drall shows better progress with progress bar(s). Or maybe provide reporting formats like progress, text, dots, etc.

• Option 1: Each process tracks its own progress.
• Option 2: The main drall command (that's running in parallel mode) tracks the progress.

For this issue, we just keep things simple and not think about tracking progress.

[jigarius]

Instead of running n drush commands with sub-lists of the sites, it seems that using a queue/workers approach will be better. I've updated the issue description to show the new proposed solution. Let's see how it goes.

[jigarius]

Here's a sample task file. This is generated by Drush for internal use, so the user doesn't need to worry about it.

Location: /tmp/drall/UUID.drallq.json.

{
  "version": "x.y.z",
  "status": {
    "total": 5,
    "pending": 3,
    "running": 1,
    "done": 1,
  },
  "task": {
    "command": "drush --uri=@@uri core:status --fields=site",
    "placeholder": "uri",
    "todo": {
      "ID": {
        "id": "ID",
        "status": "pending"
      }
    }
  }
}

The respective lock file would be located at: /tmp/drall/UUID.dralltask.lock. If the file exists, then a lock is in place.