The latest version of Disco supports to process multiple stages concurrently. Hustle could benefit from this new feature to speed up the query execution.
The tricky part is that in Hustle, every stage needs the input to be sorted, that means the stage is unable to run until all the output from previous stage is available and gets sorted. In some cases, sorting the input is unnecessary and wasteful, further, it prevents Disco from running the following stage concurrently as waiting the whole input to be available.
The following query is a typical one, which could be beneficial a lot from using Disco concurrent model,
The latest version of Disco supports to process multiple stages concurrently. Hustle could benefit from this new feature to speed up the query execution.
The tricky part is that in Hustle, every stage needs the input to be sorted, that means the stage is unable to run until all the output from previous stage is available and gets sorted. In some cases, sorting the input is unnecessary and wasteful, further, it prevents Disco from running the following stage concurrently as waiting the whole input to be available.
The following query is a typical one, which could be beneficial a lot from using Disco concurrent model,