If a shuffle is needed, we currently inject a "merge-then-shard" sequence. This produces correct results, but also creates an artificial bottleneck. In theory, the leaves in the shards of the source domain should be able to directly send their output to the appropriate shards instead. Where this gets tricky is with ancestor queries. If an ancestor query has to ask multiple shards of the source domain, there is no longer a union in place that can buffer and merge the resulting backfills correctly.
If a shuffle is needed, we currently inject a "merge-then-shard" sequence. This produces correct results, but also creates an artificial bottleneck. In theory, the leaves in the shards of the source domain should be able to directly send their output to the appropriate shards instead. Where this gets tricky is with ancestor queries. If an ancestor query has to ask multiple shards of the source domain, there is no longer a union in place that can buffer and merge the resulting backfills correctly.