Adding a DISTINCT clause to the direct export query forces all data to be loaded into one worker, which prevents the results of the query from being broken up into several files. If it is important that non-distinct rows are exported, then we will need to use a standard BigQueryToCloudStorageOperator.
Adding a DISTINCT clause to the direct export query forces all data to be loaded into one worker, which prevents the results of the query from being broken up into several files. If it is important that non-distinct rows are exported, then we will need to use a standard BigQueryToCloudStorageOperator.