Multiple partners have told us that they want to query Parquet files with DuckDB. Using different query engines that understand Parquet was one of the original design principles of our pipelines; so this is definitely doable. But it is probably worth if we also have an opinion about this that is backed by real query data. For example, having some side-by-side comparison with our example single-node Spark deployment option.
If we find this an appealing option, we should probably provide flat views in DuckDB dialect as well.
Multiple partners have told us that they want to query Parquet files with DuckDB. Using different query engines that understand Parquet was one of the original design principles of our pipelines; so this is definitely doable. But it is probably worth if we also have an opinion about this that is backed by real query data. For example, having some side-by-side comparison with our example single-node Spark deployment option.
If we find this an appealing option, we should probably provide flat views in DuckDB dialect as well.