Using the new `copy parquet` capability of the tableauhyperapi

goodwillpunning / hyperleaup

Create and manipulate Tableau Hyper files from Apache Spark DataFrames and Spark SQL

Other

29 stars 19 forks source link

Using the new `copy parquet` capability of the tableauhyperapi #20

Closed jugi92 closed 3 years ago

jugi92 commented 3 years ago

Hi Will,

are you planning to implement the experimental copy from parquet capability from the new versions of tableauhyperapi? https://help.tableau.com/current/api/hyper_api/en-us/reference/sql/sql-copy.html#SQL-COPY-APACHE_PARQUET_FORMAT I think it should be more performant and even save the manual column type mapping that you currently implemented.

Even a "no" as an answer would be helpful so we can think about implementing it on our own. Best regards

goodwillpunning commented 3 years ago

Hi @jugi92, the latest commit adds support for converting Parquet-based tables to Tableau Hyper files. You can leverage this feature by specifying creation_mode=PARQUET when you instantiate a HyperFile object.