igorborgest
commented
4 years ago Hi @Xiangyu-C, thanks for reaching out.
I've tried to replicate this issue here, but everything seems to be fine
import pandas as pd
import awswrangler as wr
df = pd.DataFrame({"c0": [1, 2], "c1": [1, 2], "c2": [1, 2]})
wr.s3.to_parquet(
df=df,
path="s3://BUCKET/PREFIX/",
dataset=True,
database="default",
table="test",
partition_cols=["c1", "c2"]
)
Could you try the snippet above in your environment? Also, could you send me a replicable snippet that exposes your issue?
Xiangyu-C
Describe the bug When I call wr.s3.to_parquet() with dataset=True, and if partition_cols are specified (two columns in the dataframe are used as partitions), the table gets created in glue and parquet file is saved. However, all the values in the table are null except for the two partition columns. Once I disable the partition_cols param, the data look good in glue (no null values).
To Reproduce Just use the same dataframe and try with or without partition_cols params in wr.s3.to_parquet call and check the table in glue to see if values are good or not. awswrangler version = 1.9.3 and through pip install