AnalysisException/DATATYPE_MISMATCH error when generating summary dataframe from sources that have a column named "summary"

databrickslabs / dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

https://databrickslabs.github.io/dbldatagen

Other

364 stars 61 forks source link

AnalysisException/DATATYPE_MISMATCH error when generating summary dataframe from sources that have a column named "summary" #264

Closed artruk closed 6 months ago

artruk commented 7 months ago

Expected Behavior

Current Behavior

Generating summary dataframe using DataAnalyzer seems to fail whenever the source being analyzed has a column named "summary"

Steps to Reproduce (for bugs)

import dbldatagen as dg

df = spark.range(10).withColumnRenamed("id", "summary") summary_df = dg.DataAnalyzer(sparkSession=spark, df = df).summarizeToDF()

Context

Your Environment

dbldatagen version used:
Databricks Runtime version:
Cloud environment used:

ronanstokes-db commented 6 months ago

we'll add a fix to this in the next hotfix.

In the meantime you can rename the "summary" field to something else - but avoid using leading underscores as these may conflict with internal column names

ronanstokes-db commented 6 months ago

Fixed in hotfix as of 05/22/24