search

databrickslabs / dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines
https://databrickslabs.github.io/dbldatagen
Other
357 stars 61 forks source link

Add support for Spark 3.1.2 #91

Closed ronanstokes-db closed 2 years ago

ronanstokes-db commented 2 years ago

Expected Behavior

Add explicit support for Spark 3.2 (included in Databricks runtime 9.1)

Current Behavior

The current versions of the framework work in Databricks 9.1 (which is based on Spark 3.2). However there are some new features in Spark 3.2 that will tidy up syntax for some date and time constructs.

alexott commented 2 years ago

Technically speaking, DBR 9.1 is still based 3.1.2 with some backports from 3.2

ronanstokes-db commented 2 years ago

Will use feature compatibility of DBR 9.1 LTS as base line going forward which is based on 3.1.2

ronanstokes-db commented 2 years ago

Current release was upgraded to Spark 3.1.3