Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines
Allows specification of constraint on generated data using withConstraint method on DataGenerator class
Current Behavior
Constraints can be added using where clauses on generated data set. However use of a withConstraint class would isolate the specification of the constraint from its implementation allowing for better implementations in the future.
It would also simplify the management of constraints and make the constraint mechanism more similar to Delta Live Table expectations.
Steps to Reproduce (for bugs)
New feature - not a bug
Context
Your Environment
dbldatagen version used: to be implemented in v0.3.6 or later
Databricks Runtime version: to be supported in Databricks runtimes from 10.3 LTS onwards
Cloud environment used: Will be supported on released Databricks environments
Expected Behavior
Allows specification of constraint on generated data using
withConstraintmethod on DataGenerator classCurrent Behavior
Constraints can be added using
whereclauses on generated data set. However use of awithConstraintclass would isolate the specification of the constraint from its implementation allowing for better implementations in the future.It would also simplify the management of constraints and make the constraint mechanism more similar to Delta Live Table expectations.
Steps to Reproduce (for bugs)
New feature - not a bug
Context
Your Environment
dbldatagenversion used: to be implemented in v0.3.6 or laterDatabricks Runtime version: to be supported in Databricks runtimes from 10.3 LTS onwards
Cloud environment used: Will be supported on released Databricks environments