[BUG] Error creating a delta table with the generated schema using "to SQL"

[waltersve]

How: List of performed steps

• Download Catena-X Model Manufactory Parts Quality Information
• Generate the SQL schema using (... to SQL)
• Use the generated schema to create a new delta table in Databricks using the SQL Editor

What: Actual result vs expected result

• Getting the following error: [DELTA_NESTED_NOT_NULL_CONSTRAINT] The element type of the field list_of_manufactured_parts contains a NOT NULL constraint. Delta does not support NOT NULL constraints nested within arrays or maps. To suppress this error and silently ignore the specified constraints, set spark.databricks.delta.constraints.allowUnenforcedNotNull.enabled = true.
• If ARRAY on the STRUCT will be removed the table will be created without error.

Where: (e.g. component, version, url, your system info)

• SQL generator

Input (files/screenshots) manufactirer-parts-quality-info.zip

Output (files/screenshots)

[chris-volk]

• Make NOT NULL statements configurable during SQL generation time (to cover cases where multiple model versions are related to one table)
• Attempt to keep NOT NULL (by default, but configurable, see above) where permitted.