atc-net / atc-dataplatform

A common set of python libraries for DataBricks
https://atc-net.github.io/repository/atc-dataplatform
MIT License
8 stars 3 forks source link

Feature/fix splitting #243

Closed mrmasterplan closed 1 year ago

mrmasterplan commented 1 year ago

While splitting into individual statements on semicolon is mandatory for spark, it may be undesirable for sql server, such as when a local variable is defined in one line and used in the next.

This PR introduces the ability to configure if, and how, the SqlExecutor should split a file into statements. The test tests.local.sql.test_SqlExecutor shows different ways that splitting may be configured.


The other commit in this PR fixes incorrect parsing of table properties.