search

liquibase / liquibase-databricks

Apache License 2.0
26 stars 7 forks source link

DAT-18792: added cluster by support for snapshot and diff table related change types #199

Closed SvampX closed 4 weeks ago

SvampX commented 1 month ago

Tested following liquibase commands:

Changelog generates with present CLUSTER BY part with appropriate columns. Tested case with 1 and 2 clustering columns. Generated changeset example:

    <changeSet author="msav1 (generated)" id="1728984464968-4">
        <databricks:createTable clusterColumns="test_id,test_present_new" tableName="test_table_clustered_new">
            <column name="test_id" type="INT"/>
            <column name="test_present_new" type="INT"/>
            <databricks:extendedTableProperties tableLocation="s3://databricks-test-harness-metastore/c93bad90-fc3e-4083-a910-1cad48d283f2/tables/3b4b3312-cd20-41d1-9d74-451be370869f" tblProperties="delta.checkpoint.writeStatsAsJson=false,delta.checkpoint.writeStatsAsStruct=true,delta.checkpointPolicy=v2,delta.columnMapping.maxColumnId=3,delta.columnMapping.mode=name,delta.enableDeletionVectors=true,delta.enableRowTracking=true,delta.feature.allowColumnDefaults=supported,delta.feature.clustering=supported,delta.feature.columnMapping=supported,delta.feature.deletionVectors=supported,delta.feature.domainMetadata=supported,delta.feature.rowTracking=supported,delta.feature.v2Checkpoint=supported,delta.minReaderVersion=3,delta.minWriterVersion=7,delta.rowTracking.materializedRowCommitVersionColumnName=_row-commit-version-col-f039a2c1-7756-4597-9259-825d27ddce58,delta.rowTracking.materializedRowIdColumnName=_row-id-col-4af75f6c-2d74-4fcf-bc2b-8a2ddce45ef4"/>
        </databricks:createTable>
    </changeSet>
SvampX commented 1 month ago

As per described in extracting and cleaning methods possible issues we might try different way to read and store TBLPROPERTIES. Partially it was done on the view side here https://github.com/liquibase/liquibase-databricks/blob/main/src/main/java/liquibase/ext/databricks/snapshot/jvm/ViewSnapshotGeneratorDatabricks.java#L81-L89 I suggest storing in the https://github.com/liquibase/liquibase-databricks/blob/main/src/main/java/liquibase/ext/databricks/change/createTable/ExtendedTableProperties.java tblProperties parameter as a Map, not a raw String. This would make easier and fail proof cleaning up of system table properties and parsing needed stored there data.

suryaaki2 commented 1 month ago

Are we writing tests for this PR?

filipelautert commented 1 month ago

Are we writing tests for this PR? @suryaaki2 so far for Databricks we are only adding new test cases for test harness, as in createClusteredTableNew.* files added in this PR.

sonarcloud[bot] commented 4 weeks ago

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud