dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
400 stars 227 forks source link

Remove dead code 💀 #759

Closed Fokko closed 1 year ago

Fokko commented 1 year ago

Resolves #758 Less code is more!

Description

Checklist

mikealfare commented 1 year ago

Two sets of integration tests didn't run for some reason. I kicked them off again in CircleCI.

Fokko commented 1 year ago

@mikealfare Thanks for re-running. Looks like there are some issues with Github Actions right now.

mikealfare commented 1 year ago

@mikealfare Thanks for re-running. Looks like there are some issues with Github Actions right now.

'tis a weekday after all

Fokko commented 1 year ago

Hmm, doesn't look directly related to me:

Invoking dbt with ['run']
21:24:31  Running with dbt=1.6.0-a1
21:24:31  Found 2 models, 0 tests, 0 snapshots, 0 analyses, 356 macros, 0 operations, 1 seed file, 0 sources, 0 exposures, 0 metrics, 0 groups
21:24:31  
21:24:34  Concurrency: 1 threads (target='default')
21:24:34  
21:24:34  1 of 2 START sql incremental model test16837538613394426772_test_persist_docs.incremental_delta_model  [RUN]
21:24:38  1 of 2 ERROR creating sql incremental model test16837538613394426772_test_persist_docs.incremental_delta_model  [ERROR in 3.79s]
21:24:38  2 of 2 START sql table model test16837538613394426772_test_persist_docs.table_delta_model  [RUN]
21:24:43  2 of 2 OK created sql table model test16837538613394426772_test_persist_docs.table_delta_model  [OK in 4.44s]
21:24:43  
21:24:43  Finished running 1 incremental model, 1 table model in 0 hours 0 minutes and 12.01 seconds (12.01s).
21:24:43  
21:24:43  Completed with 1 error and 0 warnings:
21:24:43  
21:24:43  Runtime Error in model incremental_delta_model (models/incremental_delta_model.sql)
21:24:43    ('42000', '[42000] [Simba][Hardy] (80) Syntax or semantic analysis error thrown in server while executing query. Error message from server: org.apache.hive.service.cli.HiveSQLException: Error running query: [_LEGACY_ERROR_TEMP_DBR_0210] org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: at least one column must be specified for the table\n\tat org.apache.s (80) (SQLExecDirectW)')
21:24:43  
21:24:43  Done. PASS=1 WARN=0 ERROR=1 SKIP=0 TOTAL=2
Fokko commented 1 year ago

@mikealfare Happy CI :)

mikealfare commented 1 year ago

@nssalian Can you take a quick look at this? I don't see any issue with removing the code since all the CI checks are passing. But is there a chance that Databricks is using some of this code?

Fokko commented 1 year ago

Thanks for merging @mikealfare appreciate the quick follow up