search

databrickslabs / dlt-meta

This is metadata driven DLT based framework for bronze/silver pipelines
Other
125 stars 54 forks source link

Getting ValueError while trying to launch DLT-META using interactive notebook #36

Closed mmicikevicius closed 4 months ago

mmicikevicius commented 4 months ago

Hello,

Here's a code: onboarding_params_map = { "database": "dlt_demo", "onboarding_file_path": "onboarding_files/onboarding.json", "bronze_dataflowspec_table": "bronze_dataflowspec_table", "bronze_dataflowspec_path": "dbfs:/FileStore/bronze",
"silver_dataflowspec_table": "silver_dataflowspec_table", "silver_dataflowspec_path": "dbfs:/FileStore/silver".format("silver"), "overwrite": "True", "onboard_layer": "bronze_silver", "env": "prod", "version": "v1", "import_author":"MANTAS" }

OnboardDataflowspec(spark, onboarding_params_map).onboard_dataflow_specs()

I'm getting "ValueError: missing attributes : set()" while all parameters are specified

ravi-databricks commented 4 months ago

@mmicikevicius, you do not need onboard_layer parameter while calling apis, since onboard_dataflow_specs() will onboard bronze and silver both. 

onboarding_params_map = {
"database": "dlt_demo",
"onboarding_file_path": "onboarding_files/onboarding.json",
"bronze_dataflowspec_table": "bronze_dataflowspec_table",
"bronze_dataflowspec_path": "dbfs:/FileStore/bronze",
"silver_dataflowspec_table": "silver_dataflowspec_table",
"silver_dataflowspec_path": "dbfs:/FileStore/silver".format("silver"),
"overwrite": "True",
"env": "prod",
"version": "v1",
"import_author":"MANTAS"
}
OnboardDataflowspec(spark, onboarding_params_map).onboard_dataflow_specs()

This should work!