dbx deployment missing support for DLT pipeline libraries others than notebook

[geiranton]

Expected Behavior

The docs og dbx says it support the same format as the API. According to /api/2.0/pipelines/ it is possible to create a DLT pipeline with whl type of library (among others)

Current Behavior

ValidationError: 1 validation error for Deployment workflows -> 25 -> Pipeline -> libraries -> 1 -> notebook

Only notbook library is supported. See https://github.com/databrickslabs/dbx/blob/main/dbx/models/workflow/common/pipeline.py

Steps to Reproduce (for bugs)

Context

Your Environment

• dbx version used: 0.8.17
• Databricks Runtime version:

[geiranton]

Something like:

diff --git a/dbx/models/workflow/common/pipeline.py b/dbx/models/workflow/common/pipeline.py
index 048498f..68cd607 100644
--- a/dbx/models/workflow/common/pipeline.py
+++ b/dbx/models/workflow/common/pipeline.py
@@ -4,6 +4,7 @@ from pydantic import validator

 from dbx.models.workflow.common.access_control import AccessControlMixin
 from dbx.models.workflow.common.flexible import FlexibleModel
+from dbx.models.workflow.common.libraries import Library, MavenLibrary
 from dbx.models.workflow.common.new_cluster import NewCluster
 from dbx.utils import dbx_echo

@@ -26,12 +27,16 @@ class PipelinesNewCluster(NewCluster):
             cls._omit_msg("spark_version")

-class NotebookLibrary(FlexibleModel):
+class SourceLibrary(FlexibleModel):
     path: str

 class PipelineLibrary(FlexibleModel):
-    notebook: NotebookLibrary
+    notebook: Optional[SourceLibrary]
+    jar: Optional[str]
+    maven: Optional[MavenLibrary]
+    whl: Optional[str]
+    file: Optional[SourceLibrary]

 class Pipeline(AccessControlMixin):

[aviadshimoni]

no support for DLT Afaik in dbx, they'll move support to bundle: https://www.youtube.com/watch?v=9HOgYVo-WTM

[renardeinside]

We have support for DLT in dbx, but we don't have support for source Library yet. I'll create a PR to resolve this one.

[renardeinside]

hi @geiranton

quick question on this - actually the PipelineLibrary is a flexible model, therefore it should accept any custom properties (as well as nested ones).

[renardeinside]

Okey, I found the issue - the model is flexible, but the property was required. I've made it optional, now it should pass other types too. Please upgrade to 0.8.18