microsoft / data-factory-testing-framework

A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric, Azure Data Factory and Azure Synapse Analytics.
MIT License
92 stars 21 forks source link

Pipeline variable default values are ignored #119

Closed rn-zeiss closed 3 months ago

rn-zeiss commented 3 months ago

During evaluate_pipeline all the run pipeline variables are created but defined default values are not set. Problem from my point of view is in _pipeline.py get_run_variables(), there it should be "defaultValue" not "default_value".

run_variables.append(PipelineRunVariable(variable_name, variable_value.get("defaultValue", None)))
rn-zeiss commented 3 months ago

@arjendev Here is a unit test to test the issue:

import pytest
from data_factory_testing_framework.models import Pipeline
from data_factory_testing_framework.state import RunParameter, RunParameterType

def test_when_variables_defaultvalues_are_correct_should_pass() -> None:
    # Arrange
    pipeline = Pipeline(
        pipeline_id="some-id",
        name="pipeline",
        activities=[],
        parameters=[]
    )

    pipeline.variables={
        "StringParam1": {
            "type": "String"
        },
        "StringParam2": {
            "type": "String",
            "defaultValue": "stringDefault"
        },
        "BoolParam1": {
            "type": "Boolean"
        },
        "BoolParam2": {
            "type": "Boolean",
            "defaultValue": True
        },
        "IntParam1": {
            "type": "Integer"
        },
        "IntParam2": {
            "type": "Integer",
            "defaultValue": 345
        },
        "ArrayParam1": {
            "type": "Array"
        },
        "ArrayParam2": {
            "type": "Array",
            "defaultValue": [
                "Val1",
                "Val2"
            ]
        }
    }

    # Act
    run_variables = pipeline.get_run_variables()

    # Assert
    assert len(run_variables) == 8
    assert run_variables[0].name == "StringParam1"    
    assert run_variables[0].value == None
    assert run_variables[1].name == "StringParam2"
    assert run_variables[1].value == "stringDefault"    
    assert run_variables[2].name == "BoolParam1"    
    assert run_variables[2].value == None
    assert run_variables[3].name == "BoolParam2"    
    assert run_variables[3].value == True
    assert run_variables[4].name == "IntParam1"    
    assert run_variables[4].value == None
    assert run_variables[5].name == "IntParam2"    
    assert run_variables[5].value == 345
    assert run_variables[6].name == "ArrayParam1"    
    assert run_variables[6].value == None
    assert run_variables[7].name == "ArrayParam2"    
    assert run_variables[7].value == ['Val1','Val2']
arjendev commented 3 months ago

Thanks for elaborate example, we will fix this soon.

@LeonardHd will be updating default_value to defaultValue and write some tests to ensure things keep working!

LeonardHd commented 3 months ago

Created a PR to fix #123.

To move self.variables to a property I suggest creating a new issue and implementing this separately (as it's technically a breaking change for the API, so let's try to go over all models and check what should be readonly).

arjendev commented 3 months ago

Created a PR to fix #123.

To move self.variables to a property I suggest creating a new issue and implementing this separately (as it's technically a breaking change for the API, so let's try to go over all models and check what should be readonly).

I already created the issue yesterday and, yes, let's fix it separately: https://github.com/microsoft/data-factory-testing-framework/issues/122