microsoft / data-factory-testing-framework

A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric, Azure Data Factory and Azure Synapse Analytics.
MIT License
92 stars 21 forks source link

Expression evaluation of uninitialized String variable #120

Closed rn-zeiss closed 3 months ago

rn-zeiss commented 3 months ago

I have some pipeline String variables defined that don't have a defaultValue. When really executing the pipeline in Data Factory expressions using such uninitialized variables are I guess evaluated to empty string. Using data-factory-testing-framework the variables are set to 'None' and the evaluation of such expressions is failing with DataFactoryElementEvaluationError, see sample expression below.

Would it be possible to have same behavior in data-factory-testing-framework as in Data Factory?

"variables": {
            "temp_str": {
                "type": "String"
            }
        },        

@concat(variables('temp_str'),', [',item(),']',if(contains(item(),' '),concat(' as [',replace(replace(replace(item(),' ','_'),'(',''),')',''),']'),' '))
arjendev commented 3 months ago

Thanks for another bug-report @rn-zeiss, I will look into this later this week. If you can share a unit test (i.e. with arrange, act and assert portion) which fails with above expression, that would be of great help!

arjendev commented 3 months ago

Next Monday, we will be merging the improved expression language runtime based on the original dotnet runtime instead of our python implementation (#118) which will most likely solve your issues and bring the Testing Framework closer to the real Data Factory implementation.

arjendev commented 3 months ago

@rn-zeiss, we have deployed the updated runtime. Please have a look at version 1.0.0 and let us know whether that resolves your issues, thanks!

rn-zeiss commented 3 months ago

@arjendev, tried with version 1.0.1 and it works. Expressions referencing uninitialized string variables are evaluated as in real pipeline. Thanks :-) this issue can be closed.

arjendev commented 3 months ago

Awesome, thanks for letting us know, closing this issue then!