Check types of RunParameter to catch errors and give user feedback

microsoft / data-factory-testing-framework

A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric, Azure Data Factory and Azure Synapse Analytics.

MIT License

88 stars 20 forks source link

Check types of RunParameter to catch errors and give user feedback #83

Open LeonardHd opened 7 months ago

LeonardHd commented 7 months ago

ADF only supports specific data types for what we call RunParameter and PipelineRunVariable.

We support python primitives that map to the data factory types. For RunParameters this should be:

str -> String,
int -> Int,
float -> Float,
bool -> Bool
list -> Array
dict -> Object
str -> SecureString

For PipelineRunVariable these should be:

str -> String,
bool -> Boolean,
list-> Array,
int-> Integer,

Currently we do not validate any value on RunParameter and PipelineRunVariable.

yehiaelbehery commented 6 months ago

Hi @LeonardHd, I can work on this.

LeonardHd commented 6 months ago

@yehiaelbehery Thanks for offering help. I will share some more details as this might help. The example might be a bit too ambiguous, so I will collect some more details tomorrow 👍

LeonardHd commented 6 months ago

@arjendev I updated the issue. In fact, we do not validate anything at the moment in the framework. I think simple guard clauses would be enough so only valid RunParameter and PipelineRunVariable could be instantiated. What do you think?

@yehiaelbehery let's see what Arjen's point of view here is, as he leads the API concepts.

arjendev commented 6 months ago

Sounds good!

@LeonardHd, perhaps we can also introduce the check to verify we pass in the right type of the output argument in activity.set_result and the output argument in state.add_activity_result. These should only accept dictionary and lists iirc. This was the place where you accidently pushed in a tuple and got an internal error, right?