search

microsoft / data-factory-testing-framework

A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric and Azure Data Factory.
MIT License
68 stars 15 forks source link

ExpressionParsingError for expressions containing formatDateTime #64

Closed rn-zeiss closed 6 months ago

rn-zeiss commented 6 months ago

I have a pipeline with some expressions containing formatDateTime function as shown below and always get an ExpressionParsingError exception:

"File_Name": { "value": "@concat(item().TABLESPACE_NAME, '__', item().TABLE_NAME,'.v', formatDateTime(utcnow(), 'yyyyMMdd_HHmmss'), '.parquet')", "type": "Expression" }

Looks just like a typo in the framework code, there I find formatDataTime instead of formatDateTime.

P.S.: Very nice framework you created here for ADF testing. Thanks :thumbsup:

arjendev commented 6 months ago

Hi @rn-zeiss,

Thanks for creating this issue and giving us feedback on the framework! It's indeed a typo on the function name in the function repository, I will create a PR immediately to address this.

Once the PR is complete, you will be facing an additional challenge around the current limitations around formatting datetimes due to missing formatting functionality. We have addressed this in one of the open PR's, which we will finalize in the coming week. It will require you to install the dotnet runtime, because the python framework will be invoking the dotnet datetime implementation just like Data Factory does. This will be explained accordingly in the README.md.

If you have any other feedback on the framework, feel free to share!

arjendev commented 6 months ago

I just finalized the PR that also fixes the anticipated formatting, just need a review from my colleagues. Should be tomorrow!

arjendev commented 6 months ago

It's approved and available from release version: 0.0.1a83.

Let us know if it works, dont forget to install the dotnet runtime.

It would be awesome if you could share us your use case on arjenkroezen@microsoft.com, cheers!

rn-zeiss commented 6 months ago

Thanks, with 0.0.1a83 formatDateTime function now is working fine.