Unable to list triggers from Azure Data Factory

[antyrama]

Command Name az datafactory trigger list Extension Name: datafactory. Version: 0.2.0.

Errors:

Cannot deserialize datetime object., ValueError: Invalid datetime string: 2/1/2021 1:00:00 AM
Traceback (most recent call last):
Temp\pip-unpacked-wheel-yzud0ihl\msrest\serialization.py, ln 1914, in deserialize_iso
ValueError: Invalid datetime string: 2/1/2021 1:00:00 AM

...

Temp\pip-unpacked-wheel-yzud0ihl\msrest\exceptions.py, ln 51, in raise_with_traceback
Temp\pip-unpacked-wheel-yzud0ihl\msrest\serialization.py, ln 1914, in deserialize_iso
msrest.exceptions.DeserializationError: Cannot deserialize datetime object., ValueError: Invalid datetime string: 2/1/2021 1:00:00 AM

To Reproduce:

Steps to reproduce the behavior.

• Prepare an ADF instance with at least one trigger
• az datafactory trigger list --factory-name {} --resource-group {}

Expected Behavior

• Triggers will be listed

Environment Summary

Windows-10-10.0.18362-SP0
Python 3.6.8
Installer: MSI

azure-cli 2.11.1

Extensions:
datafactory 0.2.0

Additional Context

It occurs everywhere disregarding environment, originally observed on DevOps pipeline and taken locally for closer look.

[yonzhan]

datafactory

[antyrama]

After further investigation of my deployment pipeline, I discovered that ADF ARM templating allows you to provide a date time (startTime property of the tumbling window trigger) like: 2/1/2021 1:00:00 AM while az datafactory CLI extension requires it to be in ISO format.

[yungezz]

hi @qwordy could you pls help? thanks

[antyrama]

There is an issue but not in the place where we suspect. Let me explain how actually this datetime was given to the ARM template during deployment (lack of validation for startTime property for deploying tumbling window trigger is a different story).

Somehow, when I call for triggered runs by: az datafactory trigger-run query-by-factory --factory-name {} --resource-group {} --filters operand="TriggerName" operator="Equals" values="{}" --last-updated-after "{}" --last-updated-before "{}" --order-by order-by="WindowStartTime" order="DESC"

I'm getting this:

{
  "continuationToken": null,
  "value": [
    {
      "dependencyStatus": {},
      "groupId": "###################",
      "message": null,
      "properties": {
        "TriggerTime": "2/1/2021 12:04:19 PM",
        "windowEndTime": "1/11/2021 9:51:34 AM",
        "windowStartTime": "1/11/2021 8:51:34 AM"
      },
      "runDimension": null,
      "status": "Cancelled",
      "triggerName": "###################",
      "triggerRunId": "###################",
      "triggerRunTimestamp": "2021-02-01T12:04:19.023491+00:00",
      "triggerType": "TumblingWindowTrigger",
      "triggeredPipelines": {
        "###################": "###################"
      }
    },
.
.
.

Not sure why the properties: TriggerTime, windowEndTime, windowStartTime are returned as strings with such weird date time format while for e.g. triggerRunTimestamp is in desired ISO format.

So when my ADF ARM template is being deployed, I'm using properties.windowEndTime as a startTime for my trigger, this is why I was getting serialization exception at first place when listing trigger from my ADF instance.

IMHO the issue is in az datafactory trigger-run query-by-factory by returning timestamps in inconsistent formats.

[qwordy]

@qiaozha I check the commit history. This extension was written by you. A user reported a problem.

[qwordy]

@antyrama If I write this extension, I will only support unix timestamp. It is an integer. It can avoid format problem.

[yungezz]

feature related to codegenration tool, which is used to generate datafactory command code