Azure / azure-iot-cli-extension

Azure IoT extension for Azure CLI
Other
83 stars 65 forks source link

Fixturize DPS tests #644

Closed vilit1 closed 1 year ago

vilit1 commented 1 year ago

Main goal of this is to remove the janky dps script in the pipeline.

Example run with no preset dps: image Note that only one hub is created (and deleted).

Example run with preset dps in pytest.ini: image will pass with preset dps image

Pipeline run: https://dev.azure.com/azureiotdevxp/aziotcli/_build/results?buildId=8013&view=results


This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Thank you for contributing to the IoT extension!

This checklist is used to make sure that common guidelines for a pull request are followed.

General Guidelines

Intent for Production

Basic expectations

Azure IoT CLI maintainers reserve the right to enforce any of the outlined expectations.

A PR is considered ready for review when all basic expectations have been met (or do not apply).

vilit1 commented 1 year ago

https://dev.azure.com/azureiotdevxp/aziotcli/_build/results?buildId=8083&view=results things work

Elsie4ever commented 1 year ago

The PR looks good to me, just got 2 questions mentioned above, thanks a lot!😀

Elsie4ever commented 1 year ago

I found this test_dps_device_registration_unlinked_hub test has been failed consistently in ubuntu 20.04 https://dev.azure.com/azureiotdevxp/aziotcli/_build/results?buildId=8226&view=logs&j=1e6b4794-1f9a-54f4-f473-41f4c9a18170&t=32f52c1e-fa8e-5ce8-5853-9481f10d33c0 https://dev.azure.com/azureiotdevxp/aziotcli/_build/results?buildId=8219&view=logs&j=1e6b4794-1f9a-54f4-f473-41f4c9a18170&t=32f52c1e-fa8e-5ce8-5853-9481f10d33c0

vilit1 commented 1 year ago

I found this test_dps_device_registration_unlinked_hub test has been failed consistently in ubuntu 20.04 https://dev.azure.com/azureiotdevxp/aziotcli/_build/results?buildId=8226&view=logs&j=1e6b4794-1f9a-54f4-f473-41f4c9a18170&t=32f52c1e-fa8e-5ce8-5853-9481f10d33c0 https://dev.azure.com/azureiotdevxp/aziotcli/_build/results?buildId=8219&view=logs&j=1e6b4794-1f9a-54f4-f473-41f4c9a18170&t=32f52c1e-fa8e-5ce8-5853-9481f10d33c0

The annoying part is that these errors are transient and from the runs here, happen if:

  1. the dps decides to not create an enrollment (says authorization failed when the permissions are clearly there)
  2. the device registration fails with authorization error (permissions are there) rather than there is no hub linked error.

I will investigate more on if I can make them less transient

vilit1 commented 1 year ago

https://dev.azure.com/azureiotdevxp/aziotcli/_build/results?buildId=8290&view=results tests no longer rerun constantly