ADT: Model creation scalability - eliminate size restriction on input models set

[avagraw]

The model creation command (az dt model create) currently only supports input ontologies consisting of 250 models or lesser (This limit is enforced by the underlying API). This code change allows users to pass in ontologies of size constrained only by max number of models which can be stored on the ADT instance (default is 10,000 models). The logic takes care of figuring out the dependencies and creating batches (smaller or equal to the API limit) . Each batch is then processed by a separate API call. The batching logic ensures that the dependency of each model being processed is in the same batch or has already been created on the DT instance as a part of a previous batch.

TQDM integration allows users to monitor progress.

---

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Thank you for contributing to the IoT extension!

This checklist is used to make sure that common guidelines for a pull request are followed.

General Guidelines

Intent for Production

• [ ] It is expected that pull requests made to default or core branches such as dev or main are of production grade. Corollary to this, any merged contributions to these branches may be deployed in a public release at any given time. By checking this box, you agree and commit to the expected production quality of code.

Basic expectations

• [ ] If introducing new functionality or modified behavior, are they backed by unit and/or integration tests?
• [ ] In the same context as above are command names and their parameter definitions accurate? Do help docs have sufficient content?
• [ ] Have all the relevant unit and integration tests pass? i.e. pytest <project root> -vv. Please provide evidence in the form of a screenshot showing a succesful run of tests locally OR a link to a test pipeline that has been run against the change-set.
• [ ] Have linter checks passed using the .pylintrc and .flake8 rules? Look at the CI scripts for example usage.
• [ ] Have extraneous print or debug statements, commented out code-blocks or code-statements (if any) been removed from the surface area of changes?
• [ ] Have you made an entry in HISTORY.rst which concisely explains your user-facing feature or change?

Azure IoT CLI maintainers reserve the right to enforce any of the outlined expectations.

A PR is considered ready for review when all basic expectations have been met (or do not apply).

[avagraw]

Pipeline testing evidence: https://dev.azure.com/azureiotdevxp/aziotcli/_build/results?buildId=5895&view=logs&j=e24f7e52-36f4-54e9-a8eb-6f10b62b4c84&t=85779272-bb12-5d7f-5c2e-f38ba3ce51f9

[avagraw]

Testing evidence with multiple ontologies: https://dev.azure.com/azureiotdevxp/aziotcli/_build/results?buildId=5914&view=logs&s=859b8d9a-8fd6-5a5c-6f5e-f84f1990894e

[avagraw]

More evidence - Tests process 3 ontologies: https://dev.azure.com/azureiotdevxp/aziotcli/_build/results?buildId=5922&view=logs&j=e24f7e52-36f4-54e9-a8eb-6f10b62b4c84&t=85779272-bb12-5d7f-5c2e-f38ba3ce51f9

[digimaun]

We should add a HISTORY.rst entry, and refresh the help to describe the improvements to this command. Users should be aware they can simply clone an ontology repository and upload all models in a single command -- and behavior if there is a failure during the process.

[digimaun]

It would be nice to improve this section in the contributing docs https://github.com/Azure/azure-iot-cli-extension/blob/dev/CONTRIBUTING.md#required-repositories to specifically to state clone commands including optionally fetching submodules to enable more tests.

[digimaun]

Has there been thought on including an enum based --failure-policy param that includes rollback or none options so the user can prevent the model deletion rollback behavior if desired?

[avagraw]

Has there been thought on including an enum based --failure-policy param that includes rollback or none options so the user can prevent the model deletion rollback behavior if desired?

Created a task for this: https://msazure.visualstudio.com/One/_sprints/taskboard/IoT-Platform-UPX/One/Custom/IoT/2207?workitem=14902067

[avagraw]

Latest testing evidence: https://dev.azure.com/azureiotdevxp/aziotcli/_build/results?buildId=6011&view=logs&j=e24f7e52-36f4-54e9-a8eb-6f10b62b4c84&t=85779272-bb12-5d7f-5c2e-f38ba3ce51f9