Warning Autorest.Az is not maintained. Please use aaz-dev-tools for Azure CLI code generation.
1. Introduction
2. How does Azure CLI Code Generator Work
2.1. Different Generation Options
3. How to use Azure CLI Code Generator
3.1. Preparing Environment
3.2. Authoring Readme Files
3.3. Generate Azure CLI Code
3.4. Build the Generated Code
3.5. Execute the Generated Azure CLI Commands
a. Checks
b. Live Tests
4. Advanced Features
4.1. Folder Customization
4.2. CLI User Interface Customization
a. Add Parent Extension
b. Customize Extension Description
c. Set Extension/Command Groups/Commands/Parameters Mode
d. Set min-api/max-api in Command Groups/Commands/Parameters
e. Move Command Groups/Command Layer
f. Rename/Hide Command Groups/Commands/Parameters
g. Client Factory Customization
h. Parameter Specific Customization
i. flatten a parameter
ii. set a parameter as required
iii. set default value for a parameter
iv. add alias for a parameter
v. how an action parameter is handled
vi. set an action as positional argument
vii. set an action as AWS shorthand syntax
4.3. SDK Customization
a. Flattened SDK and un-Flattened SDK
b. Track1 SDK and Track2 SDK
4.4. Manual Override
4.5. Test Customization
4.6. Special Parameter Type
4.7. Incremental Code Generation
5. Contributing
6. Autorest Pipeline Configuration
Azure CLI Code Generator provides the feature of Azure CLI module generation in Azure CLI extensions repository and Azure CLI main repository. In this document, we will first introduce how Azure CLI Code Generator works including the basic usage as well as some advanced features. Then, we will describe how to debug the code generator. Finally, we will show the Autorest Pipeline definitions.
Azure CLI Code Generator is mainly for Azure CLI developers and people who are interested in generating Azure CLI by themselves.
Azure CLI Code Generator is an Autorest extension which generates functional Azure CLI code for Azure services by using Swagger specifications defined in azure-rest-api-specs or azure-rest-api-specs-pr repos. It uses Autorest V3 to handle the configuration interpretation, pipeline resolving, pipeline scheduling and uses Autorest.Modelerfour as a basic code model.
Besides the Autorest.Modelerfour, Autorest.Az has two more Autorest extensions dependencies, the Autorest.Clicommon and Autorest.Python, the Autorest.Clicommon is responsiblefor handling the user defined CLI directives such as operations splitting, polymorphism, renaming, hiding etc. and marking them properly in the code model. The Autorest.Python is integrated into Autorest.Az for the Rest APIs calls. Azure CLI does not do the Rest call directly. It either call the vendored SDKs from Azure CLI Extensions or call the public released SDK from Azure CLI main repo modules.
Both the Autorest.Clicommon and Autorest.Python take the code model generated by Autorest.Modelerfour as input. Normally Autorest.Python will not flatten the code model because in Python, complex objects are naturally supported. However,in Azure CLI, there is a requirement that object nesting level could not be deeper than two, otherwise it would be difficult to express the complex objects without using a bunch of delimiters. Therefore, in Autorest.Clicommon, the code model will be flattened and in Autorest.az, different code models from Autorest.Python and Autorest.Clicommon will be merged together.
After the code model has been updated and merged, AutoRest.az will render the code model into template and finally generate the Azure CLI code.
Autorest Options
--use=https://github.com/Azure/autorest.az/releases/download/1.7.3-b.20210818.1/autorest-az-1.7.3.tgz Autorest.Az Specific Options
exname is the extension name. After you start autorest.az with --az.debugger, you could then attach the VSCode to the autorest process for step by step debugging.--target-mode=core if you want to generate azure-cli repo command modules. It basically equals to --sdk-no-flatten --generate-sdk=no --compatible-level=track2.Other command line options can be found in the generate with different options doc
Azure CLI code generator is integrated into the pull request of Azure rest api specs repo and users could also try it locally. The guidance on how to generate Azure CLI code in rest api specs PR pipeline can be found in rest api specs documentation. In this section, we will focus on the guidance on how to generate Azure CLI code locally.
More detailed information can be found in the how to generate doc
python -m venv env
source ./env/bin/activate // or .\env\Scripts\Activate.ps1 in windows
pip install azure-cli // This is must to have if for simple try out
pip install azdev // this is optional if for simple try out. Install Autorest
Please run this command: npm install -g autorest@latest
Download Azure CLI and Azure rest API spec repositories (Optional):
Users need to prepare the swagger and download:
python setup.py sdist bdist_wheel in the {root}/azure-cli-extensions/src/{serviceName} folder. The azure-rest-api-specs and azure-rest-api-specs-pr repo are optional for Service Team whose swagger and readme files hasn't checked in yet.
Since the Autorest.Az depends on Autorest.Clicommon and Autorest.Python, readme files including readme.az.md, readme.cli.md and readme.python.md should be prepared before Azure CLI code generation.
Users can refer to this document for more details.
Sample command for authoring Azure CLI extensions:
autorest --version=3.0.6370 --az --use=@autorest/az@latest <path-to-the-swagger-readme.md> --sdk-no-flatten --azure-cli-extension-folder=<path-to-the-azure-cli-extension-repo>
Sample command for authoring Azure CLI main modules:
autorest --version=3.0.6370 --az --use=https://github.com/Azure/autorest.az/releases/download/1.7.3-b.20210818.1/autorest-az-1.7.3.tgz <path-to-the-swagger-readme.md> --compatible-level=track2 --azure-cli-folder=<path-to-the-azure-cli-repo>
See different combination of generation options for more useful scenarios.
It's always good to start with a command like that:
$ autorest --version=3.0.6370 --az --use=@autorest/az@latest ../azure-rest-api-specs/specification/logz/resource-manager/readme.md --azure-cli-extension-folder=./azure-cli-extensionsMake sure you have these specific versions installed:
| AutoRest Core | AutoRest CLI | Node.js |
|---|---|---|
| 3.0.6370 | 3.5.1 | 12.20 |
Errors you might encounter nowadays:
One more thing, it's a good practice to clean up AutoRest extensions by:
$ autorest --resetIf you want to do a simple try, please go to the az-output-folder that you specified in your readme.az.md, build the wheel file and add the generated file into Azure CLI.
cd <az-output-folder>
python setup.py sdist bdist_wheel
az extension add --source=<path-to-the-wheel-file>If it's for Azure CLI extensions development, you need to setup the azdev development environment and add the extension.
azdev setup -r ./<path of azure-cli-extensions repository> -c ./<path of azure-cli repository>
// You only need to specify the local path of the repository you plan to work on.
azdev extension add <extension-name> // for Azure CLI extensions
// The step of adding extension is required for developing Azure CLI main repo modules az <extension-name> -h to view all the commands and parameters.Here the <extension-name> is the main resource command group name.
You can also find a report.md in generated azext_{extensionName} folder, which contains an overview of all the generated command groups, commands and parameters.
You can run code style check and linter check if you would like to onboard this generated Azure CLI code and publish it to your customers.
azdev style <extension-name>
azdev linter <extension-name>You can also run the live test against the service backend server.
azdev test <extension-name> --live --discover After all steps completed successfully and the generated module is ready to release, you can commit the generated code and create a PR in azure-cli-extension repo or in azure-cli repo for review. If there's contain behaviour that's not good enough and you need to customize, you can refer to our Advanced Features
In this section, we will introduce the advanced features and how to leverage those features to generate Azure CLI code.
Autorest.az use directive for customization:
directive:
- where:
command: datafactory factory create
set:
command: datafactory createcli:
cli-directive:
- where:
group: Factories
param: factoryName
alias:
- name
- nThe supported usage for autorest directive is to update the command groups and commands layer e.g. remove subgroups or add subgroups.
Like SQL language, you can use where clause to specify groups/operations/parameters/schemas that need to be modified, and set clause or directive action clause to specify what kind of change to make. Unlike SQL that mainly operates on data, the directive operates on the code model. See details on cli directive doc
A typical readme.az.md configuration would look like this
az:
extensions: {extensionName}
namespace: azure.mgmt.{extensionName}
package-name: azure-mgmt-{extensionName}
$(azure-cli-extension-folder)/src/{extensionName}
python-sdk-output-folder: "$(az-output-folder)/azext_{extensionName}/vendored_sdks/{extensionName}"Where all the place holder for {extensionName} should be the same.
But users are allowed to specify different value for every place holder for different scenarios, for example, in the case of storage-preview extension.
az:
extensions: storage
namespace: azure.mgmt.storage
package-name: azure-mgmt-storage
az-output-folder: $(azure-cli-extension-folder)/src/storage-preview
python-sdk-output-folder: "$(az-output-folder)/azext_storage_preview/vendored_sdks/azure_mgmt_storage/v2019_06_01"we want the extension name to be storage but we want the code in src/storage-preview folder, and since storage-preview extension has both data plane sdks and mgmt plane sdks, and the sdks contains multiple api versions, so we should use different sdk path for storage.
For the resource provider of ApplicationInsights, it's actually a sub module of Monitor. which means we should design the Azure CLI user interface like az monitor app-insight instead of az app-insight. To solve this problem, we could update the directive to use the parent extension monitor for app-insight.
az:
extensions: app-insight
parent-extension: monitorCustomers can customize what can be show when run az -h after their extension or module has been on-boarded to Azure CLI. By default, we will simply show RP name info in it.
extension-description: RP description.In Azure CLI, we allow user to set different mode like is_preview or is_experimental for different kinds of layers including extension/command groups/commands/parameters. We can configure it in readme.az.md so the generated code can work in different mode.
see how to configure is_preview/is_experimental in different levels for more details.
In Azure CLI, we allow user to set the min or max api versions for command group, command and parameter. Below is a sample configuration in readme.az.md to generate code with min or max api versions.
cli:
cli-directive:
- where:
group: groupCondition
op: opCondition
param: paramCondition
min-api: 2019-01-01
max-api: 2020-12-01Before we talk about move command groups and command layer. we will first introduce what command group name and command name come from.
In Azure rest APIs, the operationId are usually with the format of A_B where A is resource name in the format of plural and B is the action name you want to perform on that resource for example create, update, get, start, stop, delete etc.
Azure CLI code generator would assume A as group name, B as the command name and the CLI command of operationId A_B would be like az <extension-name> A B.
In Azure CLI it's quite common that we want to move the same functional command into the same command group. For example:
directive:
## remove a sub group share
- where:
group: datashare share
set:
group: datashare
## add a sub group consumer
- where:
group: datashare trigger
set:
group: datashare consumer trigger
## change the group and name of a command
- where:
command: datafactory integration-runtime create-linked-integration-runtime
set:
command: datafactory integration-runtime linked-integration-runtime create
See how to add or remove subgroups for more details.
We provide the ability to rename or hide command groups, commands and parameters. For example:
cli:
cli-directive:
## rename a parameter
- where:
group: groupCondition
op: opCondition
param: paramCondition
name: new_op_name
## hide an operation
- where:
group: groupCondition
op: opCondition
hidden: true By default, the client factory should not expose the subscriptionId, baseUrl parameters to customer as well as authentication policy but we provide customization for these scenarios.
az:
extensions: bool
parent-extension: test-server
package-name: azure-mgmt-boolean
namespace: azure.mgmt.boolean
# client factory customization for subscription bound
client-subscription-bound: false
# client factory customization for base url bound
client-base-url-bound: false
# authentication policy customization
client-authentication-policy: SansIOHTTPPolicy
az-output-folder: $(azure-cli-extension-folder)/src/boolean
python-sdk-output-folder: "$(az-output-folder)/azext_boolean/vendored_sdks/boolean"The generated _client_factory.py would take effect like:
def cf_bool_cl(cli_ctx, *_):
from azure.cli.core.commands.client_factory import get_mgmt_service_client
from azext_boolean.vendored_sdks.boolean import AutoRestTestService
from azure.core.pipeline.policies import SansIOHTTPPolicy
return get_mgmt_service_client(cli_ctx,
AutoRestTestService,
subscription_bound=False,
base_url_bound=False,
authentication_policy=SansIOHTTPPolicy())There are some customization that we provide only applicable for parameter layer.
let's say we have a parameter A which type is an object and it has three properties a, b, c, and we want the a, b, c to be the command line parameters directly. In such case, we need to flatten the parameter. An example would be like:
cli:
cli-directive:
- where:
group: Triggers
op: CreateOrUpdate#Update
param: properties
cli-flatten: trueSee how to mark a parameter as required for more details.
According to Autorest.Modelerfour, the current required logic of a parameter is that parameter has to be required in all layers in its swagger definition.
Users can use the following directive to set a default value for a parameter
cli:
cli-directive:
- where:
group: Factories
parameter: identityType
default-value: SystemAssignedThis is useful when Azure CLI wants to do some special handling for parameters like SKU tier as it doesn't want customer to pass SKU tier directly. Use customization to provide a default value and hide it.
It's quite common that in Azure CLI a parameter can have one or more aliases.
cli:
cli-directive:
- where:
group: Factories
parameter: factoryName
alias:
- name
- nAn action parameter means a simple object that is not base class of polymorphic and satisfy one of the six conditions
By default we will use the key value format to handle the action. in the case of object A has three properties a, b, c and a, b, c are all simple type. if we use action to express it. it will be like --A a=a1 b=b1 c=c1 to express an object instance {a1, b1, c1}.
see how to set an action argument as postional for more details
see how to set an action argument as aws shorthand syntax for more details.
We also provide some SDK layer customization options. See how to generate with different options for more details.
The previous version of Autorest.Az code generator can only support the flattened SDK(before 1.6.0 release), and after we have supported the Azure CLI main repo modules(since 1.6.0 release). We are using flattened SDK by default for generating Azure CLI extensions and using un-flattened SDK by default for generating Azure CLI main repo modules.(current 1.6.1 release)
Current in our private releases we have changed the default generated SDK to un-flattened way as well. which should be public release very soon.
Users can use --sdk-no-flatten to specific an un-flattened SDK and --sdk-flatten to generate a flattened SDK. If users use both --sdk-no-flatten and --sdk-flatten we will still generate the un-flattened SDK.
In the current Azure CLI main repo, most of the modules are still using track1 publish released SDKs. However, in Azure CLI extension repos, the track2 SDK are used as vendored SDK for code generation.
This means, by default, we will use track1 mode for Azure CLI main repo modules generation and track2 mode for Azure CLI extensions generation.
Users can use compatible-level=track1 or compatible-level=track2 to specific which kind of SDK you want.
In some scenarios, we might find the generated code doesn't work for us and there's no way to use customization to meet our requirements. Though we are trying to reduce the manual override work, we can't rule out the possibility of generated code won't work in some complex scenaros.
Therefore, we provide the manual override ability for users to do manual override. See manual customization for more details.
By default the Autorest.Az can generate all the tests from examples in the order of CURD and it can resolve the resource dependencies within one RP, it also support users to define test scenarios by yourselves. See test configuration for more details
To be done ...
The basic idea of current incremental code generation is to hide those operations you don't need. see above sections to find out how to set command groups/commands as hidden.
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This configuration is for Autorest Pipeline definition.
See documentation here
python:
reason: 'make sure python flag exists to load config in python.md'
azure-arm: true
output-folder: $(az-output-folder)
debug-output-folder: $(az-output-folder)/_az_debug
use-extension:
"@autorest/python": "5.4.0"
"@autorest/clicommon": "0.6.2"
#"@autorest/python": "latest"
require:
- ./readme.python.md
- ./readme.cli.md
- $(this-folder)/readme.az.common.md
try-require:
- ./readme.test.md
pipeline-model: v3
scope-clicommon:
output-folder: $(debug-output-folder)
scope-az:
is-object: false
output-artifact:
#- source-file-az-hider
#- source-file-pynamer
#- source-file-aznamer
#- source-file-modifiers
#- source-file-merger
- source-file-extension
output-folder: $(az-output-folder)
cli:
reason: 'make sure cli flag exists to load config in cli.md'
naming:
default:
parameter: 'snake'
property: 'snake'
operation: 'snake'
operationGroup: 'pascal'
choice: 'pascal'
choiceValue: 'snake'
constant: 'snake'
type: 'pascal'
modelerfour:
lenient-model-deduplication: true
group-parameters: true
flatten-models: true
flatten-payloads: true
# keep-unused-flattened-models: true#payload-flattening-threshold: 4
#recursive-payload-flattening: true
pipeline:
python/m2r:
input: clicommon/identity
az/azentry:
input: python/namer
az/hider:
input: az/azentry
#output-artifact: source-file-az-hider
python/codegen:
input: az/hider
az/merger:
input: az/azentry
#output-artifact: source-file-merger
az/aznamer:
input: az/merger
#output-artifact: source-file-aznamer
az/modifiers:
input: az/aznamer
#output-artifact: source-file-modifiers
az/azgenerator:
input: az/modifiers
output-artifact: source-file-extension
az/emitter:
input:
#- az/hider
#- az/clicommon
#- az/merger
#- az/aznamer
#- az/modifiers
- az/azgenerator
scope: scope-az
az/azlinter:
input:
- az/emitter
#payload-flattening-threshold: 4
#recursive-payload-flattening: true
cli:
naming:
m4:
parameter: 'snake'
property: 'snake'
operation: 'snake'
operationGroup: 'snake'
choice: 'pascal'
choiceValue: 'pascal'
constant: 'snake'
type: 'snake'
pipeline:
python/m2r:
input: clicommon/cli-m4namer
az/azentry:
input: clicommon/identity
az/renamer:
input: az/azentry
az/merger:
input:
- az/renamer
- python/namer
#output-artifact: source-file-merger
az/aznamer:
input: az/merger
#output-artifact: source-file-aznamer
az/modifiers:
input: az/aznamer
#output-artifact: source-file-modifiers
az/azgenerator:
input: az/modifiers
output-artifact: source-file-extension
az/emitter:
input:
#- az/hider
#- az/clicommon
#- az/merger
#- az/aznamer
#- az/modifiers
- az/azgenerator
scope: scope-az
az/azlinter:
input:
- az/emitter