LucasYao93 commented 2 years ago

Tasks

[ ] Build autorest.csharp project and autorest.powershell project

LucasYao93 commented 2 years ago

Build autorest.csharp project and autorest.powershell project

Build autorest.csharp

Following are details about how to build autorest.csharp with Visual Studio.

git clone --recurse-submodules https://github.com/Azure/autorest.csharp.git -b master
Open the solution and remove the project autorest.csharp.test
Comment out new AutoRest.Simplify.CSharpSimplifier().Run(fs).ConfigureAwait(false).GetAwaiter().GetResult(); from the Program.cs to disable simplifier plugin
Build the solution

If you run into any build errors, you probably need to install dotnet core 2.1 in Visual Studio

How to generate SDK with different generators

In this directory, run AutoRest.

To generate SDK with autorest.csharp

autorest --use:<path-to-autorest.csharp> --tag=basic-get.csharp

Autorest.PowerShell introduction

Development environment setup

Install autorest and dependencies

Install NVS to manage different versions of NodeJS

Install NodeJS 14.x with NVS

# Install NodeJS 14.20.1
nvs add 14.20.1
# Switch to version 14.20.1
nvs use 14.20.1

Install dotnet SDK 2
```
npm install -g dotnet-sdk-2.1
```
Install autorest
```
npm install -g autorest@latest
```
Install rush
```
npm install -g @microsoft/rush@5.63.1
```

IDE

visual studio code

Code repo and branch strategy

Code repo is located in https://github.com/Azure/autorest.powershell.

And there are two major branches

main
- For stable version
develop
- For dev version

Build, manage, use and debug autorest.powershell

How to build

# Install dependencies
rush update
# Clean build
rush rebuild

Other useful rush commands

Following are some useful commands of rush. Please refer here for more details about rush.

# detect issues that do not follow the lint rule
rush lint
# autofix the issues that do not follow the lint rule
rush fix
# Caculate the next release version
rush set-versions
# run all npm test
rush test
# auto packaging
rush watch
# For more cmds, you may run following cmd
rush -h

*Note: except for embedded commands of rush, we have some custom ones defined in common/config/rush/command-line.json.*

How to use local built autorest.powershell

autorest --use:C:\Users\xxx\source\autorest.powershell

Generate pipeline graph and artifacts for each step

autorest --use:C:\Users\xxx\source\autorest.powershell --interactive

How to debug

autorest --use:C:\Users\xxx\source\autorest.powershell --powershell.debugger

ModelerFour introduction

Schemas
globalParameters
operationGroups
commands

Plugins

tweakm4codemodel

addResponseHeaderSchema
addDictionaryApiVersion
removeM4DefaultDescription
handleNoinlineDirective

tweakcodemodel-v2

adds in a universal identity type
dropDuplicatePropertiesInChildSchemas
Identify models that are polymorphic in nature
find responses that have header values, and add a property to the schemas that are returned with those values
if we have an operation with a body, and content-type is a multipart/formdata then we should go thru the parameters of the body and look for a string/binary parameters and remember to add another parameter for the filename of the string/binary
identify parameters that are constants
identify properties that are constants

tweakcodemodel-azure

identifies operations that support paging
identifies operations that support long-running-operations
Api Version parameter handling for Azure

create-commands-v2

create cmdlet

create-virtual-properties-v2

create virtual properties
create virtual parameters

csharp-namer-v2

adds c# style names to the details for everything in the code model

powershell-namer-v2

adds c# style names for things powershell needs to the details for things in the code model
handle name conflict

add-azure-completer-v2

not used yet

powershell-v2

support/xxx.Competer.cs, support/xxx.TypeConverter.cs
.PowerShell.cs *.TypeConverter.cs in api/Models
Modules.cs
Cmdlets
copy others

llcsharp-v2

All generated code are in generated/api

xxx.cs, xxx.json.cs in api/Models
Rest Client
Support/xxx.cs
copy runtime/csharp/{client,json,runtime}

code generation

project→namespace→class, interface→property, field, method→local varible, statement like if else.

Dependencies

azure-tools/codegen
- escapeString, docComment, serialize, pascalCase, DeepPartial, camelCase, TextWithRegions, TextPossibilities, Text, EOL, applyOverrides, copyResources, copyBinaryResources, EnglishPluralizationService,intersect
azure-tools/codegen-csharp
- all C# related staff, e.g. namespace, class, method, property, interface
azure-tools/autorest-extension-base
- extension base, add(), run()
azure-tools/codemodel
azure-tools/codemodel-v3
- KnownMediaType, consider to replace it with codemodel

Build

Common steps

Creating exports and docs

Export-ProxyCmdlet -ModuleName $moduleName -ModulePath $modulePaths -ExportsFolder $exportsFolder -InternalFolder $internalFolder -ModuleDescription $moduleDescription -DocsFolder $docsFolder -ExamplesFolder $examplesFolder -ModuleGuid $guid

Creating format.ps1xml

Export-FormatPs1xml -FilePath $formatPs1xml

Creating psd1

Export-Psd1 -ExportsFolder $exportsFolder -CustomFolder $customFolder -Psd1Path $psd1 -ModuleGuid $guid

Creating test stubs

Export-TestStub -ModuleName $moduleName -ExportsFolder $exportsFolder -OutputFolder $testFolder

Creating example stubs

Export-ExampleStub -ExportsFolder $exportsFolder -OutputFolder $examplesFolder

Export cmdlet surface

Export-CmdletSurface -ModuleName $moduleName -CmdletFolder $exportsFolder -OutputFolder $resourcesFolder -IncludeGeneralParameters $IncludeGeneralParameters.IsPresent -UseExpandedFormat $UseExpandedFormat.IsPresent

Export model surface

Export-ModelSurface -OutputFolder $resourcesFolder -UseExpandedFormat $UseExpandedFormat.IsPresent

Generate markdown help

Export-HelpMarkdown -ModuleInfo $moduleInfo -FunctionInfo $cmdletFunctionInfo -HelpInfo $cmdletHelpInfo -DocsFolder $docsPath -ExamplesFolder $examplesPath

Other cmdlets

GetCommonParameter -- not used

GetScriptCmdlets -- an internal cmdlets used by other cmdlets

Cmdlets details

GetModuleGuid

Get guid from the psd1 file, if not existed, create a new one.

ExportCmdletSurface

Export the surface (signature) of the cmdlet

GetScriptCmdlets

List all the cmdlets in a script folder (e.g. exports)

ExportExampleStub

Generate two empty example stubs with name, code, description with the exports folder as input, if example files are there, do not do it.

ExportFormatPs1xml

Generate format file by using exported class (in model namespace and support namespace) in the assembly

ExportHelpMarkdown

There is a bug. For no profiles, "exports" should not appear in the docs path.

ExportModelSurface

Export models by using exported class (in model namespace and support namespace) in the assembly

ExportProxyCmdlet

Generate proxy (may include docs, which is controlled by -ExcludeDocs) from the dll and custom script.

ExportPsd1

Generate psd1 file, GUID will be generated if there is no GUID in a existed psd1 file.

ExportTestStub

Generate test files per cmdlet from the exports folder

GetCommonParameter

Get the common parameters (Azure, Runtime, Confirm, WhatIf, First, Skip, IncludeTotalCount)

LucasYao93 commented 2 years ago

For my local environment

autorest --use:..\..\..\..\autorest.csharp --tag=basic-get.csharp

autorest --use:..\..\..\..\autorest.powershell --tag=basic-get.powershell

LucasYao93 commented 2 years ago

Test Aks

csharp

LucasYao93 commented 1 year ago

Test basic-spec-required.powershell ❌

autorest --use:..\..\..\..\autorest.powershell --tag=basic-spec-required.powershell
AutoRest code generation utility [cli version: 3.6.2; node: v14.20.1]
(C) 2018 Microsoft Corporation.
https://aka.ms/autorest
info    |    Loading AutoRest core      'C:\Users\v-diya\.autorest\@autorestcore@3.9.3\nodemodules\@autorest\core\dist' (3.9.3)
info    | 
The default version of @autorest/powershell has been bumped from 2.1+ to 3.0+.
 > If you still want to use 2.1+ version, please specify it with --use:@autorest/powershell@2.1.{x}, e.g 2.1.401.

info    |    Loading local AutoRest extension '@autorest/powershell' (C:\Users\v-diya\repository\AutorestSdkSupport\autorest.powershell)
info    |    Loading AutoRest extension '@autorest/modelerfour' (4.23.1->4.23.1)
warning | PreCheck/CheckDuplicateSchemas | Checking for duplicate schemas, this could take a (long) while.  Run with --verbose for more detail.
warning | UsingTemporaryFlag | modelerfour.treat-type-object-as-anything options is a temporary flag. It WILL be removed in the future.
C:\Users\v-diya\repository\AutorestSdkSupport\autorest.powershell\powershell\dist\plugins\sdk-generate.js - FAILURE  {"path":"C:\\Users\\v-diya\\repository\\AutorestSdkSupport\\autorest.powershell\\powershell\\resources\\templates\\azureServiceClient.ejs"}
fatal   | TypeError: C:\Users\v-diya\repository\AutorestSdkSupport\autorest.powershell\powershell\resources\templates\azureServiceClient.ejs:20
    18|     {
    19| <%# ToDo: add ServiceClientBodyTemplate -%>
 >> 20| <%- include('serviceClientBody', {project: project}) -%>
    21|         /// <summary>
    22|         /// An optional partial-method to perform custom initialization.
    23|         /// </summary>

C:\Users\v-diya\repository\AutorestSdkSupport\autorest.powershell\powershell\resources\templates\serviceClientBody.ejs:14
    12|         /// </summary>
    13|         public Newtonsoft.Json.JsonSerializerSettings DeserializationSettings { get; private set; }
 >> 14| <% project.model.globalParameters.filter(p=>p.language.default.name != '$host').forEach(function(parameter) {-%>
    15|         /// <summary>
    16|         /// <%=parameter.language.default.description%>
    17|         /// </summary>

Cannot read property 'filter' of undefined
fatal   | Process() cancelled due to failure
error   |   Error: Plugin generate reported failure.
error   | Autorest completed with an error. If you think the error message is unclear, or is a bug, please declare an issues at https://github.com/Azure/autorest/issues with the error message you are seeing.

LucasYao93 commented 1 year ago

Test basic-spec-root.powershell ❌

autorest --use:..\..\..\..\autorest.powershell --tag=basic-spec-root.powershell    
AutoRest code generation utility [cli version: 3.6.2; node: v14.20.1]
(C) 2018 Microsoft Corporation.
https://aka.ms/autorest
info    |    Loading AutoRest core      'C:\Users\v-diya\.autorest\@autorestcore@3.9.3\nodemodules\@autorest\core\dist' (3.9.3)
info    | 
The default version of @autorest/powershell has been bumped from 2.1+ to 3.0+.
 > If you still want to use 2.1+ version, please specify it with --use:@autorest/powershell@2.1.{x}, e.g 2.1.401.

info    |    Loading local AutoRest extension '@autorest/powershell' (C:\Users\v-diya\repository\AutorestSdkSupport\autorest.powershell)
info    |    Loading AutoRest extension '@autorest/modelerfour' (4.23.1->4.23.1)
warning | PreCheck/CheckDuplicateSchemas | Checking for duplicate schemas, this could take a (long) while.  Run with --verbose for more detail.
warning | UsingTemporaryFlag | modelerfour.treat-type-object-as-anything options is a temporary flag. It WILL be removed in the future.
warning | SecurityDefinedSpecAndConfig | OpenAPI spec has a security definition but autorest security config is defined. Security config from autorest will be used.
C:\Users\v-diya\repository\AutorestSdkSupport\autorest.powershell\powershell\dist\plugins\sdk-generate.js - FAILURE  {"path":"C:\\Users\\v-diya\\repository\\AutorestSdkSupport\\autorest.powershell\\powershell\\resources\\templates\\azureServiceClient.ejs"}
fatal   | TypeError: C:\Users\v-diya\repository\AutorestSdkSupport\autorest.powershell\powershell\resources\templates\azureServiceClient.ejs:20
    18|     {
    19| <%# ToDo: add ServiceClientBodyTemplate -%>
 >> 20| <%- include('serviceClientBody', {project: project}) -%>
    21|         /// <summary>
    22|         /// An optional partial-method to perform custom initialization.
    23|         /// </summary>

C:\Users\v-diya\repository\AutorestSdkSupport\autorest.powershell\powershell\resources\templates\serviceClientBody.ejs:14
    12|         /// </summary>
    13|         public Newtonsoft.Json.JsonSerializerSettings DeserializationSettings { get; private set; }
 >> 14| <% project.model.globalParameters.filter(p=>p.language.default.name != '$host').forEach(function(parameter) {-%>
    15|         /// <summary>
    16|         /// <%=parameter.language.default.description%>
    17|         /// </summary>

Cannot read property 'filter' of undefined
fatal   | Process() cancelled due to failure
error   |   Error: Plugin generate reported failure.
error   | Autorest completed with an error. If you think the error message is unclear, or is a bug, please declare an issues at https://github.com/Azure/autorest/issues with the error message you are seeing.

LucasYao93 commented 1 year ago

Test provider-containerRegistry.powershell ❌

autorest --use:..\..\..\..\autorest.powershell --tag=provider-containerRegistry.powershell          
AutoRest code generation utility [cli version: 3.6.2; node: v14.20.1]
(C) 2018 Microsoft Corporation.
https://aka.ms/autorest
info    |    Loading AutoRest core      'C:\Users\v-diya\.autorest\@autorestcore@3.9.3\nodemodules\@autorest\core\dist' (3.9.3)
info    | 
The default version of @autorest/powershell has been bumped from 2.1+ to 3.0+.
 > If you still want to use 2.1+ version, please specify it with --use:@autorest/powershell@2.1.{x}, e.g 2.1.401.

info    |    Loading local AutoRest extension '@autorest/powershell' (C:\Users\v-diya\repository\AutorestSdkSupport\autorest.powershell)
info    |    Loading AutoRest extension '@autorest/modelerfour' (4.23.1->4.23.1)
error   | DiscriminatorNotRequired | Semantic violation: Discriminator must be a required property. (components > schemas > RunRequest)
  discriminator: { propertyName: 'type' }
    - https://github.com/Azure/azure-rest-api-specs/blob/5035a36bcd5b0543a9a65ee21f03bd12e301ea72/specification/containerregistry/resource-manager/Microsoft.ContainerRegistry/stable/2019-04-01/containerregistry_build.json:684:5
fatal   | Process() cancelled due to failure
error   | DiscriminatorNotRequired | Semantic violation: Discriminator must be a required property. (components > schemas > TaskStepProperties)
  discriminator: { propertyName: 'type' }
    - https://github.com/Azure/azure-rest-api-specs/blob/5035a36bcd5b0543a9a65ee21f03bd12e301ea72/specification/containerregistry/resource-manager/Microsoft.ContainerRegistry/stable/2019-04-01/containerregistry_build.json:1252:5
error   | DiscriminatorNotRequired | Semantic violation: Discriminator must be a required property. (components > schemas > TaskStepUpdateParameters)
  discriminator: { propertyName: 'type' }
    - https://github.com/Azure/azure-rest-api-specs/blob/5035a36bcd5b0543a9a65ee21f03bd12e301ea72/specification/containerregistry/resource-manager/Microsoft.ContainerRegistry/stable/2019-04-01/containerregistry_build.json:1721:5
error   |   Error: Semantic validation failed. There was some errors
error   | Autorest completed with an error. If you think the error message is unclear, or is a bug, please declare an issues at https://github.com/Azure/autorest/issues with the error message you are seeing.

LucasYao93 commented 1 year ago

Use script to generate test case.

Test template

### Tag: {testcase}.csharp

```yaml $(tag) == '{testcase}.csharp'
csharp: true
input-file:
  - .\{testcase}\swagger.json

namespace: Microsoft.Azure.Sample
output-folder: {testcase}\csharp

Tag: {testcase}.powershell

isSdkGenerator: true
powershell: true
input-file:
  - .\{testcase}\swagger.json

namespace: Microsoft.Azure.Sample
output-folder: {testcase}\powershell

FL-LearningGroup / LucasNote

[Autorest] upgrade module four. #48

Tasks

Build autorest.csharp project and autorest.powershell project

Build autorest.csharp

How to generate SDK with different generators

Autorest.PowerShell introduction

Development environment setup

Install autorest and dependencies

IDE

Code repo and branch strategy

Build, manage, use and debug autorest.powershell

How to build

Other useful rush commands

How to use local built autorest.powershell

Generate pipeline graph and artifacts for each step

How to debug

ModelerFour introduction

Plugins

tweakm4codemodel

tweakcodemodel-v2

tweakcodemodel-azure

create-commands-v2

create-virtual-properties-v2

csharp-namer-v2

powershell-namer-v2

add-azure-completer-v2

powershell-v2

llcsharp-v2

code generation

Dependencies

Build

Common steps

Cmdlets details

GetModuleGuid

ExportCmdletSurface

GetScriptCmdlets

ExportExampleStub

ExportFormatPs1xml

ExportHelpMarkdown

ExportModelSurface

ExportProxyCmdlet

ExportPsd1

ExportTestStub

GetCommonParameter

For my local environment

Test Aks

csharp

Test basic-spec-required.powershell ❌

Test basic-spec-root.powershell ❌

Test provider-containerRegistry.powershell ❌

Use script to generate test case.

Test template

Tag: {testcase}.powershell