Generator Notes

[fearthecowboy]

PowerShell cmdlet generation

AutoRest Tenets:

• The experience for AutoRest and extensions must be consistent between cmdline, in vscode, via CI, and as a library. This ensures that the tools can be used in all the supported scenarios without having to do special effort or work.

• Extensions should ideally work the same regardless of platform (windows[x64/x86], linux[x64], osx[x64]).

• Extensions must be implemented using JSON-RPC over stdin/stdout.

• Generators can be written in any language that can communicate over stdin/stdout and process JSON messages.

• All extensions must install as NPM packages that, if there are platform requirements, must be installed as NPM packages as well. Interactivity is not permitted during install (since it happens inline.) Elevation/Admin access to do platform-specific actions is not permitted.

• Generators are not permitted to access the filesystem directly; all files must be accessed via AutoRest RPC interface
  This ensures that the generator can work cleanly regardless how AutoRest is invoked; AutoRest uses this to supports remote filesystems (https) and Visual Studio Code (where the contents might not be on disk yet.)
  File writes must go back thru the interface too; this ensures that consumers of AutoRest have control how the files are actually handled.

• Generators can not be 'interactive' with the user; settings are opt-in, and can be retrieved via the AutoRest RPC interface.
  This ensures that the actual automation and UI consumer has control how options are presented to the user.

• Generators do not parse the OpenAPI spec directly; the code-model is the language-agnostic representation that is currently supported.
  This ensures that there is consistency between generators for support for all OAI protocol extensions, and that the OAI docs can be kept in an untainted state. (no random x-ms-XXX generator settings are permitted)

• Generators must work in the AutoRest pipeline so that artifacts can be transformed before and after the generation process as the needs of the consumer sees fit.

• Generators must specify errors or failures via the AutoRest RPC interface using the common standard for messages, including providing source-mapping of errors to model locations.

• Generators must use the https://github.com/Azure/autorest.testserver project to provide test coverage for all common REST and AZURE REST scenarios. This enables support for coverage analysis (ie https://github.com/Azure/autorest.csharp/pull/60#issuecomment-349826668 ) which drives quality and consistency between generators.

Issues with current PSSwagger:

Problematic

• written in powershell (desktop)

  • appears to require powershell desktop (see https://github.com/PowerShell/PSSwagger/blob/developer/Tests/TestUtilities.psm1#L195)
  • known potential issues:
  • there are c# class files in the powershell
  • uses csc.exe to compile c# files (not available on non-windows platform like that)
  • bootstraps some dependencies with nuget.exe (C# compiler; not available on non-windows platforms; shouldn't be handled in code.)
  • that could be done with a dependency to the NPM packages for .net core (dotnet-sdk-2.0.0)
  • hard-coded to use desktop powershell (even if run with core powershell; again not-xplat)

• would need to be refactored to add a JSON-RPC interface -- might be able to borrow our c# implementation of the JSON-RPC

• appears to just use AutoRest to generate models, and then re-implements code-generation behavior independently. This should be adapted to use the AutoRest pipeline so that the generator declares itself as a consumer of the c# output, and outputs files/messages via the interface

Unacceptable

• wraps calls to autorest to generate c# artifacts, then writes out stuff on top of that (that's backwards; it should not be a consumer of AutoRest, should be an extension) Extensions specify their inputs in the configuration, and can leverage other generators to do work on their behalf.

• re-implements logic/knowledge that is well-defined in AutoRest where we have very consistent patterns. This would hamper the generator's ability to keep up with changes that all other languages are getting for free. (Including supporting OpenAPI 3 and beyond)

• Takes information out of swagger directly, implements rest calls on top of that. ❓ This prevents supporting new representations (ie, OpenAPI3) and bypasses the transformation pipeline that can be used to customize support for generating code across all languages.

• By not supporting AutoRest RPC interface, it doesn't support multi-swagger inputs, or configuration at all.

• Reimplemented github access (which is a waste, and doesn't support all the features AutoRest already handles)

• Still has references to 'CompositeSwagger'.

• fragile; explicitly depends on output layout and naming.

• massive powershell-specific extensions to swagger.
  This taints wire-protocol specs (OpenApi) with information that is not appropriate to have in the document that describes the wire protocol. Code generation information should be persisted in a literate configuration file and retrieved thru the AutoRest RPC interface.

• would require extensive changes to support OAI3 (autorest languages will get the bulk of that for free.)

• doesn't support Azure Storage request pipeline model (may be necessary for some scenarios) since client internals hidden away

[fearthecowboy]

Ideas:

• Highlight what's good about the generator

• Identify feature gaps (what's features are not supported, that need to be)

• Identify what structural issues need to be fixed (the 'unacceptables')

• Ask: how long will it take them to be compliant so we can include this?

• Ask: what their SLA is for fixes, changes and updates; we support 75+ RPs; we need to be able to have fixes reliably handled for all of our customers.

• If they aren't going to be able to support it, or can't make it compliant with the rest of AutoRest, we can look at harvesting code and building a new generator that is compliant.
  (we're not going to support something written in powershell)

FYI:

PSSwagger has a total of 218 downloads for the current version. (since released oct 10th -- about 4 per day )

AutoRest is seeing 18,000+ downloads per month. (about 700 per day)

[olydis]

Done swell

• naming convention conversion doohickey (converts typical Swagger verbs like "Create", "List", "Get", ... to corresponding PS idioms)
• x-ms- extension support (x-ms-parameter-grouping, flattening, ...)

Problems

• <see Garrett's comment>
  • doesn't support Swaggers with external references, which includes all that have examples and hence most of the specs repo:
  • doesn't support Swaggers that are not JSON

Benefits of making AutoRest extension

• technical things for free (ext. reference handling, configuration file support, most "unacceptable problems" solved)
• deduplication of quite some logic
• automation/VSCode/uniformity/... goodness

Expectations for generated PowerShell

Parameters

• have parameter sets both for "flattened" version of objects and the full objects (and maybe intermediate stuff)
  • one specific example is something like IResource vs rgname, name, etc. for Get operations
• use parameter sets in case of polymorphic parameters
• performance impact of many parameter sets?
• lower priority: think about using simpler parameter types mapping to complex internal types, e.g.
  • natural string representation for complex concepts
  • if one property influences/restricts the next

Common Code

• resource type definition (base type or interface => interoperability)
• error/exception types
• factor out common code, e.g. for Patch, Post, Delete, ...
• asjob wrapper (job that wraps a task)

Also generate...

• shouldProcess automatically
• completers
• format.ps1.xml (at least templatey version that overapproximates what you would want; or taking in simply formatted version of that and spits the complicated doohickey out)
• types.ps1.xml (PS remoting)

TODO: check perf of statically compiled C# part vs C# sources