Change Request: READMEs (specially at the `test` and `samples` folders) could provide how to deploy resources

[sadasant]

Users of the Azure SDKs could encounter issues while trying to run our samples or live tests, since they might require specific configurations to run. Our readmes could have additional indications around how to deploy the required resources.

In this issue, I will describe the history behind this request, and the possible solutions. For each solution, I will include the pros and cons that I personally can think of while writing this issue.

Background

While developing the Azure SDK for JavaScript and TypeScript, our team was encountering an issue regarding how to properly express how to deploy the exact resources needed to run our live tests. We started going through what other teams had been using, and we spotted that the Service Bus SDK for .Net had a Deploy Button in their README:

We also discovered their use of ARM templates. This started a multi-month effort of bringing ARM templates to our projects. Nowadays, most (if not all) of our SDKs make heavy use of ARM templates for the automation of test resources.

Having these templates at hand, it seemed natural to continue to use these Deploy Buttons. They currently can be seen in some of the READMEs of the Azure SDK for JavaScript/TypeScript. Here's one example from the KeyVault Certificates' readme: https://github.com/Azure/azure-sdk-for-js/blob/2afdad15f4b2d6ffc449ed6e245e0389c428b64a/sdk/keyvault/keyvault-certificates/README.md#getting-started

In parallel, the Engineering Team has been working with different members of our team at large to continue improving on the test tooling, and now these tools are more approachable and can be used more easily by anyone who wants to build their test resources.

Currently, the TS/JS team has halted the adoption of the Deploy Button and has started the conversation to move towards the PowerShell Tooling.

General requirements

If we decide to expand our READMEs to include information on how to deploy resources, this information must at least clearly state:

• How to deploy TEST resources.
• How to deploy SAMPLES resources.
• What resources will be deployed in each case.

Keep in mind that these indications could be in either the main README of a project, or in READMEs located at the samples folder, or at the test folder. At first glance it seems reasonable to suggest that this is better explained in the individual folders.

Possible solutions

Deploy Button

The Deploy Button is a simple image that links to an Azure endpoint that generates a form based on an ARM template. The Markdown source code is as follows:

[![](http://azuredeploy.net/deploybutton.png)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-sdk-for-js%2Fmaster%2Fsdk%2Fkeyvault%2Ftest-resources.json)

The image ![](http://azuredeploy.net/deploybutton.png) is linked to the Azure endpoint https://portal.azure.com/#create/Microsoft.Template/uri/, to which we pass at the end the URL encoded path of an ARM template. In that case: https://github.com/Azure/azure-sdk-for-js/blob/2afdad15f4b2d6ffc449ed6e245e0389c428b64a/sdk/keyvault/test-resources.json

Pros and cons of the Deploy Button

Pros:

• It is a visible element on our READMEs that could be interpreted as a call to action.
• A button and a web form might be easier to work with for a greater population, in comparison to PowerShell.
• The button goes to an auto-generated form, and the ARM template used can re-use some of the contents of the ARM template used for live tests, so some of the work up-front, and perhaps even maintenance work, can be saved.

Cons:

• This approach requires ARM template knowledge for developers.
• Requires writing, testing and maintaining several ARM templates.
• The automatic generation of the form might not be intuitive depending on how the ARM template is written.
• The number of resources generated is hidden from plain sight for users unaware of how this deployment happens. People might end up purchasing undesired resources.
• The READMEs might end up needing indications on how to fill these forms, and what resources are being purchased.

PowerShell Tooling

In the process of improving the Engineering Automation of the test resources, our teams have made our Test tools more approachable. We currently have a guide and examples of how to use these tools to deploy test resources. Something similar could be used for both tests and samples.

• The current guide on Integration Testing with live services for JS/TS.
• Examples of using the test tooling to generate resources based on ARM templates: https://github.com/Azure/azure-sdk-for-js/blob/master/eng/common/TestResources/New-TestResources.ps1.md#example-1
• A PR to the JS/TS repository which we've doing in parallel to this issue, in which we are changing from the Deploy Buttons to the already documented and properly working Test tools, only for test resources though: https://github.com/Azure/azure-sdk-for-js/pull/8398/files

Pros and cons of the PowerShell Tooling

Pros:

• These tools already exist in the majority of our SDKs (if not in every SDK).
• Documentation for these tools already exist.
• New specific ARM templates, one for test and one for samples, can be made from existing ARM templates, and they will be instantly compatible with the existing documentation.
• Minimal new documentation would need to be added to the samples README and the test README.
• Any improvement to these tools can be carried on to every other SDK we have with minimal effort.
• Most of our engineers can work well with PowerShell.

Cons:

• Requiring powershell might be a barrier to some of our users, even if small.
• This approach requires some level of maintenance to new ARM templates and small sections in many READMEs.

Tutorial

Another approach would be to provide a step by step tutorial on how to deploy the necessary resources for both tests and samples.

Some of our packages currently employ a minimal tutorial showing how to deploy the most minimal resource that they are targeting. An example can be seen in the TS/JS @azure/search-documents README: https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/search/search-documents#prerequisites

Pros and cons of the Tutorial

Pros:

• A tutorial is a more human to human approach of guiding contributors to participate in our projects.
• The tutorial can start with the most minimal setup for tests and samples, and then link to more advanced settings from existing documentation outside of our projects.
• Potentially requires no knowledge of ARM templates or PowerShell. The Azure CLI is likely sufficient.
• The resources being deployed will be clearly stated as part of the tutorial. There will be no abstracted out procedures or configurations (besides the intrinsic behavior of the Azure CLI).

Cons:

• Test and samples tutorial might take some time to be properly written.
• A template tutorial, and a sample tutorial, should probably be made to show what are the expectations for these documents.

Closing statement

Deploying resources are a key part of running tests and samples. The fact that each project has been approaching the related documentation slightly differently most likely means that we're getting ready to work together on some standard to include as part of the general guidelines, so that future work can minimize variations, and potentially maximize the possibilities of seeing contributions from users at every level of knowledge, as well as increasing the incentives to try and purchase Azure resources.

[heaths]

This is a good use of these files, though we may need to consider how the user supplies inputs required by the scripts I wrote that process test-resources.json for both CIs and dev workflows. It definitely centralizes and unifies resource deployment across languages.

[jsquire]

I'd make the distinction between audiences, personally. We should be providing a lightweight and free-standing template for customers wishing to explore the library and create applications. I see this as part of our "getting started" area where customers can quickly use these to create a minimal set of resources to play with snippets and use with samples.

I'd avoid trying to keep things at this level for contributors who which to compile the library and run tests; the test resources are typically heavier and have deeper ties to the ambient environment (CI, developer setup) which would make using the simple button rather difficult. I'd love to see us have a full "F5" experience where contributors could run a setup script or get a plug-in for their development environment that helps make running tests using the "per-run" test resources seamless as well - but I see the two efforts as independent.

[jsquire]

//cc: @czubair, for visibility as our documentation owner.

[maggiepint]

All of the things described here are definitely of benefit to customers - but I'm not entirely sure that they are entirely appropriate to the readme itself.

Our user studies consistently show that users don't read almost any text on documentation, instead scrolling directly to code samples. The consistent ask of customers is for quick snippets of code that help them get done what they are doing. The normal case for customers is that they already have Azure resources deployed, and they are looking to quickly interact with existing resources used in existing projects.

We are working on a re-work of the developer hub documentation - these are the language-level pages linked from here: https://docs.microsoft.com/en-us/azure/developer/

Information about how to best test with the SDK is already on the roadmap for that rework - perhaps this would be more appropriate in that content?

[sadasant]

Here's a new issue emphasizing only the main README template: https://github.com/Azure/azure-sdk/issues/1248

Please let me know if this new issue is better! I can make sure to improve it to properly present the issue I'm trying to express.

While the main README of a package can express how to deploy the most basic version of the resource that the package is managing, other users might want to run samples and tests. This problem then also occurs in those readmes, and even if the solution could be similar in nature, the configurations and implications will be for sure different.

@maggiepint what you mention about the roadmap of the test docs rework, that sounds good to me!

Please help me: How can I bring up issues about those other readmes?

Thank you for your time.

[sadasant]

I have updated the title and the first paragraphs to more accurately mention that this issue is not specifically about the main README of each package.

[heaths]

@jsquire that's what these test-resources.json files facilitate. Right now, they require a separate invocation but are otherwise uniform in how they are used (all by running eng/common/TestResources/New-TestResources.ps1 - both for CI and devs). But this is for the README experience, not CONTRIBUTING (which should mention these steps). The same file can be used to proffer resources customers need and will do so in a manner consistent with CI and dev environments, which - because we should be compiling and running samples that use those resources - we know will work. So we have validation built-in as well.

[heaths]

Getting to just an F5 experience is still on the backlog. Other things have been priority, but having a simple step prior to that to prepare things is still simpler than what many OSS repos (including in Microsoft) have you do. Still seems like a win, and we can evolve it over time.

[github-actions[bot]]

Hi @sadasant, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.