Add support for Azure

[brikis98]

Terratest has an aws package with a number of utilities useful for AWS. It would be great to add similar utilities for testing code in Azure.

[foreverXZC]

I am a member of Azure group and I am also thinking about this these days. I wonder what kind of features are important and thus really needed.

[brikis98]

Hi @foreverXZC! Good question :)

I suspect many of the same utilities we have in the aws package would be useful, such as functions to make it easier to work with authentication, regions, machine images, logging, user accounts, and so on.

[lewalt000]

Hi folks - thanks for introducing the Terratest tool! Have there been any updates on this issue? Azure support would be a great addition for those working with Azure.

[brikis98]

I know some teams using Azure are using Terratest for testing... But no one has contributed any PRs back yet. We would very much welcome that :)

[lewalt000]

@brikis98 - thanks for the quick reply! So to use Azure, one would need to write the go code in their tests to talk with Azure, or develop their own helper library, correct?

[brikis98]

You'd write a library of helpers similar to the aws helpers, but focused on authenticating to Azure and calling various services you're testing. You'd then use your helpers to test your Azure code.

[nmiodice]

I'm contributing back some relevant work. See https://github.com/gruntwork-io/terratest/pull/332

[drmmarsunited]

@nmiodice Would love to work with you on this, as I wrote the beginnings of one as well. I was going to open a PR later tonight! My work is laid out a little differently than yours, so I might open a PR as well just to get the mainters' thoughts on how we could come together on approach.

[nmiodice]

@drmmarsunited Absolutely. Looking through your PR I like the approach you took with the authorizer as it is more robust and works with non-service principal use cases. Other than that, these look very compatible.

I suggest that we get one merged (either one) and then we can work off of that common base. Thank you for putting this PR up, and it is too funny that we did it within an hour of each other! OSS community hard at work :)

[brikis98]

@nmiodice and @drmmarsunited Thank you both for the Azure PRs! Love seeing support for another cloud 👍

Any thoughts on how you'd like to merge your two approaches together?

[drmmarsunited]

@brikis98 - We could pretty easily merge our code bases. My only concern with the approach that @nmiodice took, is that it's too focused on creating clients. On one hand, that pretty closely aligns with the how the Azure Go SDK itself works, but for testing purposes, I'd prefer to see targeted, specific, and prescriptive methods, much like the AWS module.

As someone who will immediately be a consumer of this module for my team and other teams in the business here, I'd like to see the style/approach to these modules follow as closely as possible. It makes the developer experience pretty seamless, and that would be a big win, instead of having to learn a different pattern for each cloud provider's module.

[cdhunt]

I spent a day trying to build some Azure functions and dealing with all of the Azure SDK Clients creation was off-putting. Hiding that tedium would be very appreciated as a possible user.

[brikis98]

Agreed on maintaining a common style for Terratest with prescriptive methods. That said, @nmiodice, wasn't your PR just one of several planned ones? Are the follow-up PRs similar in style to what @drmmarsunited is mentioning?

[drmmarsunited]

@brikis98 Not to be nagging on you guys, but this initiative is super important to the teams here (and I hope to the larger community). Can we time box the feedback window here? I'm ready to start contributing more helper methods to the Azure module to enable my teams. I'm totally game to work with @nmiodice, but I want to keep the ball rolling and not get stalled.

[brikis98]

Agreed on not holding it up too long. @nmiodice Let me know your thoughts. If we don't hear back in a day or two, we'll roll forward and hopefully you can add your contributions later on.

[brikis98]

OK, moving forward with @drmmarsunited's PR.

[MathieuBuisson]

Following the work that @drmmarsunited has done to introduce Azure support, I would like to contribute coverage for network-related resources in the azure module :

• Virtual Networks
• Subnets
• Network Security Groups
• Public IPs

Should I open a separate issue regarding Azure networking resources ?

[brikis98]

I don't think any of the Azure PRs were ever completed... So I think we still need a base PR!

[drmmarsunited]

@brikis98 mine was merged a while back.

[brikis98]

@drmmarsunited Ah, I'm sorry, you're right, I was thinking of https://github.com/gruntwork-io/terratest/pull/332.

[TsuyoshiUshio]

Hi @brikis98

I'd like to contribute AKS (a.k.a. ContainerService as AzureSDKForGo). Can I do it? It is not included this open PR. https://github.com/gruntwork-io/terratest/pull/332

[yorinasub17]

Hi @TsuyoshiUshio

As Terratest has grown, we've had to think more about what types of helper functions we want in the library to make sure it doesn't become so big as to be unusable. Please check out our contribution guidelines, especially the part about infrastructure and validation helpers.

Specifically, to decide if we want to add something like this to terratest, we need to pass four tests:

• [Platforms] Is the function for one of the supported platforms?
• [Creating infrastructure] Is the function validating infrastructure (as opposed to creating or modifying)?
• [Complexity] Is the function complex to implement directly with the existing APIs?
• [Popularity] Is the function a popular use case?

I think this passes the litmus test for "platforms" and "popularity", but I am unsure about "complexity" and "creating infrastructure". Since I don't have experience with the Azure SDK, it is hard for me to gauge the potential test functions here. Can you share some more details on some of the APIs you would like to support, their use cases, and what makes it hard about using the SDK directly such that terratest wrappers would be beneficial?

PS: Thanks for writing about terragrunt and terratest in Japanese! 全部読んでます！

[TsuyoshiUshio]

Thank you for your comment, @yorinasub17

I'm planning to create some validation helper methods. It doesn't create/delete infrastructure, just validate the deployment of the resource. It is something like this code without Create/Update/Delete method.

https://github.com/gruntwork-io/terratest/blob/master/modules/aws/ecs.go

I might add retry feature since aks resource creation takes time to up and running. When I'm writing the code of terratest with aks and my original test helper, it required to wait until the resource is being ready.

The method I'm gonna use is

https://github.com/Azure/azure-sdk-for-go/blob/master/services/containerservice/mgmt/2019-11-01/containerservice/managedclusters.go#L270

Test will be something like this. Just validate if the cluster is now up and running.

https://github.com/gruntwork-io/terratest/blob/master/test/terraform_azure_example_test.go

In addition of retrying, in case of Azure, Just calling get method of Azure SDK for go might be difficult for many Azure users. Not very difficult, however, a lot of Azure users doesn't familiar with go coding.

The Azure SDK for go documentation is not rich. https://docs.microsoft.com/en-us/azure/go/

There is no documentation/example for AKS resource.

Reference is also not rich. :( https://godoc.org/github.com/Azure/azure-sdk-for-go/services/containerservice/mgmt/2019-11-01/containerservice

Users need to read through the code of the repo, find resources, authenticate it,... something like that required.

AKS is a managed service of kubernetes. One of the top usage of terraform. After accepting this contribution, if it doesn't take long time, I'll keep on contributing several resources on Azure. (these are also validation helper methods.) I have an actual workload for my customer. I believe, it helps to include more Azure users for terratest.

P.S. Thank you for reading my blog post, I'll keep on posting it. I just love your product. ;) ありがとうございます！

[yorinasub17]

Thanks for sharing detailed notes and links with the use case! That definitely makes sense. Feel free to contribute a PR and we'd be happy to take a look.

[TsuyoshiUshio]

Thank you, I'll do it.

[robmorgan]

The only thing I'm concerned about here is that we don't have a proper sandbox for testing our Azure helper methods. I think it's okay to welcome this contribution, but I'm wary of the support for Azure growing in our codebase and not having the proper system in place to test the code.

[TsuyoshiUshio]

@robmorgan I see. We need to discuss the point. I'll also discuss with my colleague. (I'm Microsoft guy).

[rguthriemsft]

@robmorgan, whats needed for a proper sandbox? If you can give some requirements I can look into it.

[robmorgan]

@TsuyoshiUshio @rguthriemsft from my side I would see a proper sandbox as being an Azure cloud account controlled by Gruntwork but with credits provided by Microsoft. Obviously the way Terratest works is by creating real, live running infrastructure so I would imagine an annual Azure spend of somewhere between $4-6k required to validate the community's work. Right now we only test that the Azure module compiles so I'd like to extend this to validating the actual APIs.

[yorinasub17]

@robmorgan Let's also discuss this internally. I think I have a few ideas for limiting the spend, but we should think through all the implications as well. I think we will need credits eventually, but for the super short term, we might be able to stay afloat with a bit of creativity.

[rguthriemsft]

If yall can let us know what might work I can look into options. We are eager to add support for several resources in Azure. I can't commit for sure we can do credits but will investigate.

[TsuyoshiUshio]

Hi @robmorgan @yorinasub17 I finish the PR and get reviewed finished. Could you answer the @rguthriemsft 's question? He is waiting for you guys answer.

[rguthriemsft]

@robmorgan @yorinasub17 did you have some ideas on how you might work with azure for integration testing of Azure PRs?

[yorinasub17]

Hey @rguthriemsft and @TsuyoshiUshio,

First, apologies for our slow response on the topic here. To explain the current situation and be completely transparent, we as a team are not focused on Azure right now. We want to eventually move on to supporting Azure in our company, but this is not a high priority at the moment as we focus on AWS and GCP.

With that said, we believe that Terratest should support it reasonably, especially given the interest from Microsoft (e.g Microsoft wrote a blog post about Terratest! https://docs.microsoft.com/en-us/azure/terraform/terratest-in-terraform-modules), so we are looking for ideas for how to do it. That is, the following areas are where we need help:

• A community lead (perhaps from Microsoft?) who can help review and maintain the quality of the Azure features of terratest. We are happy to help with PR reviews, but given our limited experience with Azure, our reviews will be limited to general code quality of terraform and golang.

• Credits for running unit and integration tests for the library. Since Azure is not a priority for us, and since we are a small startup, it is hard for us to justify spending $4-5k annually to maintain a community project. By the way, this estimate is based on our spend of AWS and GCP from terratest. Since the Azure library is still small, this is likely to be less, but as we get more and more contributions, this will probably be a reasonable estimate.

As far as the latest contribution you have shared on AKS, since it looks ready to go, we'll be moving forward with it. That is, we'll kick off our sanity check build and if that passes, go ahead to merge it. However, for contributions going forward, we have agreed that it will be difficult for us without some kind of automated testing. So to move forward, we will need to address the above two concerns.

Hope that makes sense. Happy to answer any questions you might have!

[rguthriemsft]

Thanks for the feedback. I am investigating options and will get back shortly.

[robmorgan]

Okay, great!

[tsalright]

Hey everyone, I am also looking at contributing to the Azure part of Terratest. We recently discovered this and love it. At present, we are mostly a .Net shop with a couple of people dipping their toes into Go. What @TsuyoshiUshio said in his Jan 15th post about limited Go knowledge and documentation around the API, having wrapper methods here for Azure validation would be helpful. While I am not an employee of Microsoft, I have spent the last 3 years working in Azure and went from ARM templates, to Azure CLI, to using Terraform with Terratest. I have been looking for a project to support and this one really excites me and would love to help in this space around Azure.

I would be open to helping lead, review, and maintain the quality of Azure features of Terratest.

In regards to automated testing, there are many services that have free tiers that we can make sure the automated tests are using to validate these. For all the module validations I am doing for Alaska Airlines, I have a single Resource Group and Service Principal created to apply and destroy terrafrom against. I also have a nightly job that blows away anything in that resource group in case something went wrong during the tests so I don't have resources living longer than they should. I will keep my eyes on the cost of our solution and can provide some data if that will help.

I plan on adding the app service plan, web app, function app, auto scaling, alerting, app insights, and action group suite that is generally used together when creating PaaS web apps.

We also have plans for Redis, CosmsoDB, SQL Server, Service Bus, Event Grid, Event Hub, Key Vault, and Resource Groups.

[yorinasub17]

@tsalright Thanks for the interest! We're discussing this internally with the team and will get back to you on this.

[rguthriemsft]

gruntworks team, I have a proposal for moving forward. Would it be possible to have a discussion of which the outcome we can post back into this issue? You can reach me at rguthrie at microsoft dot com of which I can outline our thinking? We would love to imrpove the azure support and @tsalright happy to work together if we can find a good solution for accepting PRs.

@robmorgan @yorinasub17

[yorinasub17]

@rguthriemsft Just sent you an email!

[tsalright]

@robmorgan @yorinasub17 Have you thought about adding some additional tags to the issues to identify Terratest Core, GCP, AWS, Azure, etc to quickly identify the areas they are related to? I was thinking of this when I was looking at the issues and wanting to see which ones were Azure related so I could see what I could help with beyond the stuff I would like to add.

[robmorgan]

@tsalright we already have one for GCP, but I guess we could for the other packages. @yorinasub17 any preferences?

[yorinasub17]

UPDATE: we're in talks with @rguthriemsft and have a call scheduled next week. Once agreed on a plan forward, we'll post the notes here. @tsalright , we appreciate your help! We'll be following up with you once we figure out our plans going forward with Microsoft, hopefully by end of week next week.

Have you thought about adding some additional tags

I think this is a good idea! We haven't done a good job with tagging even though we have the gcp label... I think we can do this going forward, but we don't quite have the bandwidth to go through old issues and label things at the moment.

EDIT: just created the labels core , AWS, and Azure.

[yorinasub17]

Oh also, forgot to mention that we will continue to welcome feedback from anyone in the community no matter how the call turns out with MS. We hope to work with MS to set up some basic scaffolding to make maintenance easier, but we are eager to get help from the community on PRs, issues, etc.

[brikis98]

Update for everyone: We are starting to do some work around setting up proper CI / CD for Terratest with Azure. We can't share any details or timelines right now, but just wanted to give you all a heads up that we haven't forgotten about this!

[brikis98]

Also, I just tagged a number of issues/PRs with the Azure label. We'll come back to those when the work mentioned in my comment above has been completed.

[tsalright]

Just a quick update to the Terratest Azure community. We are working on getting the pipeline up and running to start processing the Azure PRs that are out there. We are in the testing and validating phase of it. We will have an update in another week or two.

[rguthriemsft]

@brikis98 / @yorinasub17 I think we can consider this issue resolved and would like to just add a link to the Developing for Azure as well as the examples. Thoughts?

[yorinasub17]

Sounds good to me! I'll mark this ticket as resolved.