Integrate with Hashicorp Terraform

rafavallina commented 3 years ago

Extend support on AWS Proton to enable defining and provisioning infrastructure using Hashicorp Terraform

mwarkentin commented 3 years ago

I'm really curious what this will look like - is it going to run terraform for us? Hook into Terraform Cloud / Atlantis?

rafavallina commented 3 years ago

Hi. Can I flip the question around and ask how you would like this to work?

We are indeed looking at the best way to make it work, but I don't want to take a stance just yet. Would love to get some suggestions!

raymyers commented 3 years ago

@rafavallina There are going to be different perspectives, but just weighing in for my use case. I'd prefer it didn't depend on another tool, so Proton running the Terraform within AWS, just as though it had been executed by Code Build but with better integration. Maybe with an option to approve the tf plan before executing.

Very excited for Proton btw, thanks!

mwarkentin commented 3 years ago

A couple of random thoughts:

being able to review plans and approve them before apply is a must have
a couple of other major hosted workflows for terraform are Atlantis and Terraform Cloud. They have some differences in workflows that are probably worth understanding (I prefer atlantis locking and apply from branch before merge vs terraform clouds apply from master on merge)
support for module installation from terraform cloud private registries would be important (maybe some sort of secret manager integration for api tokens?)
would you provide state file management natively or expect people to configure that themselves on s3 or terraform cloud?
would users get access to the raw generated terraform configuration .tf files somehow or would that all be behind the scenes?

shlomimatichin commented 3 years ago

@mwarkentin +1 on everything you wrote. Also wanted to mention env0 as a rising player in this domain, they do most of the things you mentioned.

rafavallina commented 3 years ago

Hi all! Thanks @raymyers @mwarkentin @shlomimatichin for the input. We're looking into Terraform Cloud as our first approach at this point - not a final decision just yet. To the question about state file management: no plans for Proton to provide it at this point (TF Cloud would take care of that). Regarding access to the .tf files: I don't have a fast answer, but there should be a way for administrators to see the template that was executed - Proton is working with those services, but not replacing them, so we don't want to make using them more difficult. With CloudFormation, this can be done by going to the corresponding stack and seeing the template. Not sure how we can do it in Terraform, but we'd have to at least make sure it's possible to see.

mwarkentin commented 3 years ago

I believe Terraform Cloud has a state viewing functionality but I'm not sure if that would be enough on its own.

oshah97 commented 3 years ago

Is there any update on the status of Proton integration with Terraform? Interest from Elsevier to look into using Proton as an enterprise solution.

rafavallina commented 3 years ago

Sorry for the delay in answering @oshah97 - Are you asking for an update on the status of integrating with Terraform? Nothing to share on this front for now, still working on it.

oshah97 commented 3 years ago

@rafavallina Okay, thank you for getting back to me!

pgdad commented 3 years ago

Terraform Cloud.... Would it be possible to integrate with Terraform Enterprise? Lots of larger users/customers will not be able to use Terraform Cloud, but do use Terraform Enterprise.

rafavallina commented 3 years ago

Thank you @pgdad! Still not decided, but we're looking at all options. Do you have any specific examples (that can be mentioned) for customers for Terraform Enterprise? Any data point helps!

pgdad commented 3 years ago

I work for a 'large regulated industry' company, and as such can't divulge any real details.

It should be sufficient to mention that if it makes sense to integrate proton with terraform cloud, then it makes sense to integrate with terraform enterprise. No serious player in the 'large regulated industry' will ever use terraform cloud. They can't.

I would be more than happy to have a real discussion on this (and other topics). If you want to, let me know and I'll have our TAM setup a meeting.

From: Rafael Alvarez notifications@github.com Sent: Monday, March 8, 2021 8:25 PM To: aws/aws-proton-public-roadmap aws-proton-public-roadmap@noreply.github.com Cc: pgdad pgdad@hotmail.com; Mention mention@noreply.github.com Subject: Re: [aws/aws-proton-public-roadmap] Integrate with Hashicorp Terraform (#1)

Thank you @pgdadhttps://github.com/pgdad! Still not decided, but we're looking at all options. Do you have any specific examples (that can be mentioned) for customers for Terraform Enterprise? Any data point helps!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/aws/aws-proton-public-roadmap/issues/1#issuecomment-793240348, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAFL5D5GAGLKKNDUVZ4ZMHLTCV2IZANCNFSM4T57DMXA.

Zambonilli commented 3 years ago

Anyone tried using the AWS CloudFormer tool to port terraform created objects for Proton? There might be a stop gap where we could keep source terraform, terraform apply to an account, use cloudformer to export cloud formation templates from the terraform generated objects. It's not perfect, and would require hand inspection but source would be terraform.

rafavallina commented 3 years ago

Hello everyone, this request has a bunch of notes and +1s, which is very exciting. Thanks! I'm actually in the lookout for customers using Terraform Open Source, to learn more about how it works today and ask questions about what Proton can do. If anyone in this thread is interest, please tell me and we can figure out a way to connect! Thanks :)

oshah97 commented 3 years ago

Hello everyone, this request has a bunch of notes and +1s, which is very exciting. Thanks! I'm actually in the lookout for customers using Terraform Open Source, to learn more about how it works today and ask questions about what Proton can do. If anyone in this thread is interest, please tell me and we can figure out a way to connect! Thanks :)

Hi @rafavallina, Would love to get in contact with you to talk about this! Our team has been looking forward to using proton since it was announced :)

rafavallina commented 3 years ago

Hello everyone, this request has a bunch of notes and +1s, which is very exciting. Thanks! I'm actually in the lookout for customers using Terraform Open Source, to learn more about how it works today and ask questions about what Proton can do. If anyone in this thread is interest, please tell me and we can figure out a way to connect! Thanks :)

Hi @rafavallina, Would love to get in contact with you to talk about this! Our team has been looking forward to using proton since it was announced :)

Awesome! How can I reach out to you? You can also try to send me a Twitter DM on twitter.com/rafavallina

oshah97 commented 3 years ago

Hello everyone, this request has a bunch of notes and +1s, which is very exciting. Thanks! I'm actually in the lookout for customers using Terraform Open Source, to learn more about how it works today and ask questions about what Proton can do. If anyone in this thread is interest, please tell me and we can figure out a way to connect! Thanks :)

Hi @rafavallina, Would love to get in contact with you to talk about this! Our team has been looking forward to using proton since it was announced :)

Awesome! How can I reach out to you? You can also try to send me a Twitter DM on twitter.com/rafavallina

Please feel free to reach me at o.shah@elsevier.com!

rantoniuk commented 3 years ago

I'm happy to join the discussion, especially around CI/CD of terraform maintaned infra and how Proton is positioning itself there. TBH I'm looking at Proton, but I haven't found (yet?) any examples of e.g. serverless workflows for CI/CD of multi-service applications. For example, we have multiple repositories with repo-per-service and one infra repository that manages via CDK the shared infrastructure like DynamoDB tables etc. - I'm curious if Proton could help in this kind of workflow.

rafavallina commented 3 years ago

Hi @warden Thanks! I'd be happy to talk - I'm not sure I fully understand your setup, so it'd be helpful for me to figure it out

ahilsend commented 3 years ago

Hi @rafavallina! We are heavy users of Terraform Open Source and are currently looking into AWS Proton for our future services deployments. This would be a huge benefit for our teams. Please feel free to get in touch via andre.hilsendeger@tink.com.

rafavallina commented 3 years ago

Thanks @ahilsend! I'll drop you an email and we can go from there

adi-ads commented 3 years ago

@rafavallina we are running 40 different software teams and evaluating proton to help drive support for all the teams from a core release & infrastructure team. Can be reached on adi.chikara@triconinfotech.com

CloudMarshall commented 3 years ago

Hi Rafa,

We are heavily interested in AWS Proton since its announcement. All our IaC is heavily Terraform based and we are keen upcoming integration between Proton and TF. I can be reached at matthew.marshall@ao.com

rantoniuk commented 3 years ago

@rafavallina how about having a roundtable of people interested (maybe in a second stage) so that we all know where Proton will be heading? :-) (ps. you can reach me at radek.antoniuk@cloudfolks.io)

RichiCoder1 commented 3 years ago

Much smaller team, but also interested in what Terraform + Proton could/would look like: rsimpson@uship.com

rafavallina commented 3 years ago

This is exciting! Thanks so much everyone for the participation. @MPM3278, @RichiCoder1, @adi-ads I'll reach out to you on email. @warden I personally prefer a round of 1:1s, even if it takes longer, so we can discuss more in-depth - happy to talk now or as we get closer to it :)

yanivpaz commented 3 years ago

Hi @rafavallina is there any estimation if it's going to be added to Proton roadmap?

rafavallina commented 2 years ago

Hi @yanivpaz Yes, we are actively working on this right now. I can't promise dates, but this is one of our top priorities.

rafavallina commented 2 years ago

Hi everyone! Wanted to provide an update on this issue as I know is of much interest to a lot of people. Although I still can't promise any dates, we are making good progress on a Terraform Open Source support mechanism.

Our current thinking is to rely on the customer's existing infrastructure pipeline. You will be able to create Proton templates using your HCL modules, and at the time of service creation or update Proton will render the template(s) and make a PR to a configuration repository, allowing your existing infrastructure pipeline to run Terraform plan/apply. The service templates can be made up of one or more Terraform Modules, so as a platform team you can break them down into individual components, but still provide a single template to developers. This reduces the need for developers to explore all individual modules, provides a way to ensure that they are used according to standards, and lets the platform team decide which versions of each individual module go into the template.

I'd love to get feedback and suggestions! Either here or in direct conversations. Feel free to chime in!

rafavallina commented 2 years ago

This is live on preview as of 11/24! We'd love to get feedback on the feature. There are a couple of items for which we will be looking for more input, so anyone in this thread that is interested please feel free to reply and we can schedule time to discuss. Thanks everyone for your patience!

I'm keeping the issue open until we hit GA

yanivpaz commented 2 years ago

@rafavallina Thanks alot ! it will be nice to have SIG /virtual meeting on this feature

mwarkentin commented 2 years ago

Here are some references in case you wanted to check out the preview release:

rafavallina commented 2 years ago

@yanivpaz Awesome! Do you happen to have an Account Management team that you work with and can help us get in touch? If not, I'd suggest sending me a DM on Twitter: twitter.com/rafavallina

rafavallina commented 2 years ago

Closing this as we shipped the feature in GA

aws / aws-proton-public-roadmap

Integrate with Hashicorp Terraform #1