Use terraform instead of own provisioner code

[katexochen]

Terraform/OpenTofu is a common and well established provisioning mechanism. The provisioning code in terraform could be used in the e2e tests, but would be test agnostic, so we could also use it in development to setup a cluster in a predefined way, and even expose it to users as entry point to set up the needed infrastructure.

I know some of us already have some terraform code or deployment scripts out of tree, including me. We also have some terraform code for CI and IBMCloud. Providing a common and simple solution to provision the infra would also lower the hurdle for new developers.

[katexochen]

Terraform should cover all supported cloud providers, including libvirt.

[stevenhorsman]

I think the theory behind switching to the e2e framework code for provisioning was to try and keep it all consumed in the same place (and also we only had terraform for IBMCloud). Are you proposing making the e2e tests two step again, or calling the terraform from the e2e framework (and setting up the terraform.tfvars (or whatever the OpenTofu equivalent is) from the e2e properties input if required?

[katexochen]

Are you proposing making the e2e tests two step again

This. Infra provisioning and e2e testing should happen in two separate steps. First do the provider specific provisioning, then execute the generic tests based on the kubeconfig.

[katexochen]

The azure e2e seems to already be divided into these steps.

[stevenhorsman]

ok, sorry for the questions, but I just want to clarify - are you just factoring the cluster creation into the provisioning stage, or does it include the installation of the operator and kustomizisation of the cloud provider?

[katexochen]

are you just factoring the cluster creation into the provisioning stage, or does it include the installation of the operator and kustomizisation of the cloud provider?

I'm only talking about infrastructure here. So no kubectl steps are done within terraform.

[stevenhorsman]

are you just factoring the cluster creation into the provisioning stage, or does it include the installation of the operator and kustomizisation of the cloud provider?

I'm only talking about infrastructure here. So no kubectl steps are done within terraform.

Ok, the operator install is in the provisioner code at the moment, so I wanted to check.

It would be a bit frustrating to revert the changes that we did at the beginning of the year to switch the ibm cloud provisioning back from code to terraform, but that's not a reason to not do it.

[katexochen]

It would be a bit frustrating to revert the changes that we did at the beginning of the year to switch the ibm cloud provisioning back from code to terraform, but that's not a reason to not do it.

What was the reason for the switch?

[stevenhorsman]

What was the reason for the switch?

We had the terraform scripts to set up our cluster (and do the provisioning of the CAA) from when we first contributed the code to open-source and no-one else was using it and the e2e framework was the 'strategic' approach for testing and included the intention to do all the provisioning there, so we switched over in order to limit the inconsistency between using the ibm cloud provider and other providers.

[katexochen]

We had the terraform scripts to set up our cluster (and do the provisioning of the CAA) from when we first contributed the code to open-source and no-one else was using it and the e2e framework was the 'strategic' approach for testing and included the intention to do all the provisioning there, so we switched over in order to limit the inconsistency between using the ibm cloud provider and other providers.

That sounds more like an organizational issue than a technical reason not to use terraform. I think the benefit of sharing tf code between tests, devs and users would be huge.

[stevenhorsman]

That sounds more like an organizational issue than a technical reason not to use terraform. I think the benefit of sharing tf code between tests, devs and users would be huge.

I agree - hence the comment "It would be a bit frustrating to revert the changes ..., but that's not a reason to not do it." I think that the only thing that has changed is that if/when we do this work we should switch to use OpenTofu to avoid and terraform/hashicorp license issues.

[tumberino]

For IBM terraform provider there are a large set of examples that we could direct people to for setting up a basic cluster? https://github.com/terraform-ibm-modules/terraform-ibm-cluster/tree/master/examples

In terms of the CI using terraform/opentofu this would be my prefered option and would be easier to maintain than the provisioner code long term but I don't think we should make a solution that is highly customizable for creating clusters for the end users. We should direct our users to the cloud providers documentation for creating custom clusters, perhaps with some beginner friendly suggestions.

[katexochen]

but I don't think we should make a solution that is highly customizable for creating clusters for the end users.

@jtumber-ibm Just to clarify, my suggestion wasn't to create anything customizable. I think terraform code is customizable per definition, devs and users can just add another resource or so locally if needed. Our repository would contain a single ways to deploy, through, with as less configuration as possible.

We made good experience with this in Constellation, where our infra requirements are a bit more complex. We provide code to deploy the basic infrastructure, and customers can adapt it to their needs. It is much easier and faster than setting up the whole thing yourself.

[mkulke]

I would also like to see infra provisioning untangled from running the tests. CAA + operator are provisioned implicitly as part of the test suite anyway.