Support Centralized Provider Configurations

bloudraak commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions; they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Assume you're building many templates for your organization or a large build farm consisting of several operating systems, architectures, and configurations. As a result, you end up with many different Packer templates. Some provider-specific information, like how to authenticate, which hosts to use, and whatnot, is being duplicated everywhere. You could define a local or variable and reuse that, but the extra boilerplate adds noise to the templates.

This feature request is to be able to centralize the configuration of a provider and reuse that. The provider can be seen as having defaults that sources and builders can use if nothing is specified but perhaps be overwritten in the builder and sources themselves.

Use Case(s)

You tend to duplicate provider-specific information when you have numerous packer templates using VMware vSphere (as an example). It just adds toil and unnecessary boilerplate. With this feature request, we can reduce the boilerplate required by Packer and reuse the configuration.

Potential configuration

Taken from Terraform

provider "vsphere" {
  host                                 = var.vsphere_host
  insecure_connection     = true
  username                       = var.vcenter_username
  password                       = var.vcenter_password
  vcenter_server              = var.vcenter_server
  folder                             = var.vcenter_folder
  datastore                      = var.vsphere_datastore
}

source "vsphere-clone" "ubuntu-2204" {
  // will use the single vSphere provider defined earlier
}

source "vsphere-clone" "ubuntu-2004" {
  // will use the single vSphere provider defined earlier
}

source "vsphere-clone" "rhel7" {
  // will use the single vSphere provider defined earlier
}

source "vsphere-clone" "rhel9" {
  // will use the single vSphere provider defined earlier
}

Or, in the case where you need to support multiple isolated installations of VMware vSphere (as an example):

provider "vsphere" {
  alias  = "oak02"
  // oak02 specific information
}

provider "vsphere" {
  alias  = "sjc01"
  // sjc01 specific information
}

source "vsphere-clone" "ubuntu-2204-sjc01" {
  provider = vsphere.sjc01
  // will use the SJC01 vSphere provider defined earlier
}

source "vsphere-clone" "ubuntu-2204-oak02" {
  provider = vsphere.oak02
  // will use the OAK02 vSphere provider defined earlier
}

Something Terraform does not support but would be somewhat helpful is something as follows:

provider "vsphere" {
  alias  = "oak02"
  // oak02 specific information
}

provider "vsphere" {
  alias  = "sjc01"
  // sjc01 specific information
}

source "vsphere-clone" "ubuntu-2204" {
  providers = [vsphere.sjc01, vsphere.oak02]
  // This is essentially doing a foreach provider. 
}

I'm using VMware vSphere purely for illustrative examples. It also applies to other providers.

Potential References

Provider Configuration

nywilken commented 2 years ago

Hi @bloudraak thanks for reaching out. The HCL configuration language while similar to Terraform's does have a different syntax and does not currently support the ability for things like aliases. TBH we have some work to do before we can get to this point. Terraform's HCL usage has matured nicely over the years.

That said, I do wonder if you had a chance to look at Packer's source reference syntax, which may help in this case.

Taking your examples from above I've provided a few options for sharing the source block configuration.

Example 1: Single Source Reused

```hcl source "vsphere-clone" "base" { host = var.vsphere_host insecure_connection = true username = var.vcenter_username password = var.vcenter_password vcenter_server = var.vcenter_server folder = var.vcenter_folder datastore = var.vsphere_datastore } build { source "source.vsphere.base" { name = "ubuntu-2204" template = "path/to/template" // will use the single vSphere configuration defined earlier } source "source.vsphere.base" { name = "ubuntu-2004" template = "path/to/template" // will use the single vSphere configuration defined earlier } source "source.vsphere.base" { name = "rhel7" template = "path/to/rhel7/template" // will use the single vSphere configuration defined earlier } source "source.vsphere.base" { name = "rhel9" template = "path/to/rhel9/template" // will use the single vSphere configuration defined earlier } } ``` In this example you can only set configuration options in the source reference block if they have not been defined in the base source block. For example trying to change the value of `host` would throw an error.

Example 2: Multiple Source Blocks

```hcl source "vsphere-clone" "base" { host = var.vsphere_host insecure_connection = true username = var.vcenter_username password = var.vcenter_password vcenter_server = var.vcenter_server folder = var.vcenter_folder datastore = var.vsphere_datastore } source "vsphere-clone" "base2" { host = var.vsphere_host insecure_connection = true username = var.vcenter_username password = var.vcenter_password vcenter_server = var.vcenter_server folder = var.vcenter_folder datastore = var.vsphere_datastore } build { source "source.vsphere.base" { name = "ubuntu-2204" template = "path/to/template" // will use the single vSphere configuration defined earlier } source "source.vsphere.base" { name = "ubuntu-2004" template = "path/to/template" // will use the single vSphere configuration defined earlier } source "source.vsphere.base2" { name = "rhel7" template = "path/to/rhel7/template" // will use the single vSphere configuration defined earlier } source "source.vsphere.base2" { name = "rhel9" template = "path/to/rhel9/template" // will use the single vSphere configuration defined earlier } } ``` Please let me know if this functionality is what solves the issue you are describing above.

Fuco1 commented 2 years ago

I have an azure-arm source with somewhat "complicated" networking setup due to VPNs and such. I want to have this source defined in one file and then "include" it somehow into different builds. So far the only way I found how to reuse this is have all the stuff in one file, name the builds and then use packer build -only worker-big or packer build -only worker-small and so on to run different named builds.

Since each of our images is quite different, we keep them in different directories with all the provisioning bash scripts there. But I can't find a way of including a file from outside the "build" directory into the template. I think that's what's being asked by OP as well.

tenthirtyam commented 6 months ago

But I can't find a way of including a file from outside the "build" directory into the template.

Might a symlink work?

hashicorp / packer