Allow include/exclude for file provisioner

[steviecoaster]

It seems #1811 was closed as won't fix for....no other reason than it would be confusing.

This is a misstep as having that functionality would severely improve the usability of the file provisioner, without having to have multiple steps or convoluted shell/powershell code to clean up after it.

Please consider adding this feature.

[vexx32]

Aside-meta-note: closing issues as "wontfix" with a version target is fundamentally confusing and sends mixed messages. By all means, target for whatever future version, but closing it is a pretty clear "not gonna happen".

But yeah, please provide for this. I'd like to be able to upload the contents of a git repo without having to wait half an hour for it to slowly upload the .git hidden folder and then have to delete it after the fact. This is a basic useability feature, I don't know how y'all failed to include it in the initial implementation.

[azr]

Hello there, thanks for opening; I think this should be solved with the hcl2 fileset function !

[SwampDragons]

I think you're right that the rationale for having closed the original issue was weak. This would be a nice UI improvement.

Probably the right way to address this in the provisioner itself rather than the hcl2 fileset function would be to change the UploadDir call here https://github.com/hashicorp/packer/blob/master/provisioner/file/provisioner.go#L180 to a directory-walking function that checks against an "exlcude" list that the user can pass in as an option to the provisioner.

Since it does look like HCL provides a built-in solution for this, I probably won't direct the Packer team to work on it, but I'd be happy to review a community PR that does this.

[NoahAndrews]

Maybe I'm missing something, but I don't see any way to exclude specific files or directories using HCL2's fileset. I'd be happy to switch to HCL2 if it can provide an actual solution.

[azr]

Hello there, yes sorry, my previous message was rather short.

The HCL fileset function allows to specifically and recursively select files with a glob pattern. So it's only 'include' but it can be exclusive depending on how you call it, so to take the docs page example, if you have:

> tree pkr-consul
pkr-consul
├── build-linux.pkr.hcl
└── linux
    ├── files
    │   ├── hello.txt
    │   └── world.txt
    └── scripts
        ├── README.txt
        ├── script.bash
        ├── script-1-install.sh
        └── script-2-setup.sh

And you only want the .txt suffixed files, you can do:

$ echo 'fileset("**", "*.txt")' | packer console pkr-consul
[
  "../files/hello.txt",
  "../files/world.txt",
  "../linux/scripts/README.txt",
]

or only .sh suffixed files in linux/scripts:

$ echo 'fileset("linux/scripts", "*.sh")' | packer console pkr-consul
[
  "script-1-install.sh",
  "script-2-setup.sh",
]

I'd advise on sorting the fileset after this because fileset will not sort anything ( so depending on how your filesystem works this might differ).

packer console will help you find the best solution.

If that does not help you to solve your problem; please describe your issue a bit more so that we can help you better 🙂

[micchickenburger]

I'm wondering if anyone has tried to tackle this yet. I'm administering a Rails app that has a bunch of stuff I need to exclude and in my case I can't delete the excess after the fact. I'm not sure if I will have time to tackle this issue anytime soon, but I am motivated to see it solved.

[SwampDragons]

I don't think anyone has, but I'd be interested to know if the fileset function solves this for you.

[NoahAndrews]

Just for the record, I didn't respond because I ended up not using packer at all.

[NoahAndrews]

A docker-based solution

[enykeev]

I don't think anyone has, but I'd be interested to know if the fileset function solves this for you.

It sadly does not. I've removed my previous message where I thought I solved the problem with this snippet

  provisioner "file" {
    sources = [for s in fileset("source", "**") : s if !can(regex("/node_modules/|/dist/|/.git/|/data/", "./${s}"))]
    destination = "/home/ec2-user/target/"
  }

But I quickly discovered that doing it this way results in just a pile of files in target directory with no directory structure being kept. (I guess) You can potentially create a number of provisioners for every subdirectory, but even if it works I would hardly call it a solution.

[msilveirabr]

From the looks of my tree below, you may imagine what I've done: dynamic source block with dynamic file and powershell provisioners blocks....

Some way to control how to ignore files would be great. I'll have to create a powershell to recurse into the target builds to remove files with one-line comment...

windows-server-2019/
├── [ 1.2K]  buildall.sh
├── [ 1.2K]  build.sh
├── [  166]  config
│   ├── [ 1.0K]  build.pkrvars.hcl
│   ├── [  946]  common.pkrvars.hcl
│   ├── [  663]  esxi.pkrvars.hcl
│   ├── [ 1.1K]  strings_en-US.pkr.hcl
│   ├── [ 1.2K]  strings_pt-BR.pkr.hcl
│   └── [  825]  vcenter.pkrvars.hcl
├── [   59]  data
│   ├── [   25]  certificates
│   │   └── [ 7.3K]  root-ca.cer
│   ├── [   20]  drivers
│   │   └── [   19]  pvscsi
│   │       └── [   80]  amd64
│   │           ├── [  10K]  pvscsi.cat
│   │           ├── [ 4.1K]  pvscsi.inf
│   │           ├── [  63K]  pvscsi.sys
│   │           └── [  595]  txtsetup.oem
│   └── [   33]  unattended
│       ├── [   66]  core
│       │   ├── [  15K]  autounattend.pkrtpl.hcl
│       │   └── [ 9.4K]  unattended.pkrtpl.hcl
│       └── [   66]  desktop
│           ├── [  15K]  autounattend.pkrtpl.hcl
│           └── [ 9.4K]  unattended.pkrtpl.hcl
├── [ 1.3K]  debug_build.sh
├── [ 1.7K]  locals.pkr.hcl
├── [    6]  manifests
├── [  230]  NOTES.txt
├── [   44]  scripts
│   ├── [   38]  all
│   │   ├── [  102]  others
│   │   │   ├── [  759]  Extend-Partition.ps1
│   │   │   ├── [  134]  lang
│   │   │   │   ├── [   95]  AdminUsersCustomConfig.ps1
│   │   │   │   ├── [   93]  AllUsersCustomConfig.ps1
│   │   │   │   ├── [ 1.2K]  LocalUserCustomConfig.ps1
│   │   │   │   └── [   91]  UsersCustomConfig.ps1
│   │   │   ├── [   28]  post-sysprep.ps1
│   │   │   └── [   36]  Remove-Store-Taskbar.ps1
│   │   └── [  179]  unattended
│   │       ├── [   34]  custom-activesetup.ps1
│   │       ├── [   87]  lang
│   │       │   ├── [ 1.1K]  custom-activesetup.ps1
│   │       │   ├── [ 1.5K]  windows-init.ps1
│   │       │   └── [ 1.3K]  windows-prepare.ps1
│   │       ├── [  970]  reboot-bw-updates.ps1
│   │       ├── [ 1.8K]  windows-init.ps1
│   │       ├── [ 8.5K]  windows-prepare.ps1
│   │       ├── [ 3.9K]  windows-vmtools.ps1
│   │       └── [ 1.7K]  wmf51-update.ps1
│   ├── [   38]  core
│   │   ├── [  102]  others
│   │   │   ├── [   33]  Extend-Partition.ps1
│   │   │   ├── [  134]  lang
│   │   │   │   ├── [   84]  AdminUsersCustomConfig.ps1
│   │   │   │   ├── [   82]  AllUsersCustomConfig.ps1
│   │   │   │   ├── [   83]  LocalUserCustomConfig.ps1
│   │   │   │   └── [   80]  UsersCustomConfig.ps1
│   │   │   ├── [   28]  post-sysprep.ps1
│   │   │   └── [   37]  Remove-Store-Taskbar.ps1
│   │   └── [   75]  unattended
│   │       ├── [   35]  custom-activesetup.ps1
│   │       ├── [   33]  lang
│   │       │   └── [   33]  windows-prepare.ps1
│   │       └── [   32]  windows-prepare.ps1
│   └── [   38]  desktop
│       ├── [  102]  others
│       │   ├── [   33]  Extend-Partition.ps1
│       │   ├── [  171]  lang
│       │   │   ├── [  679]  AdminUsersCustomConfig.ps1
│       │   │   ├── [   85]  AllUsersCustomConfig copy.ps1
│       │   │   ├── [  980]  AllUsersCustomConfig.ps1
│       │   │   ├── [   86]  LocalUserCustomConfig.ps1
│       │   │   └── [   83]  UsersCustomConfig.ps1
│       │   ├── [   28]  post-sysprep.ps1
│       │   └── [ 2.9K]  Remove-Store-Taskbar.ps1
│       └── [   75]  unattended
│           ├── [   38]  custom-activesetup.ps1
│           ├── [   33]  lang
│           │   └── [   36]  windows-prepare.ps1
│           └── [   35]  windows-prepare.ps1
├── [   28]  strings.pkr.hcl -> config/strings_pt-BR.pkr.hcl
├── [ 1.2K]  validate.sh
├── [  11K]  variables.pkr.hcl
├── [ 4.0K]  windows-server.auto.pkrvars.hcl
└── [  16K]  windows-server.pkr.hcl

[SemiConscious]

I know this is ancient, but in case anyone else finds their way here looking for answers, I ended up using a gnarly construct like this:

# packer hcl file
...

locals {
  # where the files are on your local drive
  srcprefix = "..."
  # where to put them in the packer instance
  dstprefix = "..."
  # list of directories and filenames to ignore
  ignore = convert(["node_modules", "build", "dist"], set(string))
  # build map of file=>dir with the files we want removed from the list.
  # to explain the `if`: `setintersection` is used to check if any of the path segments in the file name
  # match directories or filenames we want to ignore. if the length of the result `is > 0`, discard 
  # the file. conversion to set(string) is needed as packer doesn't want to do the conversion itself.
  filetodir = {
    for file in setunion(
      fileset(".", "${local.srcprefix}/foo/**"), 
      fileset(".", "${local.srcprefix}/bar/**"), 
    ): file => dirname(file) if length(setintersection(convert(split("/", file), set(string)), local.ignore)) == 0 
  }
  # build map of destination dir => [ source files ]
  uploads = { 
    for dir in distinct(values(local.filetodir)): 
      replace(dir, local.srcprefix, local.dstprefix) => [for f,d in local.filetodir: f if d == dir ] 
  }
}

build {

...
  # precreate all needed directories
  provisioner shell {
    inline = [for dir in keys(local.uploads): "mkdir -p ${dir}"]
  }

  # generate a dynamic provisioner for each directory with each directory's source files in a list
  dynamic provisioner {
    for_each = local.uploads
    labels = ["file"]
    content {
      sources = provisioner.value
      destination = "${provisioner.key}/"
    }
  }

...
}

Hope this helps someone