search

hashicorp / packer

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.
http://www.packer.io
Other
15.06k stars 3.32k forks source link

Packer Vagrant Builder File Provisioner Cannot Handle Large Directories #8927

Open eHildy opened 4 years ago

eHildy commented 4 years ago

Packer version

1.5.1

Operating system and Environment details

Windows 10

Overview of the Issue

When using the file provisioner (at least with the Vagrant builder), uploading of large directories fails.

Specifics.

I have a base box that I have built in Packer and successfully used to build a few other boxes on top of. However, when I tried to build the latest one, the file provisioner steps failed. I was able to determine that it is associated with the size of the directory I am trying to upload. In my case, this is a rather large code base that has multiple nested directories. I could perhaps ignore some of them and rebuild them when I finally spin up the VM in Vagrant from the box I'm trying to build, but there's no way to do so in the file provisioner. I could perhaps also create a multitude of file provisioners, explicitly uploading just what I wouldn't be building in the vm, but I'd need a huge amount.

NOTE: I can create a Vagrant box using the same base box I'm trying to build on top off, mount the directory in question and spin up no problem, so the disk is large enough.

Reproduction Steps

I suppose you could create a package.json with a ton of deps in it and npm-install; but in the end what you need is just a large directory with lots of nest dirs and files.

NOTE: the file it shows in the log I can confirm to exist and it finds the other files that live with it. It just gets to a certain point and can upload no more.

Packer Buildfile

{
  "builders": [
    {
      "communicator": "ssh",
      "ssh_username": "{{user `ssh_username`}}",
      "ssh_password": "{{user `ssh_password`}}",
      "ssh_timeout": "1h",
      "source_path": "dd-centos-7-base",
      "provider": "vmware_desktop",
      "type": "vagrant",
      "name": "centos_django",
      "teardown_method": "destroy",
      "skip_add": true
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "echo '{{user `ssh_password`}}' | {{.Vars}} sudo -S -E sh -eux '{{.Path}}'",
      "inline": [
        "mkdir /home/{{user `ssh_username`}}/dd_source",
        "mkdir /home/{{user `ssh_username`}}/dd_source/tools",
        "mkdir /home/{{user `ssh_username`}}/dd_source/<our main app>",
        "chown -R {{user `ssh_username`}}:{{user `ssh_username`}} /home/{{user `ssh_username`}}/"
      ]
    },
    {
      "type": "file",
      "source": "../../../tools/",
      "destination": "/home/{{user `ssh_username`}}/dd_source/tools"
    },
    {
      "type": "file",
      "source": "../../../<our main app>/",
      "destination": "/home/{{user `ssh_username`}}/dd_source/<our main app>"
    },
    {
      "type": "shell",
      "execute_command": "echo '{{user `ssh_password`}}' | {{.Vars}} sudo -S -E sh -eux '{{.Path}}' '{{user `ssh_password`}}'",
      "scripts": [
        "./scripts/configure-server.sh"
      ]
    }
  ],
  "variables": {
    "ssh_username": "<my username>",
    "ssh_password": "<passed in via the CLI>"
  }
}

Packer Log

The log is HUGE, so here's the end of it:

2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: ko
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=105
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: pl
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=93
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: pt
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=89
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: ru
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=101
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=980
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: sw
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=73
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: th
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=226
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: tr
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=90
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: yi
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=109
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: zh-cn
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=92
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: zh-tw
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=92
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading text.js: perms=C0666 size=447
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading throttle.js: perms=C0666 size=1032
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading topic.js: perms=C0666 size=1822
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading when.js: perms=C0666 size=2585
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: _base
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading all.js: perms=C0666 size=359
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading array.js: perms=C0666 size=8498
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading browser.js: perms=C0666 size=301
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading Color.js: perms=C0666 size=1543
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading connect.js: perms=C0666 size=6369
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading declare-strict.js: perms=C0666 size=8953
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading declare.js: perms=C0666 size=15234
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading Deferred.js: perms=C0666 size=7249
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html.js: perms=C0666 size=21105
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading json.js: perms=C0666 size=1974
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading lang.js: perms=C0666 size=10598
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading NodeList.js: perms=C0666 size=35649
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading query.js: perms=C0666 size=331
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading sniff.js: perms=C0666 size=1201
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading window.js: perms=C0666 size=6344
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading xhr.js: perms=C0666 size=9204
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading tests.js: perms=C0666 size=245
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: testsDOH
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading baseonly.js: perms=C0666 size=24
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: data
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading countries.json: perms=C0666 size=483
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading countries_commentFiltered.json: perms=C0666 size=472
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading countries_idcollision.json: perms=C0666 size=473
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading countries_references.json: perms=C0666 size=2516
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading countries_withBoolean.json: perms=C0666 size=645
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading countries_withDates.json: perms=C0666 size=690
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading countries_withNull.json: perms=C0666 size=466
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading countries_withoutid.json: perms=C0666 size=460
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading data_multitype.json: perms=C0666 size=534
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading geography_hierarchy_large.json: perms=C0666 size=1720
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading geography_hierarchy_small.json: perms=C0666 size=530
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading ItemFileReadStore.js: perms=C0666 size=311
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading ItemFileWriteStore.js: perms=C0666 size=57888
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading ObjectStore.js: perms=C0666 size=3944
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading readOnlyItemFileTestTemplates.js: perms=C0666 size=99926
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading reference_integrity.json: perms=C0666 size=872
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading runTests.html: perms=C0666 size=318
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading utils.js: perms=C0666 size=11361
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading data.js: perms=C0666 size=178
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading DeferredList.js: perms=C0666 size=4650
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: dnd
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading dndDefault.css: perms=C0666 size=1404
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading flickr_viewer.html: perms=C0666 size=7188
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: robot
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_dnd.html: perms=C0666 size=3652
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_autoscroll.html: perms=C0666 size=9661
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_box_constraints.html: perms=C0666 size=3934
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_container.html: perms=C0666 size=2189
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_container_markup.html: perms=C0666 size=2293
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_custom_constraints.html: perms=C0666 size=2397
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_dnd.html: perms=C0666 size=10402
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_dnd_handles.html: perms=C0666 size=2904
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_form.html: perms=C0666 size=3105
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_moveable.html: perms=C0666 size=9084
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_moveable_markup.html: perms=C0666 size=8546
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_params.html: perms=C0666 size=3173
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_parent_constraints.html: perms=C0666 size=1723
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_parent_constraints_margins.html: perms=C0666 size=2505
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_selector.html: perms=C0666 size=2788
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_selector_markup.html: perms=C0666 size=2889
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_timed_moveable.html: perms=C0666 size=9361
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: html
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_set.html: perms=C0666 size=15500
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html.js: perms=C0666 size=162
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading i18n.html: perms=C0666 size=2485
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading i18n.js: perms=C0666 size=1826
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading module.js: perms=C0666 size=261
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: nls
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: ar
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=318
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: cs
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=85
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: de
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=87
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: el
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=99
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: en-au
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=71
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: en-us-hawaii
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=71
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: en-us-new_york-brooklyn
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=68
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: en-us-texas
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=71
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: es
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=124
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: fa
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=93
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: fr
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=91
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: he
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=93
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: hi
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=111
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: it
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=87
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: ja
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=105
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: ko
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=105
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: pl
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=93
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: pt
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=89
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: ru
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=101
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=980
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: sw
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=73
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: th
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=226
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: tr
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=90
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: yi
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=109
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: zh-cn
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=92
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: zh-tw
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading salutations.js: perms=C0666 size=92
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading NodeList-data.html: perms=C0666 size=4700
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading NodeList-data.js: perms=C0666 size=171
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: on
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading bench.html: perms=C0666 size=2369
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading connectKey.html: perms=C0666 size=802
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading event-focusin.html: perms=C0666 size=1786
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: resources
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading AMDMixin.js: perms=C0666 size=170
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading amdmodule.js: perms=C0666 size=86
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading AMDWidget.js: perms=C0666 size=148
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading AMDWidget2.js: perms=C0666 size=212
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading AMDWidget3.js: perms=C0666 size=149
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading ApplicationState.js: perms=C0666 size=974
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading JSON.php: perms=C0666 size=32073
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading nodeamd.js: perms=C0666 size=712
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: nodemod
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading m.js: perms=C0666 size=96
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading package.json: perms=C0666 size=38
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading nodemodule.js: perms=C0666 size=114
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading noderequire.js: perms=C0666 size=104
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading noderequireamd.js: perms=C0666 size=185
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading testClass.php: perms=C0666 size=264
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading testClass.smd: perms=C0666 size=523
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_css.html: perms=C0666 size=5947
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_JsonRPCMediator.php: perms=C0666 size=1148
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading yahoo_smd_v1.smd: perms=C0666 size=10643
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading runTests.html: perms=C0666 size=320
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading sniff.html: perms=C0666 size=898
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: store
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading node1.1: perms=C0666 size=165
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading node1.2: perms=C0666 size=63
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading treeTestRoot: perms=C0666 size=419
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test_fx.html: perms=C0666 size=3259
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading TODO: perms=C0666 size=245
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: uacss
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading sniffQuirks.html: perms=C0666 size=2069
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading sniffStandards.html: perms=C0666 size=1917
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading uacss.js: perms=C0666 size=339
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: _base
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading absQuirksIframe.html: perms=C0666 size=432
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading absStrictIframe.html: perms=C0666 size=478
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading eventKeyPress.html: perms=C0666 size=3171
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading eventKeyPressRobot.html: perms=C0666 size=4236
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading eventMouse.html: perms=C0666 size=3341
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading eventMouseRobot.html: perms=C0666 size=4352
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html.html: perms=C0666 size=25991
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html.js: perms=C0666 size=1025
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html_box.html: perms=C0666 size=6320
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html_box_quirks.html: perms=C0666 size=6241
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html_docScroll.html: perms=C0666 size=2040
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html_element.html: perms=C0666 size=9936
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html_id.html: perms=C0666 size=2421
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html_isBodyLtr.html: perms=C0666 size=1742
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html_quirks.html: perms=C0666 size=11240
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading html_rtl.html: perms=C0666 size=5283
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading i18nExhaustive.js: perms=C0666 size=6503
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: loader
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: 14808
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading App.js: perms=C0666 size=241
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading Module.js: perms=C0666 size=294
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading 14808.html: perms=C0666 size=554
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading 8976.html: perms=C0666 size=1242
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading a.js: perms=C0666 size=24
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading addLoadEvents.html: perms=C0666 size=1547
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading afterOnLoad.html: perms=C0666 size=2692
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading amdModule.js: perms=C0666 size=206
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading amdModule1.js: perms=C0666 size=209
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading amdModule2.js: perms=C0666 size=208
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading amdModuleDep.js: perms=C0666 size=196
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading amdModuleDep1.js: perms=C0666 size=198
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading amdModuleDep2.js: perms=C0666 size=198
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading bootstrap.js: perms=C0666 size=4382
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading cdnTest.html: perms=C0666 size=3241
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading cdnTest.js: perms=C0666 size=1689
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: coolio
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading calendar-amd.js: perms=C0666 size=311
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading calendar.js: perms=C0666 size=147
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading calendar1.js: perms=C0666 size=149
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading coolio-built.html: perms=C0666 size=2036
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading coolio-dev-async-with-packageMap.html: perms=C0666 size=2848
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading coolio-dev-async.html: perms=C0666 size=2851
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading coolio-dev-legacy-async-with-packageMap.html: perms=C0666 size=3162
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading coolio-dev-legacy-async.html: perms=C0666 size=3165
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading coolio-dev-legacy.html: perms=C0666 size=1507
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading coolio.profile.js: perms=C0666 size=1032
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading test.html: perms=C0666 size=3648
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading debugConsole.html: perms=C0666 size=1454
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading fastbackTest.html: perms=C0666 size=1153
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading getText.txt: perms=C0666 size=26
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading hostenv_rhino.js: perms=C0666 size=419
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading hostenv_spidermonkey.js: perms=C0666 size=327
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: hostenv_webworkers
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading console.js: perms=C0666 size=792
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading strings.js: perms=C0666 size=39
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading worker1.js: perms=C0666 size=416
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading worker2.js: perms=C0666 size=533
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading worker3.js: perms=C0666 size=561
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading worker4-1.js: perms=C0666 size=523
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading worker4.js: perms=C0666 size=962
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading worker5.js: perms=C0666 size=935
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading worker5.json: perms=C0666 size=31
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading worker6.js: perms=C0666 size=808
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading hostenv_webworkers.js: perms=C0666 size=9699
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: i18n-exhaustive
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: built-i18n-test
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: 152-build
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading legacyModule.js: perms=C0666 size=1644
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: nls
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: ab
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] scp: Uploading legacyBundle.js: perms=C0666 size=96
2020/03/20 12:20:14 packer.exe plugin: [DEBUG] SCP: starting directory upload: ab-cd-ef
2020/03/20 12:20:15 packer.exe plugin: [DEBUG] scp: Uploading legacyBundle.js: perms=C0666 size=138
2020/03/20 12:20:15 packer.exe plugin: [DEBUG] scp: Uploading legacyBundle.js: perms=C0666 size=65
2020/03/20 12:20:15 packer.exe plugin: [DEBUG] SCP: starting directory upload: 152-build-with-layers
2020/03/20 12:20:15 packer.exe plugin: [DEBUG] scp: Uploading legacyModule.js: perms=C0666 size=2061
2020/03/20 12:20:15 packer.exe plugin: [DEBUG] SCP: starting directory upload: nls
2020/03/20 12:20:15 packer.exe plugin: [DEBUG] SCP: starting directory upload: ab
2020/03/20 12:20:15 packer.exe plugin: [DEBUG] scp: Uploading legacyBundle.js: perms=C0666 size=96
2020/03/20 12:20:15 packer.exe plugin: [DEBUG] SCP: starting directory upload: ab-cd-ef
2020/03/20 12:20:15 [INFO] (telemetry) ending file
2020/03/20 12:20:15 ui: ==> centos_django: Provisioning step had errors: Running the cleanup provisioner, if present...
2020/03/20 12:20:15 ui: ==> centos_django: destroying Vagrant box...
2020/03/20 12:20:15 packer.exe plugin: Calling Vagrant CLI: []string{"destroy", "-f", "source"}
2020/03/20 12:20:20 packer.exe plugin: [vagrant driver] stdout: ==> source: Stopping the VMware VM...
2020/03/20 12:20:23 packer.exe plugin: [vagrant driver] stdout: ==> source: Deleting the VM...
2020/03/20 12:20:25 ui: ==> centos_django: Deleting output directory...
2020/03/20 12:20:25 [INFO] (telemetry) ending vagrant
2020/03/20 12:20:25 ui error: Build 'centos_django' errored: open ..\..\..\<our main app>\<our main app>\ESRI\node_modules\dijit\node_modules\dojo\testsDOH\_base\loader\i18n-exhaustive\built-i18n-test\152-build-with-layers\nls\ab-cd-ef\legacyBundle.js: The system cannot find the path specified.
nywilken commented 4 years ago

Thanks for reaching out. This is good to know. When you run the build multiple times does it error on different directories?

The ssh communicator has an sftp option that might be helpful here. You can use sftp instead of scp to copy the directories by adding the following configuration option to the builder settings.

"ssh_file_transfer_method": "sftp"

If that doesn't work a possible workaround could be to zip up the node_modules prior to the build and upload a single file, then use another provisioner to unzip on the guest machine. Or, if possible, run an npm install on the guest machine.

eHildy commented 4 years ago

@nywilken I had hope this was going to work, because it went for longer; but it failed as well.

Building node modules in the guest might be one way; but one of the uses for this beyond local development is building a machine for export to a network that would have to no access to anything npm would pull down. So the idea was to package the machine and export it.

I'll have to experiment later to see if it's only the Vagrant builder that's doing this.

As for "ssh_file_transfer_method": "sftp", I didn't see that in the docs under the Vagrant builder or the File provisioner. Is it listed elsewhere? Could be useful sometime.

SwampDragons commented 4 years ago

It's documented here: https://packer.io/docs/communicators/ssh.html#ssh_file_transfer_method