search

xorpaul / g10k

my r10k fork in Go
Apache License 2.0
124 stars 50 forks source link

Puppetfile mode: allow configuration of cache directories #10

Closed mikey179 closed 8 years ago

mikey179 commented 8 years ago

When using g10k -puppetfile inside a Docker container the final copy from /tmp to module dir causes problems when both are on different devices, e.g. because the module dir is mounted into the Docker container. This could be solved when the local forge directory could be influenced via configuration or command line argument.

For reference, here is the output of a g10k -puppetfile run in such a container:

root@c748758fac38:~/environments# g10k -puppetfile -verbose
2016/01/26 10:25:47 Executing git --git-dir /tmp/https-__github.com_dalenys_puppet-supervisor.git remote update --prune took 0.71367s
2016/01/26 10:25:47 Executing git --git-dir /tmp/https-__github.com_xp-framework_puppet-xp-runners.git remote update --prune took 0.72577s
2016/01/26 10:25:47 Executing git --git-dir /tmp/https-__github.com_mayflower_puppet-php.git remote update --prune took 0.81712s
2016/01/26 10:25:47 Executing git --git-dir /tmp/git@puppet-repo.server.lan-core_modules_core_sshd.git remote update --prune took 0.93008s
2016/01/26 10:25:47 Executing git --git-dir /tmp/git@puppet-repo.server.lan-core_modules_core_facts.git remote update --prune took 1.14196s
2016/01/26 10:25:47 Executing git --git-dir /tmp/git@puppet-repo.server.lan-core_modules_core_osum.git remote update --prune took 1.29663s
2016/01/26 10:25:47 Executing git --git-dir /tmp/git@puppet-repo.server.lan-core_modules_core_syslog.git remote update --prune took 1.32702s
2016/01/26 10:25:47 Executing git --git-dir /tmp/git@puppet-repo.server.lan-bit_installapp.git remote update --prune took 1.34041s
Need to sync .//external_modules/ntp/
Need to sync .//external_modules/postfix/
Need to sync .//external_modules/apt/
Need to sync .//external_modules/inifile/
Need to sync .//external_modules/stdlib/
2016/01/26 10:25:48 Executing git --git-dir /tmp/https-__github.com_mayflower_puppet-php.git log -n1 --pretty=format:%H 3.4.2 took 0.04670s
2016/01/26 10:25:48 Executing git --git-dir /tmp/git@puppet-repo.server.lan-core_modules_core_facts.git log -n1 --pretty=format:%H master took 0.04681s
Need to sync .//external_modules//php
Need to sync .//external_modules/concat/
2016/01/26 10:25:48 Executing git --git-dir /tmp/git@puppet-repo.server.lan-core_modules_core_syslog.git log -n1 --pretty=format:%H master took 0.04797s
2016/01/26 10:25:48 Executing git --git-dir /tmp/git@puppet-repo.server.lan-core_modules_core_osum.git log -n1 --pretty=format:%H master took 0.04800s
2016/01/26 10:25:48 Executing git --git-dir /tmp/https-__github.com_dalenys_puppet-supervisor.git log -n1 --pretty=format:%H master took 0.05688s
2016/01/26 10:25:48 Executing git --git-dir /tmp/git@puppet-repo.server.lan-bit_installapp.git log -n1 --pretty=format:%H master took 0.05132s
2016/01/26 10:25:48 Executing git --git-dir /tmp/https-__github.com_xp-framework_puppet-xp-runners.git log -n1 --pretty=format:%H master took 0.05062s
2016/01/26 10:25:48 Executing git --git-dir /tmp/git@puppet-repo.server.lan-core_modules_core_sshd.git log -n1 --pretty=format:%H master took 0.06124s
2016/01/26 10:25:48 Executing cp --link --archive /tmp/puppetlabs-ntp-4.1.0/* .//external_modules/ntp/ took 0.70185s
2016/01/26 10:25:48 Failed to execute command:  cp --link --archive /tmp/puppetlabs-ntp-4.1.0/* .//external_modules/ntp/  Output:  cp: cannot create hard link './/external_modules/ntp/CHANGELOG.md' to '/tmp/puppetlabs-ntp-4.1.0/CHANGELOG.md': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/CONTRIBUTING.md' to '/tmp/puppetlabs-ntp-4.1.0/CONTRIBUTING.md': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/Gemfile' to '/tmp/puppetlabs-ntp-4.1.0/Gemfile': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/LICENSE' to '/tmp/puppetlabs-ntp-4.1.0/LICENSE': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/README.markdown' to '/tmp/puppetlabs-ntp-4.1.0/README.markdown': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/Rakefile' to '/tmp/puppetlabs-ntp-4.1.0/Rakefile': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/checksums.json' to '/tmp/puppetlabs-ntp-4.1.0/checksums.json': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/lib/puppet/parser/functions/ntp_dirname.rb' to '/tmp/puppetlabs-ntp-4.1.0/lib/puppet/parser/functions/ntp_dirname.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/manifests/config.pp' to '/tmp/puppetlabs-ntp-4.1.0/manifests/config.pp': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/manifests/init.pp' to '/tmp/puppetlabs-ntp-4.1.0/manifests/init.pp': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/manifests/install.pp' to '/tmp/puppetlabs-ntp-4.1.0/manifests/install.pp': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/manifests/params.pp' to '/tmp/puppetlabs-ntp-4.1.0/manifests/params.pp': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/manifests/service.pp' to '/tmp/puppetlabs-ntp-4.1.0/manifests/service.pp': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/metadata.json' to '/tmp/puppetlabs-ntp-4.1.0/metadata.json': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/class_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/class_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/disable_monitoring_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/disable_monitoring_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/centos-59-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/centos-59-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/centos-64-x64-pe.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/centos-64-x64-pe.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/centos-64-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/centos-64-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/centos-65-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/centos-65-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/default.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/default.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/fedora-18-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/fedora-18-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/fedora-21-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/fedora-21-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/sles-11-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/sles-11-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/sles-12-64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/sles-12-64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/ubuntu-server-10044-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/ubuntu-server-10044-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/ubuntu-server-12042-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/ubuntu-server-12042-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/ubuntu-server-1404-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/ubuntu-server-1404-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/ubuntu-server-14042-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/ubuntu-server-14042-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/ntp_config_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/ntp_config_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/ntp_install_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/ntp_install_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/ntp_parameters_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/ntp_parameters_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/ntp_service_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/ntp_service_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/preferred_servers_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/preferred_servers_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/restrict_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/restrict_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/unsupported_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/unsupported_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/classes/ntp_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/classes/ntp_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/spec.opts' to '/tmp/puppetlabs-ntp-4.1.0/spec/spec.opts': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/spec_helper.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/spec_helper.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/spec_helper_acceptance.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/spec_helper_acceptance.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/unit/puppet/provider/README.markdown' to '/tmp/puppetlabs-ntp-4.1.0/spec/unit/puppet/provider/README.markdown': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/unit/puppet/type/README.markdown' to '/tmp/puppetlabs-ntp-4.1.0/spec/unit/puppet/type/README.markdown': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/templates/ntp.conf.erb' to '/tmp/puppetlabs-ntp-4.1.0/templates/ntp.conf.erb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/tests/init.pp' to '/tmp/puppetlabs-ntp-4.1.0/tests/init.pp': Invalid cross-device link

2016/01/26 10:25:48 syncForgeToModuleDir(): Error while trying to hardlink /tmp/puppetlabs-ntp-4.1.0/ to .//external_modules/ntp/ :exit status 1
xorpaul commented 8 years ago

I've added an environment variable called g10k_cachedir, which overwrites the default /tmp/ directory for all cache directories.

https://github.com/xorpaul/g10k/commit/38b7da514c7715df4ab1292984c0c5b3f494ef94

You can test it here: https://github.com/xorpaul/g10k/releases/tag/v0.1.1

Using

$ export g10k_cachedir=/tmp/foobar ; g10k -puppetfile -debug
...
2016/02/08 14:57:23 DEBUG Using as cachedir environment variable g10k_cachedir: /tmp/foobar/
...

I will add a parameter to the puppetfile mode that will do the same, but Docker mainly uses environment variables to solve stuff like this.

mikey179 commented 8 years ago

Tested this, changing the cache dir with the environment variable works nicely. Unfortunately, it doesn't solve the problem with the hard link inside the container. I tried to use a cache directory on the same volume as external_modules, but even then I get this error. I suspect it has to do with the underlying file system stuff of Docker, and can't be solved directly in g10k. Of course, being able to change the cache dir is still a useful option.

I suspect the only way to make this error go away is not to hardlink when copying the modules from cache dir to target dir, but I don't know what the consequences for g10k would be.

xorpaul commented 8 years ago

Okay, I added a new parameter -docker availabe for testing here https://github.com/xorpaul/g10k/releases/tag/v0.1.3 (source https://github.com/xorpaul/g10k/commit/7f00124261604bdc88973fe6a0fb898821b04220)

$ g10k -puppetfile -docker

This uses mv instead of cp --link which should work in your Docker environment. This also purges the Forge cache directory after each run, which isn't important for a Docker environment, but it frees up disk space and is necessary that a following non puppetfile run without the -docker parameter works.

mikey179 commented 8 years ago

Maybe the option shouldn't be called -docker, as it must be used on OS X as well. Detected this while working on #11 - OS X' cp command doesn't know the --link and --archive options (see man).

Other than that it seems to work fine.

xorpaul commented 8 years ago

Okay, I renamed it to -usemove