Allow vagrant-cachier to support offline provisioning of apt packages

[lemoncurd]

Hi,

I have made some small changes to allow vagrant-cachier to support provisioning a debian box while offline. This is particularly useful when developing on the train etc. Hopefully you will find this pull request useful.

According to Wikipedia /var/lib/apt/lists is the "storage area for state information for each package resource specified in sources.list". I have added this directory to the cached directories in the apt bucket. This enables apt to install cached packages without hitting the remote repositories for the main package lists.

[lemoncurd]

Just to add to this, I have tested the Yum bucket on a CentOS machine and the current implementation works without any changes allowing me to provision a machine while offline.

[fgrehm]

Thanks, I'll try to give this a go before releasing 0.6.0.

[fgrehm]

@lemoncurd I did some quick tests on a Ubuntu box and it seems to not have much effect whether I have the /var/lib/apt/lists cached or not when the current cache dir has been warmed up. I can install the packages on a Precise box when I'm offline and the time to provision the VM lowered about 2 seconds only but :-( (YMMV)

Maybe I missed something on my tests but Is this a Debian only thing? I'm done with the things I wanted to have in place for 0.6.0 and this is the last feature that I'm willing to bring in =)

[lemoncurd]

thanks for the response @fgrehm!

I took another look using Ubuntu (and debian using the puppet labs 73 image) this morning. For me, not having /var/lib/apt/lists cached prevents an offline provision at all.

I have included my Vagrantfile below. I create a fresh instance each time using vagrant destroy && time vagrant up. Note, the performance timings are so skewed because I am connected over an intermittent tethered 3g connection. When I move to offline mode, I turn off Wi-Fi at the OS level.

Without caching /var/lib/apt/lists:

Connection	Cache	Time
Online	Cold	13m45s (!)
Online	Warm	1m28s
Online	Warm	2m1s
Offline	Warm	fail

When the offline the provision fails attempting to install openjdk. , it reports the following error:

...
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/precise-updates/Release.gpg  Could not resolve 'us.archive.ubuntu.com'
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/precise-backports/Release.gpg  Could not resolve 'us.archive.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

and then fails with the following errors...

...
Failed to fetch http://us.archive.ubuntu.com/ubuntu/pool/main/i/icedtea-web/icedtea-netx-common_1.2-2ubuntu1.2_all.deb  Could not resolve 'us.archive.ubuntu.com'
Failed to fetch http://us.archive.ubuntu.com/ubuntu/pool/main/i/icedtea-web/icedtea-netx_1.2-2ubuntu1.2_amd64.deb  Could not resolve 'us.archive.ubuntu.com'
Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/o/openjdk-6/openjdk-6-jdk_6b24-1.11.4-1ubuntu0.12.04.1_amd64.deb  Could not resolve 'security.ubuntu.com'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Caching: /var/lib/apt/lists:

Connection	Cache	Time
Online	Warm	1m36s
Online	Warm	1m46s
Offline	Warm	1m6s
Offline	Warm	1m6s

In this case, the offline provision works without any errors.

Vagrantfile:

$provision = <<SCRIPT
apt-get update || true
apt-get -y install openjdk-6-jdk
SCRIPT

Vagrant.configure("2") do |config|
    config.vm.box = "precise64"
    config.vm.box_url = "http://files.vagrantup.com/precise64.box"

    if Vagrant.has_plugin?("vagrant-cachier")
        config.cache.auto_detect = true
    end

    config.vm.define :vagrant do |vagrant|
        vagrant.vm.hostname = "vagrant"
        vagrant.vm.network :private_network, ip: "192.168.50.2"
        vagrant.vm.provision :shell, inline: $provision
    end
end

[fgrehm]

Thanks for all the information! I'll try to bring this in tonight :-)

Just so you know, I'll probably do a couple changes on top of this patch in order to make things backwards compatible for those who have cached packages around. This patch "as is" will end up dropping packages on a new folder (cache/apt/archives instead of cache/apt) and the previously downloaded packages won't be picked up during provisioning, so files will end up being duplicated eating more disk space.

I'll keep you posted!

[fgrehm]

@lemoncurd thanks for the PR and sorry for not merging it but I ended up doing things in a different way for the reasons explained above. The code is available on a branch called next and will be released as soon as I have some time to do some extra testing on it as it involves a big refactoring on the codebase :-)

For more information about the release please check GH-79 and fell free to build and install the plugin from sources and let us know how it goes ;)

[oker1]

This is great, now the plugin is on par with using a real apt-cacher, but slightly more convenient!