Instructions to build iso with preloaded images unclear

[Silex]

Hello,

I can't find how to preload my user images in the iso.

http://docs.rancher.com/os/configuration/prepacking-docker-images/ suggest to put the files in /var/lib/rancher/preload/docker, but it's not clear where that should be.

I tried to put myimages.tar.gz in rancher_os_repository/var/lib/rancher/preload/docker/ but that does not work.

Maybe related to #710

RancherOS Version: (ros os version) latest

Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.) baremetal

[joshwget]

Preloading is designed to load images from the root filesystem during boot. Adding images to an ISO is something a bit different. What is your use case?

[Silex]

@joshwget: we are looking to replace our existing system with RancherOS. Our current system is a debian iso with preseeds which just formats the HDD, installs dockers & calls docker load on docker images stored on the DVD, then configures the container to start at boot... so it's a "no questions asked" installation and at the end you have a functional system with our applications running.

We often need to do offline installations (local servers without any internet access).

If modifying the ISO is complicated, maybe the installation could use a local registry that runs on a laptop, that way the images would be "downloaded" from the local laptop... but it's more work than simply taking a DVD to the installation site.

Also, I don't understand how we can put these images on the root filesystem if its going to be formatted when we do ros install?

[SvenDowideit]

yes! I'm thinking of doing something to create a local laptop registry cache :)

At the moment, preload images are tarred up and added to the initrd, which is then put onto the ISO. I'm also doing some things to allow us to add them to the ISO outside the initrd - so perhaps all we really need to do to simplify your use case, is to have a "build custom ISO service image".

[Silex]

@SvenDowideit: okay, so at the moment there's no clear way of achieving it?

I tried modifying the generated tar'ed initrd, but then the CD was not behaving properly (kernel panics)... so I gave up.

[SvenDowideit]

Ah - one reason that happens is because each image you add requires more memory to load from disk - and once loaded, uses up ramdisk - ie, memory.

[joshwget]

Is the goal to pack images into the ISO just so they can be transferred to disk during installation? Or are you actually using the images from the ISO?

If it's the former case, the images could be stored outside of the initrd so they don't use any memory.

[Silex]

@joshwget: just to transfer them to the disk during installation.

Ok, so I just pack them somewhere on the iso, then I use some post-install script to copy them at /var/lib/rancher/preload/docker and reboot?

[joshwget]

Yeah, that should work. It would be cool if we supported something like this automatically from the installer.

[SvenDowideit]

funnily enough - for the installer testing i'm doing, I've added exactly this kind of thing.

[Silex]

@SvenDowideit: anything online I can look at? :wink:

[SvenDowideit]

not quite - I've been hitting all sorts of corner cases during development, and need to fix them, and then clean up

[tylert]

I am also in a situation where I would benefit from being able to pre-pack system and/or user docker images before rolling my own ISO. Some of our containers have the potential of being quite large and not suitable for being crammed inside the initrd/rootfs.

@SvenDowideit do you think any of these changes you're planning would be terribly tricky to back-port to 0.7 (I plan to move our team to use latest RancherOS but can't be sure when this'll happen)? I will likely be doing the same ugly hack that @Silex was proposing (copying tarballs and post-install script) until a better fix is available.

[tylert]

In case this helps others, I have been successful at getting extra stuff crammed onto my custom ISO by adding stuff to scripts/package-iso. (You might also want to change the order of things in scripts/package as well to ensure you can actually pull your newly-built os image onto the same ISO you're building too.)

My first ugly hack to scripts/package-iso looks like this:

...

extra_system_images="
registry/foo/bar:latest
registry/foo/baz:latest
registry/foo/qux:latest
"
extra_user_images="
registry/foo/quux:latest
registry/foo/quuz:latest
registry/foo/corge:latest
"

for image in ${extra_system_images} ${extra_user_images}; do
  if ! docker inspect ${image} > /dev/null 2>&1; then
    docker pull ${image}
  fi
done

mkdir -p ${CD}/images
docker save ${extra_system_images} | gzip > ${CD}/images/extra_system_images.tar.gz
docker save ${extra_user_images} | gzip > ${CD}/images/extra_user_images.tar.gz

# xorrisso/mkisofs stuff follows
...

After you boot from the ISO, you may then simply type mount /dev/cdrom /wherever -t iso9660 and issue a (manual) sudo system-docker load --input /wherever/images/extra_system_images.tar.gz and/or docker load --input /wherever/images/extra_user_images.tar.gz.

I just need to finish up the rest of my hacks to cmd/control/install.go and scripts/installer/lay-down-os (and, possibly, others) once I get them fully working.

I'll try to post the rest of my temporary/horrible workaround back here when I'm done.

[SvenDowideit]

@tylert 👍 :) that's pretty much what i was going to add to #1456 - except I'm going to get the installer to copy the tar files into the preload dir on the newly made disk - and not load them - that can be done by the next boot process (that way we use less memory installing)

[tylert]

@SvenDowideit niiice! Yes, I had also intended to put stuff into /var/lib/rancher/preload/* for the post-install-to-disk-reboot-preload and was experimenting with trying to make the installer.tar "os" image available at after-first-boot-from-iso-but-before-reboot-to-populate-preload so that ros install may be called with -i.

At the moment, my ugly hack mounts the CD fine when you call ros install but I'm still missing a nice way to get lay-down-os to actually "see" my new tarballs so they can be copied into the preload directories (probably a minor "User Too Stupid" error).

I only need to be able to specify a few extra images from our local, private registry that will be available post-install so everything plops down on disc sans network same as @Silex needs--local offline servers (VMs).

Mercifully, I got most of my team switched over to Rancher 0.7.1 this week but a few are still stuck on 0.4.x. Having a self-contained install ISO for them to just fire up with Packer/Vagrant is the last hurdle. I'll try to make some semi-useful comments on your PR ASAFP. Thanks a bunch!

[Silex]

Oookay, I finally have some more time to work on this.

I was able to hack scripts/package-iso just like @tylert showed in a previous comment in order to add my custom images.

@tylert: is https://github.com/SvenDowideit/os/pull/1 somewhat usable? that way I can help you with the hacking :-) also, I will probably need to do these in the future:

• Add a custom cloud-config.yml file on the CD for easier offline installation.
• Add a custom init script on the CD, which will basically do "ros install -f cloud-config.yml". Maybe with some disk formatting beforehand, if I'm not able to do what I want with http://docs.rancher.com/os/storage/state-partition/

Does these ideas make sense? Maybe all these cloud-config.yml and scripts could go in some "extra_cd_files" directory that is also packed on the CD.

[tylert]

@Silex I wouldn't use my PR verbatim. I was merely trying to offer what meager input I could to the main PR (RancherOS, Docker and golang are still very new toys for me). I avoided adding anything in my little stub PR that was already done better by @SvenDowideit 's PR (https://github.com/rancher/os/pull/1456). It is my hope that I can eventually scrap my private junk once my team again makes the jump to a newer release of RancherOS.

In my case, I packed the "os" a.k.a. "installer" image separately onto the ISO (as "extra_install_images.tar.gz") alongside my "extra_system_images.tar.gz" and "extra_user_images.tar.gz" tarballs since I don't need to use/load most of the stuff crammed onto the ISO until after install-to-disk is complete and the system is rebooted and preload pulls everything in for me.

If you need something working now and don't want to wait for the full fix, you might want to import some of the proposed changes here in cmd/control/install.go to mount the ISO and call {system-,}docker load --input /fancypath/extra_install_images.tar.gz if you need to load any extra stuff at install time (I added a "-s" option to ros install ... to specify the device to mount).

[Silex]

@tylert: yeah, I've been playing with your PR, it was semi-working (with a few adjustments, e.g checking ../.extra-user-images is the wrong path (remove the ../)), but then I discovered automatic logins are broken by some commits after 0.8.0-rc2

Since that I'm simply playing with scripts/package-iso on a branch that forks the 0.8.0-rc2 tag, based on the same idea as your commits.

To be honest getting extra files on the ISO is pretty easy, what I have trouble with is adding helper scripts to be used before running ros install on the disk, or simply how to package a cloud-config.yml that can be used without requiring me to mount the CD.

The system I'm trying to replace is dead simple to use, the RancherOS replacement needs to be simple. E.g boot the CD, type "./install" and press enter :-)

[SvenDowideit]

we'll get there soon - I'm having a bad week finding issues in the new installer code - those wonderful details when you hit real hardware :)

[cloud-rocket]

Guys, anything new on this effort? Packing custom / openvmtools images in standalone ISO and copying them to preload is a super important feature.

[cloud-rocket]

@tylert, @SvenDowideit : I used your examples/PR to add custom images (vmware tools) to RancherOS 1.1 ISO. I also modified cmd/control/install.go::layDownOS to copy those images to /var/lib/rancher/preload/system-docker. It works fine while running ros install....

But I still can't find a way to copy the images to /var/lib/rancher/preload/system-docker when creating the host via docker-machine.... Can you give me a clue how to proceed?

thanks!

[tylert]

@cloud-rocket Sorry, I haven't used that stuff for many months now so I'm probably not the best person to ask now. I worked a bit with @SvenDowideit many moons ago on some code changes on top of a pull-request of his but I lost track of these changes after they fell into disuse. Things are probably very different today. Good luck.

[cloud-rocket]

Thanks @tylert, I succeeded in tracking your and @SvenDowideit changes (in his own repo) for version 0.7 and recreated something similar for 1.1. It works great for preloading images during sudo ros install (I can publish my efforts here if it helps).

My problem is that ros install script is not called during docker-machine installation. And I am not sure how the disk layout is created during this process. If you can put some light on it - this will be fantastic.

Thanks

[crochik]

@cloud-rocket @SvenDowideit I see this issue is still open. I am trying to address the exact same scenario ("offline" install with pre-defined docker images in the iso). Has anything changed since the last update? Any "official" way to get it done? Thanks in advance

[niusmallnan]

@crochik Looks like you can track this issue: https://github.com/rancher/os/issues/2697