search

gusztavvargadr / packer

Packer templates for building Windows and Ubuntu images
https://app.vagrantup.com/gusztavvargadr
The Unlicense
452 stars 66 forks source link

Document build usage #14

Open gusztavvargadr opened 7 years ago

gusztavvargadr commented 7 years ago
rgl commented 4 years ago

Can you explain how you publish the boxes to https://app.vagrantup.com/gusztavvargadr/? I'm quite curious, because you publish a lot of boxes, and I wonder how you do that! Do you have an home/internal CI/CD cluster of windows machines where you build all of this? Like, do you build them all at once, one at a time? Also, it seems there some azure devops ci/cd scripts at https://github.com/gusztavvargadr/packer/tree/master/src/core/azp, do you actually use azure devops to build this?

gusztavvargadr commented 4 years ago

@rgl indeed, I'm using Azure DevOps for orchestrating the process, see all the builds at https://dev.azure.com/gusztavvargadr/packer/_build?view=folders.

I have some home machines indeed for testing purposes, and use mainly Packet for building the boxes, creating custom agents for Azure DevOps on bare metal.

For the actual publishing I use the Vagrant CLI commands.

You can take a look at the src/core/ folder for all the components. The Packet setup is not yet fully automated, involves a couple of manual steps for provisioning. This is my repo I use for creating the phyisical machines with Terraform https://github.com/gusztavvargadr/packet.

I hope that answers some questions at least, please let me know if there's anything else I can help with.

rgl commented 4 years ago

Awesome Gusztáv! Thanks for the explanation, it really helped to understand all of this! :-)

From what I could tell from looking at the run output at https://dev.azure.com/gusztavvargadr/packer/_build/results?buildId=3011&view=logs&j=2831d33e-84dd-5533-eda0-8105954f1bc8 there's a machine/agent called packet-hyperv-0-1, where are you specifying these names? I didn't find anything with hyperv in your packet repo.

Reading https://github.com/gusztavvargadr/packet/blob/51e0feb770730aac4f18af3b234d1399982b34c8/samples/device-windows/main.tf#L13-L19 and https://github.com/gusztavvargadr/packet/blob/51e0feb770730aac4f18af3b234d1399982b34c8/samples/device-linux/main.tf#L13-L19 you are using at least two t1.small.x86 packet machines, one for windows and another for linux. But I didn't find where you are putting the number of instances of those machines, is that stored in your computer? Are they really one of each?

Are all the packet nodes always running? Are you paying all of this from your own pocket (I'm just curious on how much money does it take to run something like this "on the cloud")? Or does packet have something free for open-source project?

gusztavvargadr commented 4 years ago

@rgl yeah, I'm publishing some Docker images to https://hub.docker.com/u/gusztavvargadr based on the Packet repo, then reusing them here. I use c1.small.x86 for the builds, the t1.small.x86 is not too performant.

The agent names you see in the build output I enter for now manually when provisioning the agent. I first do some tools installation when the Packet machines come online, then configure the agent as described at https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops.

I pay now these from my own pocket, yes, roughly $50 a month for building all the images, but you might see, it's a lot of them, only building a couple of these would be significantly cheaper (roughly half the price is the compute resource cost, and a significant part is the outgoing bandwidth, it's several hundreds of GBs per "iteration").

My plan is to check for other cloud providers later, e.g. Hyper-V can be nested, so does not need bare metal necessarily, but I was not able to make it with e.g. VirtualBox yet, although they seem to support it now for AMD processors.

rgl commented 4 years ago

Woah, the c1.small.x86 is much better. Is 120G of disk enough thou? Anyways, somehow, you make it work!

If I understood correctly, that packet machine is running windows 2016, then you use packer to launch hyper-v and virtualbox VMs?

My math is falling me, that vm type is $0.4h*730hoursmonth~= $292month even without the network bandwidth and windows license, how are you just paying $50 for everything?

BTW, packet has some kind of support for open-source projects at https://www.packet.com/community/open-source/ :-)

Again, thanks for all the explanations :-)

gusztavvargadr commented 4 years ago

@rgl I use c1.small.x86 or x1.small.x86 depending on the disk needs - cleaning up after each build to save space. Indeed, building the bigger images like the ones with Visual Studio require a lot of space.

I sometimes create multiple Azure DevOps agents per machine to use the most of the power available (one VM build does not keep that machine busy :) Hyper-V works really well with multiple virtual machines, but I noticed issue with running several VMs in parallel for VirtualBox with Packer. For now I use Windows Server 2016 indeed, but I'm experimenting with Linux for VirtualBox and soon for VMware.

Sometimes I launch several Packet machines in parallel, resulting in 4-8 agents, depending on what do I need to scale for - you cannot run VirtualBox with Hyper-V being active, so I usually have multiple Packet hosts.

Yeah, maybe I forgot to add, I don't run these Packet hosts all the time, only on demand, so I don't pay a full price whole month. For testing purposes I use my machines at home, I launch Packet hosts when I want to publish new boxes (the bandwidth there is way better than my home connection :)

Thanks for the tip about the open source projects, I will give it a try :)

rgl commented 4 years ago

Thank you very much for explaining the current workflow in more detail, my curiosity is now satisfied! :-)

The latest versions of VirtualBox mention they can now run under hyper-v (although they say the VMs run a bit slower) but I'm not sure if they can be run with the regular VirtualBox, i.e., if a box build in VB over hyperv can be run by the regular VB. Please note that I didn't try that combination, nor did I try hyper-v with packer/vagrant.

I had success using VirtualBox/libvirt/kvm/vmware-vsphere in a ubuntu 18.04 host to launch windows machines, if you need some help, feel free to ping me :-)

BTW, packet at https://www.packet.com/bare-metal/ say then can give us a machine under one minute, is that really true?

gusztavvargadr commented 4 years ago

@rgl yes, thanks for the tips, I will keep experimenting and will reach out. I tried VirtualBox with Hyper-V once but did not succeed, I think I'd go for Linux instead. If you have known setups that work, please share, I'd appreciate it :)

Packet can boot some Linux distros in a minute, yes, for Windows it's usually 10+ minutes :)

rgl commented 4 years ago

I have some packer/vagrant environments in my github profile, but they are not using any kind of configuration management tools like yours. they all use plain bash (which for learning I find much easier/straightforward to follow).

For example, https://github.com/rgl/xfce-desktop-vagrant is a bare-bones vagrant environment that is a ubuntu 18.04 xfce desktop. that should show how to install the virtualization tools (and if run under nested virtualization you can also try libvirt/kvm besides virtualbox) and you can use it for launching other packer/vagrant environments under linux.

Inside that environment you could launch my other environments, which can use libvirt/virtualbox/vsphere (vsphere needs an external vsphere infrastructure):

https://github.com/rgl/ubuntu-vagrant https://github.com/rgl/windows-vagrant

If you have any comments, please let me known.

gusztavvargadr commented 4 years ago

@rgl thanks mate, I will take a look and will get back to you, appreciate you sharing all these details.