Resizing data disk is tedious without resize2fs

[kennu]

I wanted to resize my boot2docker-vm.vmdk, because it fills up with Docker images quite quickly. This turned out to be fairly tedious:

• Clone the .vmdk into a .vdi
• Resize the .vdi
• Clone the .vdi back to a .vmdk (maybe not necessary)
• Install and boot a new Ubuntu VM in VirtualBox, attach the new .vmdk to it
• Run fdisk and resize2fs in Ubuntu to resize the partition and filesystem
• Detach and reattach the .vmdk back to boot2docker

The process would be significantly simpler, if:

• Boot2docker included resize2fs, and/or:
• Boot2docker used the .vdi hard disk format, which can be resized

If both were available, the process would be:

• Resize the .vdi and start boot2docker
• Umount disk, run fdisk and resize2fs, remount disk

For reference, I used these instructions to resize the .vmdk: http://stackoverflow.com/questions/11659005/how-to-resize-a-virtualbox-vmdk-file

[GrantGochnauer]

+1

[pheuter]

+1

[SvenDowideit]

I wonder why we used vmdk's @tianon ?

[tianon]

I would guess that it was for compatibility reasons, since VMDKs are more generally and broadly supported than VDIs are, in my experience. If something else makes more sense, I don't see a problem with switching (especially if we don't try to reuse the persistent disk across separate VMs, but I'm not familiar enough with our persistent disk support to say for sure).

[kennu]

It seems like a question of which is the more common use case:

• user moving their data disk from VirtualBox to some other VM platform
• user just wanting to enlarge the data disk

I think quite many users might want to do the latter, so using .vdi would make it easier for them. Anybody can still convert the .vdi to .vmdk if they want to move the data.

According to this question on superuser, .vmdk also supports splitting the data to 2GB files which .vdi does not. But I think it's also a rarer use case and can still be done by converting from .vdi to .vmdk if really needed.

[tianon]

Yeah, I think that sounds reasonable, and I'd be +1 on a switch to VDI.

[yvess]

another use case is if you want to shrink you image, this also needs a VDI image

[tianon]

Anyone want to make a PR for this? ;)

[astubbs]

+1 another help might be able to query for a disk size on install..

[hellonico]

+1, as I am resizing manually right now, and it's not so fun.

[astubbs]

Just start from from scratch and set the default size in your config. I don't think it's worth doing the resize.

On 21/10/2014, at 9:25 pm, Nicolas Modrzyk notifications@github.com wrote:

+1, as I am resizing manually right now, and it's not so fun.

— Reply to this email directly or view it on GitHub.

[pheuter]

I've created a startup script for my team that takes care of automatically resizing and attaching a new HDD to the image, might be of use?

$ VBoxManage clonehd $HOME/VirtualBox\ VMs/boot2docker-vm/boot2docker-vm.vmdk $HOME/VirtualBox\ VMs/boot2docker-vm/boot2docker-vm.vdi --format VDI --variant Standard
$ VBoxManage modifyhd $HOME/VirtualBox\ VMs/boot2docker-vm/boot2docker-vm.vdi --resize $BOOT2DOCKER_IMAGE_SIZE
$ VBoxManage storageattach boot2docker-vm --storagectl "SATA" --device 0 --port 1 --type hdd --medium $HOME/VirtualBox\ VMs/boot2docker-vm/boot2docker-vm.vdi

where BOOT2DOCKER_IMAGE_SIZE is the size in MBs.

[pheuter]

Note: this is done before the initial startup of the image

[astubbs]

Right. But the second time I ran out of space, I just deleted to vm (boot2docker delete), set the default size, and boot2docker init. You just have to redownload all your images. Imo, we should try mounting a dir from OS X into the vm, and use that for image storage, so we don't even have that problem. It at least make it an option.

On 22/10/2014, at 9:28 am, Mark Fayngersh notifications@github.com wrote:

Note: this is done before the initial startup of the image

— Reply to this email directly or view it on GitHub.

[kennu]

If you run a database or a samba server in your Docker containers, it's not "just redownload all your images". You have to preserve the data somehow. That's why resizing the disk is necessary.

Mounting a data disk from OS X would be nice, but likely quite problematic for said use case of running databases, which might do some quite elaborate things with the filesystem.

[astubbs]

Ah yes, good point.

Would like to hear ain't examples or ideas of "might do some quite elaborate things with the filesystem."?

On 22/10/2014, at 10:07 am, Kenneth Falck notifications@github.com wrote:

If you run a database or a samba server in your Docker containers, it's not "just redownload all your images". You have to preserve the data somehow. That's why resizing the disk is necessary.

Mounting a data disk from OS X would be nice, but likely quite problematic for said use case of running databases, which might do some quite elaborate things with the filesystem.

— Reply to this email directly or view it on GitHub.

[kennu]

All databases do locking, which might go badly through a VirtualBox share. MongoDB also does huge mmap()s on the data files. If they actually all work just fine, I'm happy to be proven wrong about this. :-)

[astubbs]

How about just storing images/image cache from repos on the mapped drive from host host? Leave container storage as is. I don't know if that's possible with the architecture though.

On 22/10/2014, at 10:27 am, Kenneth Falck notifications@github.com wrote:

All databases do locking, which might go badly through a VirtualBox share. MongoDB also does huge mmap()s on the data files. If they actually all work just fine, I'm happy to be proven wrong about this. :-)

— Reply to this email directly or view it on GitHub.

[blaggacao]

After deleting an image (to gain space) AND rebooting the b2d (to refresh something about this free space), I got resize2fs successfully into ram with tce-load -wi e2fsprogs.tcz

arguably, a 164 KB tcz would by tiny enough to make up for the annoiance of having to deleta an image an reboot the b2d

btw, hyper-v uses VHDX which is rezisable, if you are on a windows machine and prefer built-in hypervisor (without b2d-cli niceties)

Update: For reference, a brutal online resize worked like a charm resize2fs /dev/sda - made as sudo -s, probaby it isn't even needed to be root

[phpguru]

After following the tedious process my boot2docker vm won't start. I just get a black screen in the VM.

[RomanSaveljev]

@phpguru For me it works better, when I resize with GParted live ISO

[yvess]

I make a new disk image and clone it clonezilla and than delete the old image. Really fast!

[robinbb]

+1 for using resizable disk images with VirtualBox.

[patsissons]

please include resize2fs in the boot2docker iso, this would make the process so much better!

[SvenDowideit]

@tianon what do you recon?

[dmacbride]

If you're uncertain about the disk format then but that on the back burner for now and start by including resize2fs in the boot2docker.iso. It would make things simpler without causing much disturbance.

[tyrken]

Apparently (from @ailispaw) you can run: tce-load -w -i e2fsprogs to install e2fsprogs (as non root), then assuming you've used fdisk to bump up the disk partition size do: resize2fs /dev/sda1 to finish the resizing as root.

[tianon]

Does anyone have a size comparison handy for before/after on ISO size with the package included? If the ISO difference is negligible, let's do it. :+1:

[tyrken]

I don't know about the ISO delta, but tce-load claimed to only download 184kB of e2fsprogs.tcz to add the package.

[jamshid]

What is the fdisk command to resize /dev/sda without losing data? It's not "delete partitions sda1 and sda2 (swap), then create a new partition". That causes "docker-machine restart" to hang and fail at "Waiting for an IP...".

[yosifkit]

@jamshid, That is the only instructions I can find for using fdisk to resize a partition. I would just grab a GParted live ISO and use it, since gparted removes the manual steps by doing both the partition resize and file system resize.

[masaeedu]

The problem is that the version of parted available on tinycore is precisely the most unfortunate possible version for getting anything done. parted deleted the ability to resize partitions in 3.0 because it was tied to some weird filesystem juggling stuff, then added it back later under the resizepart command. Guess what version of parted you have available on tinycore? fdisk only supports totally nuking and recreating your partition, so you can't use that either, because otherwise you won't have a filesystem to grow.