RHCS 3.0 in Vagrant
A Vagrant setup for Red Hat Ceph Storage version 3.0 running in VMs on your machine.
This will setup as many RHCS nodes as you want with a number of OSDs that you can define! You can choose between rpm-based install or Ceph running in containers.
Optionally you can choose to deploy the monitoring UI ceph-metrics.
Requirements
- macOS with Virtualbox (starting 5.1.30) or
- RHEL 7.4/Fedora 27 with KVM/libvirt
- Ansible (starting 2.4.0.0)
- Vagrant (starting 1.9.1)
- git
python-netaddr
Before you start - installation instructions for Vagrant / Ansible and dependencies
On RHEL 7.4 / CentOS 7
- make sure you are logged in as a user with
sudo
privileges
- on RHEL 7 make sure your system has the following repositories enabled (
yum repolist
)
- rhel-7-server-rpms
- rhel-7-server-extras-rpms
- install the requirements
sudo yum groupinstall "Virtualization Host"
sudo yum install ansible git gcc libvirt-devel
sudo yum install python-netaddr
sudo yum install https://releases.hashicorp.com/vagrant/2.0.1/vagrant_2.0.1_x86_64.rpm
- start
libvirtd
sudo systemctl enable libvirtd
sudo systemctl start libvirtd
- enable libvirt access for your current user
sudo gpasswd -a $USER libvirt
- as your normal user, install the libvirt plugin for vagrant
vagrant plugin install vagrant-libvirt
On Fedora 27
- make sure you are logged in as a user with
sudo
privileges
- make sure your system has the following repositories enabled (
dnf repolist
)
- install the requirements
sudo dnf install ansible git gcc libvirt-devel libvirt qemu-kvm
sudo dnf install vagrant vagrant-libvirt
sudo dnf install python-netaddr
- start
libvirtd
sudo systemctl enable libvirtd
sudo systemctl start libvirtd
- enable libvirt access for your current user
sudo gpasswd -a $USER libvirt
- as your normal user, install the libvirt plugin for vagrant
vagrant plugin install vagrant-libvirt
On macOS High Sierra
- install the requirements
- install Virtualbox
- install Vagrant
- install homebrew
- install git
brew install git
- install ansible
brew install ansible
- install python pip
sudo easy_install pip
- install python-netaddr
pip install netaddr
Get started
- You must be in the Red Hat VPN
- Clone this repository
git clone https://github.com/red-hat-storage/RHCS-vagrant.git
- Goto the folder in which you cloned this repo
- if you are a returning user run
git pull
to ensure you have the latest updates
- if you are on RHEL/Fedora and you don't want your libvirt storage domain
default
to be used, override the storage domain like this
export LIBVIRT_STORAGE_POOL=images
- Run
vagrant up
- Decide between the installation type (rpm-based vs. containerized)
- Decide whether you want to use filestore or bluestore (the latter is preview)
- Decide how many OSD nodes and how many devices you need
- rpm-based install: decide how many MON, RGW, MDS, etc. nodes you want
- containerized install: decide whether or not and how many additional ceph services (RGW, MDS, etc.) you want to co-locate to the OSD nodes
- Decide if you want a separate client node
- Decide if you want vagrant to initialize the cluster (using
ceph-ansible
) for you
- If you opted to initialize the cluster, decide whether you want to deploy
ceph-metrics
(only available for rpm-based filestore-backed clusters)
- Wait a while
- If you like to start over:
vagrant destroy -f
Usage
- Always make sure you are in the git repo - vagrant only works in there!
- After
vagrant up
you can connect to each VM with vagrant ssh
and the name of the VM you want to connect to
- FYI the password for the
vagrant
user is 'vagrant' but you shouldn't need it
- Each VM is called according to the Ceph node type (e.g.
OSDx
or MONx
where x starts with 1)
- There is an additional VM called
METRICS
which hosts the Ceph Metrics Monitoring Stack if you selected to deploy it (URL is displayed at the end of vagrant up
)
- If you selected an rpm-based install log on to one of the MON nodes to use ceph client utilities and administer the cluster
- If you selected an containerized install log on to one of the OSD nodes and execute a shell inside one of the MON containers
docker exec -it ceph-mon bash
- run
sudo ceph -s
on one of the MONx nodes (rpm-based installed) or ceph -s
inside the MON containers on OSDx nodes (containerized install) to check for your cluster status
- There are also other vagrant commands you should check out!
- if you want to throw away everything:
vagrant destroy -f
- if you want to freeze the VMs and continue later:
vagrant suspend
- Try
vagrant -h
to find out about them
- if you run
vagrant up
again you without running vagrant destroy
before you will overwrite your configuration and vagrant may loose track of some VMs (it's safe to remove them manually)
- modify the
VMMEM
and VMCPU
variables in the Vagrant file to change RHCS VM resources, adjust VMDISK
to change OSD device sizes
What happens under the covers
- After starting the RHCS VMs on all nodes:
- the hosts file is pre-populated
- all ceph packages and docker images are pre-installed (allows you to continue offline)
- the RHEL images are subscribed to YUM repositories on the RHT VPN
- a
ceph-ansible
setup is prepared in /usr/share/ceph-ansible
and the inventory file is set up in /etc/ansible/hosts
- If you decided to have vagrant initialize the cluster and you chose an rpm-based install:
ceph-ansible
's site.yml
playbook was executed
- the selected ceph roles were installed on the respective VMs (named after the role)
- cluster is up and in HEALTHY state
- If you decided to have vagrant initialize the cluster and you chose an containerized install:
- the selected amount of OSD VMs were created
- docker service is enabled and started (using overlay2 on a separate logical volume)
ceph-ansible
's site-docker.yml
playbook was executed
- the selected ceph roles were installed in containers on the OSD nodes (one per node)
- as of today containerized iSCSI is not yet supported
- cluster is up and in HEALTHY state
- If you decided to deploy
ceph-metrics
(only available when initializing the cluster and only for rpm-based install with filestore
as the OSD backend)
- an additional VM called METRICS will run
ceph-metrics
dashboard components
- at the end of the metrics deployment you will see the URL to reach the dashboard displayed
- If you opted out of cluster initialization a working
ceph-ansible
was left in place for your convenience
Customize your Ceph installation
- in the ansible/templates folder there are group variable files for
ceph-ansible
for every host type it supports
- if you like to make customizations, edit these files as needed
- during provisioning (
vagrant up
or vagrant provision
) these files will be copied to the nodes so they are picked up by the installation routine
- WARNING: these files are processed as Jinja2 template files by this Vagrant setup - modify contents in square brackets (
[[ ... ]]
or [% %]
at your own risk)
Clean up / Refresh images
If you like to clean up disk space or there are updates to the images do the following:
- run
rm ~/.vagrant.d/boxes/*-rhcs-*.box
and rm ~/.vagrant.d/boxes/*-metrics-*.box
to delete older Vagrant images
- on VirtualBox - remove the VM instances named
packer-...
(these are base images for the clones)
- on libvirt
- run
virsh vol-list default
to list all images in your default
storage pool (adjust the name if you are using a different one)
- run
virsh vol-delete packer-... default
to delete the images starting with packer-...
(replace with full name) from the default pool
Next time you do vagrant up
it will automatically pull new images.
Known issues
vagrant up
overrides your state - if there are still VMs running and you do vagrant up
it will override the vagrant_env.conf
and vagrant will loose track of your existing VMs
- try to remember to run
vagrant destroy -f
before you do another vagrant up
- delete left-over VMs manually in case you forgot
- containerized installation of RHCS fails at stage
pull {{ ceph_docker_image }} image
with the error message:
- ceph-metrics is not showing current data (doesn't show RGW or MDS)
- if you deployed many VMs and your system is under memory pressure ceph-metrics might need up to 10 minutes reconcile cluster state
- on RHEL/CentOS you will get some Ruby warnings caused by older Gem versions when running Vagrant commands:
... .vagrant.d/gems/2.4.2/gems/nokogiri-1.6.8.1/lib/nokogiri/xml/document.rb:44: warning: constant ::Fixnum is deprecated
- this is just a warning, functionality is not impacted
Creating your own vagrant box
If you - for whatever reason - do not want to use my prebuilt box, you can create your own box very easy!
BEWARE this is for advanced users only!
- Get packer
- run
git checkout -b packer
to switch the "packer" branch of this repository, follow the README in packer directory
Author
Daniel Messer - dmesser@redhat.com -
Technical Marketing Manager @ Red Hat
Original Authors
Christopher Blum
Stephan Hohn