OSD Mount Type Support?

[nicgrayson]

It looks like only directory mode is used for osds according to this code block: https://github.com/vivint-smarthome/ceph-on-mesos/blob/master/src/main/scala/com/vivint/ceph/JobBehavior.scala#L103

    val taskInfo = launchCephCommand(
      taskId = taskId,
      jobId = job.id,
      role = job.role,
      offer = offer,
      location = taskLocation,
      templatesTgz = templatesTgz,
      command =
        runState match {
          case RunState.Running =>
            s"""
            |FS_TYPE="$$(df -T /var/lib/ceph | tail -n 1 | awk '{print $$2}')"
            |if [ "$${FS_TYPE}" != "xfs" ] && [ "$${FS_TYPE}" != "btrfs" ]; then
            |  echo "Cowardly refusing to OSD start on non-xfs / non-btrfs volume."
            |  echo "Cowardly refusing to OSD start on non-xfs / non-btrfs volume." 1>&2
            |  echo "Please see http://docs.ceph.com/docs/jewel/rados/configuration/filesystem-recommendations/#not-recommended for more information"
            |  sleep 60
            |  exit
            |fi
            |set -x -e
            |echo "Pulling monitor map"
            |ceph mon getmap -o /etc/ceph/monmap-ceph
            |
            |if [ ! -f /etc/ceph/my_osd_id ]; then
            |  ceph osd create > /etc/ceph/my_osd_id
            |fi
            |OSD_ID=$$(cat /etc/ceph/my_osd_id)
            |mkdir -p /var/lib/ceph/osd/ceph-$${OSD_ID}
            |chown ceph:ceph /var/lib/ceph/osd/ceph-$${OSD_ID}
            |
            |exec /entrypoint.sh osd_directory
            |""".stripMargin
          case RunState.Paused =>
            s"""
            |sleep 86400
            |""".stripMargin
        }
    )

1. Is there a way to change the directory used to be a volume mount like: /dcos/volume0
2. Is there a way to use a disk mount like the ceph-docker project explained here: https://github.com/ceph/ceph-docker/tree/master/ceph-releases/jewel/ubuntu/14.04/daemon#ceph-disk

[fersantxez]

AFAIK mount type is indeed used. And if you're looking for DC/OS support there's a tutorial here: https://github.com/dcos/examples/tree/master/1.8/ceph

[nicgrayson]

@fernandosanchezmunoz I followed that tutorial. The ceph cluster was showing the usage of the whole cluster instead of the mounted disks so I started to investigate the framework. I then found that the command the osd's run is ./entrypoint.sh osd_directory which means that only the osd directory config is used.

[fersantxez]

@nicgrayson I've ran it a good amount of times I'm fairly sure that the OSDs use the additional volumes provided as mounts. For example, if I have a cluster with three nodes running OSDs where the system volume is 16G, and the additional volume configured for Ceph and added to mesos-resources is 8G, my mesos dashboard shows that the cluster is using 24G, and I can see a total 24G available for Ceph in the ceph-dash. Can't provide an answer for that entrypoint question though.

[timcharper]

To answer directly:

Is there a way to change the directory used to be a volume mount like: /dcos/volume0

Yes. This is the recommended way to deploy the framework. See this config option: https://github.com/vivint-smarthome/ceph-on-mesos/blob/v0.2.11/src/main/resources/deployment-config.conf#L28

Is there a way to use a disk mount like the ceph-docker project explained here: https://github.com/ceph/ceph-docker/tree/master/ceph-releases/jewel/ubuntu/14.04/daemon#ceph-disk

While the framework supports Mesos mount disks, it does not support Ceph mount disks. This is because it is currently not possible to offer up a raw block device with Mesos as a resource. However, it should be possible to create a contiguous, grow-on-write file which spans the entire file system... and then mount a-la-loopback. See https://github.com/vivint-smarthome/ceph-on-mesos/blob/master/TODO.md#maybe .