/var/run/ceph-radosgw/ disappears on reboot

gravitystorm commented 9 years ago

https://github.com/ceph/ceph-cookbook/commit/0845115b52d355e4c81eca149c81afafabdfb698 introduced the /var/run/ceph-radosgw directory. Unfortunately that's not created in any of the init scripts, and disappears on reboot (Ubuntu 14.04 and likely others).

This means that on a reboot, the radosgw daemon won't work until the next chef run occurs, and even then not until radosgw is manually restarted.

gravitystorm commented 9 years ago

For what it's worth, here's my workaround. I have a wrapper cookbook, and within it I do the following:

template '/etc/init/radosgw-run-dir.conf' do
  variables(user: node['apache']['user'], group: node['apache']['group'])
end

templates/default/radosgw-run-dir.conf.erb:

# Managed by chef

start on starting radosgw

task

script
  if [ ! -d /var/run/ceph-radosgw ]; then
    mkdir -p /var/run/ceph-radosgw
    chown <%= @user %>:<%= @group %> /var/run/ceph-radosgw
    chmod 0755 /var/run/ceph-radosgw
  fi
end script

I'm no upstart expert, but it works for me. It basically jumps in just before any radosgw daemon starts, and checks if the run directory exists.

I've come to the conclusion that it's a bad idea to change the run directory away from what ceph expects. Within the packaged upstart files there are checks to ensure /var/run/ceph exists, but by changing dir in the chef recipe we miss out on these checks and it has the potential for trouble, like here. Do the other deployments like via ainsible or ceph-deploy also change the directory? Isn't it better to stick with the way that upstream expect it to work?

guilhem commented 9 years ago

It seems to be a problem with packaging (upstart scripts come from there). I don't want to fix problem from another project (here ceph itself) in a cookbook (it add too much complexity).

But your wrapper seems to be perfect for a quick fix :)

guilhem commented 9 years ago

Oh no... I begin to understand... I will try to find a solution

guilhem commented 9 years ago

Woo, problem is bigger than expected: http://tracker.ceph.com/issues/11453

ktdreyer commented 9 years ago

As background on that ticket "run RGW as root" ticket, see also http://tracker.ceph.com/issues/9001 . The long-term plan for RGW and everything else is to simply run as a "ceph" UID that will have rights the appropriate locations on disk.

By the way, for systemd we create /var/run/ceph with https://github.com/ceph/ceph/blob/master/systemd/ceph.tmpfiles.d . It looks like we need to handle this within the upstart configuration files, too?

Please note, if you guys are having to hack around packaging bugs, it would be great to have Redmine tickets filed upstream (and assign them to me :)

ceph / ceph-cookbook

/var/run/ceph-radosgw/ disappears on reboot #177