search

Ulexus / docker-ceph

DEPRECATED: use ceph/ceph-docker instead
http://github.com/ceph/ceph-docker
MIT License
42 stars 11 forks source link

Several osds per node #8

Open bobrik opened 10 years ago

bobrik commented 10 years ago

It seems that you cannot run many osds per node with ulexus/ceph-osd. I posted description to ceph-users mailing list to figure out what is wrong:

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-November/044996.html

Ulexus commented 10 years ago

Out of curiosity, have you made sure you are not suffering from port exhaustion or conntrack overflow? Check dmesg and netstat. This is usually the problem when I see Ceph go asymptotic.

bobrik commented 10 years ago

dmesg does not reveal anything, netstat shows many connections (>50k) in TIME_WAIT, because osds open and close connections very often in problematic case.

Ulexus commented 10 years ago

So, it's definitely port exhaustion, then. Obviously, that's an effect, not the root cause, but it does, perhaps, allow you to filter out some symptoms (load due to I/O wait, for instance).

bobrik commented 10 years ago

This happens with empty osds too, they cannot cause any io wait.

Ulexus commented 10 years ago

Correct; I am merely pointing out that it is possible to, for the purposes of testing a hypothesis, separate out symptoms which are related to port exhaustion, such as load due to I/O wait.

bobrik commented 10 years ago

Can you publish 0.80.7 images? Maybe the bug was fixed already, even though I wasn't able to find it it changelog.

Ulexus commented 10 years ago

Done

bobrik commented 10 years ago

No luck, still the same issue.