The above problem happens because my job config has multiple osds on 1 node (osd.0 and osd.1 deployed on same host) and there is only 1 device available on each testnode container that can be zapped for osd deployment.
Using ceph-devstack setup, the teuthology function get_scratch_devices() returned 1 devices for each testnode. So the mapping (devs_by_remote) looks like this :
And because we pop the loop device from the above devs_by_remote after 1st osd is deployed, the 2nd osd on same testnode has no more available devices to deploy 2nd osd on.
I rerun my test with 1 osd/node config and that worked (test went through the ceph setup okay).
As for a proper solution... does this mean we should create more loop devices per testnode in ceph-devstack?
Let me know, I'll love to pick this issue. It'll be a good gateway to understand more of ceph-devstack.
When I ran a test with config which had multiple OSDs on a single testnode, I got the following error:
Each testnode had one loop device: https://pastebin.com/raw/8z5gj0CU (
ls /dev
output)The above problem happens because my job config has multiple osds on 1 node (osd.0 and osd.1 deployed on same host) and there is only 1 device available on each testnode container that can be zapped for osd deployment. Using ceph-devstack setup, the teuthology function get_scratch_devices() returned 1 devices for each testnode. So the mapping (devs_by_remote) looks like this :
And because we pop the loop device from the above
devs_by_remote
after 1st osd is deployed, the 2nd osd on same testnode has no more available devices to deploy 2nd osd on. I rerun my test with 1 osd/node config and that worked (test went through the ceph setup okay).As for a proper solution... does this mean we should create more loop devices per testnode in ceph-devstack? Let me know, I'll love to pick this issue. It'll be a good gateway to understand more of ceph-devstack.