ceph / ceph-iscsi

Ceph iSCSI tools
GNU General Public License v3.0
62 stars 59 forks source link

spec: make sure we always have the target dir #271

Closed lxbsz closed 1 year ago

lxbsz commented 1 year ago

rtslib now requires /etc/target or /var/target to be setup (it depends on the version). The issue is that rtslib does not make those dirs and instead relies on targetcli or ceph-iscs to make them.

For container cases we do not install targetcli, so ceph-iscsi's daemons will crash or fail to setup disks, when we make rtslib calls that try to access the etc/var target dirs.

When rtslib is fixed drop this and update the dep. Note that we add both etc and var, because the kernel and userspace developers keep switching the dir they want to use.

idryomov commented 1 year ago

The issue is that rtslib does not make those dirs and instead relies on targetcli or ceph-iscs to make them.

Typo: ceph-iscsi

For container cases we do not install targetcli, so ceph-iscsi's daemons will crash or fail to setup disks, when we make rtslib calls that try to access the etc/var target dirs.

ceph-container doesn't mount /etc/target and/or /var/target from the host either, so I suspect the only reason it happens to work when these directories are created during package installation is that ceph-iscsi doesn't (need to) communicate with the kernel via either of them. This get us past the checks in rtslib but no meaningful ceph-iscsi <-> kernel communication can occur because they would be reading from/writing to different locations (container vs host).

lxbsz commented 1 year ago

The issue is that rtslib does not make those dirs and instead relies on targetcli or ceph-iscs to make them.

Typo: ceph-iscsi

For container cases we do not install targetcli, so ceph-iscsi's daemons will crash or fail to setup disks, when we make rtslib calls that try to access the etc/var target dirs.

ceph-container doesn't mount /etc/target and/or /var/target from the host either, so I suspect the only reason it happens to work when there directories are created during package installation is that ceph-iscsi doesn't (need to) communicate with the kernel via either of them. This get us past the checks in rtslib but no meaningful ceph-iscsi <-> kernel communication can occur because they would be reading from/writing to different locations (container vs host).

Yeah, sounds reasonble.

idryomov commented 1 year ago

Additional note: since https://github.com/open-iscsi/rtslib-fb/commit/99072f0cee35e1591f76b1c470a57f5299e2e527 rtslib is more aggressive about changing dbroot to what it likes better (upstream it's the "preferred" /etc/target which aligns with LIO in the kernel; however in Fedora, CentOS and RHEL it's the "default" /var/target). This can cause trouble on live upgrades:

1) rtslib attempts to change dbroot upon restart of ceph-iscsi 2) kernel rejects the change because dbroot is in use (i.e. there are registered devices from before the upgrade) 3) rtslib checks whether the directory it attempted to change dbroot to exists, sees that it doesn't exist, assumes that to be reason for the rejection and throws an exception

https://github.com/open-iscsi/rtslib-fb/blob/745d51a46e6718e34c59b728fd96ec08bbf906dc/rtslib/root.py#L169-L182