Closed ajaincdsc closed 3 days ago
Check your reservation_key
setting in multipath.conf.
multipaths {
multipath {
wwid "----"
alias san0
reservation_key 0x2
}
}
looks correct on both nodes (the other one is 0x1), unless I'm missing something?
I feel like I have a lead on this, but I'm not sure. I found a thread from 2010 that suggests that all 6 active round robin ports that hold a particular active key (e.g. 0x1) should be able to access the SCSI device at the same time, however, when the scripts run, only 1 path out of 6 becomes a reservation holder. Does anyone have any hints as to what I should change to fix this behaviour?
% sudo mpathpersist -i -s /dev/mapper/san0
PR generation=0x42
Key = 0x2
All target ports bit clear
Relative port address: 0x201
not reservation holder
FCP-2 xx xx xx xx xx xx xx 48
Key = 0x2
All target ports bit clear
Relative port address: 0x202
not reservation holder
FCP-2 xx xx xx xx xx xx xx 49
Key = 0x1
All target ports bit clear
Relative port address: 0x101
<< Reservation holder >>
scope: LU_SCOPE, type: Write Exclusive, registrants only
FCP-2 xx xx xx xx xx xx xx 54
Key = 0x2
All target ports bit clear
Relative port address: 0x101
not reservation holder
FCP-2 xx xx xx xx xx xx xx 48
Key = 0x2
All target ports bit clear
Relative port address: 0x301
not reservation holder
FCP-2 xx xx xx xx xx xx xx 48
Key = 0x1
All target ports bit clear
Relative port address: 0x201
not reservation holder
FCP-2 xx xx xx xx xx xx xx 54
Key = 0x1
All target ports bit clear
Relative port address: 0x301
not reservation holder
FCP-2 xx xx xx xx xx xx xx 54
Key = 0x1
All target ports bit clear
Relative port address: 0x302
not reservation holder
FCP-2 xx xx xx xx xx xx xx 55
Key = 0x1
All target ports bit clear
Relative port address: 0x102
not reservation holder
FCP-2 xx xx xx xx xx xx xx 55
Key = 0x1
All target ports bit clear
Relative port address: 0x202
not reservation holder
FCP-2 xx xx xx xx xx xx xx 55
Key = 0x2
All target ports bit clear
Relative port address: 0x102
not reservation holder
FCP-2 xx xx xx xx xx xx xx 49
Key = 0x2
All target ports bit clear
Relative port address: 0x302
not reservation holder
FCP-2 xx xx xx xx xx xx xx 49
To my understanding, all paths with the same key as the reservation holder (0x1) should be able to access the LUN. Otherwise I'd assume that multipathd would show them as failed. Can you confirm?
:~ % sudo multipath -ll
san0 (----) dm-10 NFINIDAT,InfiniBox
size=6.0T features='0' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 11:0:0:11 sdb 8:16 active ready running
|- 13:0:0:11 sde 8:64 active ready running
|- 11:0:1:11 sdk 8:160 active ready running
|- 13:0:1:11 sdi 8:128 active ready running
|- 11:0:2:11 sdg 8:96 active ready running
`- 13:0:2:11 sda 8:0 active ready running
Sorry, wasn't sure how to confirm that. Does this help?
Sorry, I was mistaken. According to the SCSI spec, with reservation type 5 (Write exclusive / registrants only), only one I/T nexus is the reservation holder. Write access should be allowed for every registered I_T nexus (i.e. both nodes; you could test this by writing to the SCSI devices but that would of course be dangerous). So what you report above is correct, AFAICT.
Can you try running the above mpathpersist command with -v4
?
I should add that I have little experience with cluster configurations, perhaps @bmarzins will see more clearly than me what's going wrong.
Thanks for the help! Here's the output, though I apologize for how long it is... There are 6 Seagate drives on the host that we've blacklisted from multipath, which show up in the output. I've left them in but can trim if preferred.
Martin was wondering if you could run this command
/usr/sbin/mpathpersist -o --preempt-abort --prout-type=5 --param-rk=1 --param-sark=2 -d /dev/mapper/san0
with -v4.
the output of
multipath -T
would also be helpful
Don't bother with the command outputs. I can reproduce this myself. It looks to be a legit mpathpersist bug. I'll work up a fix next week.
I know you said not to bother, but on the off chance it helps:
Sent a fix for this to dm-devel.
https://lore.kernel.org/dm-devel/20240722233616.2496277-1-bmarzins@redhat.com/T/#u
We built v0.9.4 and added this patch to the list of patches from debian's maintainers and are currently running it on 2x debian 12 hosts. The error looks to be resolved and so far no negative effects have reared their head (though its been only 2 days). Looks good overall, thank you!
I will leave this issue open for you to tie in to the next pull from the queue.
Thanks a lot, @bmarzins!
It's strange that this hasn't caused issues earlier, the affected code is ancient.
@bmarzins, I don't quite understand your explanation yet why we haven't seen this error before. "This appeared to work for many commands because of an error in checking if the correct reservation key was used." The code that compared mpp->reservation_key
against the command parameters has been in place for a long time, and it should always have had the result observed by @ajaincdsc, unless either key
or sa_key
happened to be 0.
Is it just that? Many of the "simple" PROUT commands just need either RK or SARK, and other commands would always have failed, but nobody noticed?
Yep. The following commands wouldn't fail before, because the command doesn't set both the RK and SARK.
registering a key:
mpathpersist -o -G -S <sark> <devpath>
reserving a device:
mpathpersist -o -R -K <rk> -T <type_nr> <devpath>
clearing a reservation:
mpathpersist -o -L -K <rk> -T <type_nr> <devpath>
clearing a registration:
mpathpersist -o -G -K <rk> <devpath>
As long as nothing goes wrong, those may well be the only commands that ever get run. But doing something like preempting and aborting an existing reservation fails, since it requires both a RK (which should match mpp->reservation_key) and a SARK:
mpathpersist -o -A -T <type_nr> -K <rk> -S <sark> <devpath>
Also, this only fails for configurations where the key is set in the multipaths section. It works fine if reservation_key is set to "file" in the defaults section, which is much easier to configure, and what, I assume, most people are using.
Right ... "file" would be set in the defaults section, not the multipaths section. That was the part I'd been missing. Thanks.
Hello,
I can't seem to narrow down this error. On running this command:
/usr/sbin/mpathpersist -o --preempt-abort --prout-type=5 --param-rk=1 --param-sark=2 -d /dev/mapper/san0
I receive this error:
san0: configured reservation key doesn't match: 0x0
However, 0x0 doesn't match any configured keys:
There are 2 hosts running a Pacemaker stack using
fence_mpath
to generate the commands, with the keys 1 and 2 respectively. Other commands seem to work fine, I'm just unable to figure out why there's a failure. I have tried manually releasing the reservation, but it ends up at the same place eventually.I am running Debian 12 with multipath-tools 0.9.4
Any help is appreciated, thanks!