mwilck
commented
1 month ago Check your reservation_key setting in multipath.conf.
Closed ajaincdsc closed 3 days ago
mwilck
commented
1 month ago Check your reservation_key setting in multipath.conf.
ajaincdsc
commented
1 month ago multipaths {
multipath {
wwid "----"
alias san0
reservation_key 0x2
}
}looks correct on both nodes (the other one is 0x1), unless I'm missing something?
ajaincdsc
commented
1 month ago I feel like I have a lead on this, but I'm not sure. I found a thread from 2010 that suggests that all 6 active round robin ports that hold a particular active key (e.g. 0x1) should be able to access the SCSI device at the same time, however, when the scripts run, only 1 path out of 6 becomes a reservation holder. Does anyone have any hints as to what I should change to fix this behaviour?
% sudo mpathpersist -i -s /dev/mapper/san0
PR generation=0x42
Key = 0x2
All target ports bit clear
Relative port address: 0x201
not reservation holder
FCP-2 xx xx xx xx xx xx xx 48
Key = 0x2
All target ports bit clear
Relative port address: 0x202
not reservation holder
FCP-2 xx xx xx xx xx xx xx 49
Key = 0x1
All target ports bit clear
Relative port address: 0x101
<< Reservation holder >>
scope: LU_SCOPE, type: Write Exclusive, registrants only
FCP-2 xx xx xx xx xx xx xx 54
Key = 0x2
All target ports bit clear
Relative port address: 0x101
not reservation holder
FCP-2 xx xx xx xx xx xx xx 48
Key = 0x2
All target ports bit clear
Relative port address: 0x301
not reservation holder
FCP-2 xx xx xx xx xx xx xx 48
Key = 0x1
All target ports bit clear
Relative port address: 0x201
not reservation holder
FCP-2 xx xx xx xx xx xx xx 54
Key = 0x1
All target ports bit clear
Relative port address: 0x301
not reservation holder
FCP-2 xx xx xx xx xx xx xx 54
Key = 0x1
All target ports bit clear
Relative port address: 0x302
not reservation holder
FCP-2 xx xx xx xx xx xx xx 55
Key = 0x1
All target ports bit clear
Relative port address: 0x102
not reservation holder
FCP-2 xx xx xx xx xx xx xx 55
Key = 0x1
All target ports bit clear
Relative port address: 0x202
not reservation holder
FCP-2 xx xx xx xx xx xx xx 55
Key = 0x2
All target ports bit clear
Relative port address: 0x102
not reservation holder
FCP-2 xx xx xx xx xx xx xx 49
Key = 0x2
All target ports bit clear
Relative port address: 0x302
not reservation holder
FCP-2 xx xx xx xx xx xx xx 49To my understanding, all paths with the same key as the reservation holder (0x1) should be able to access the LUN. Otherwise I'd assume that multipathd would show them as failed. Can you confirm?
ajaincdsc
commented
1 month ago :~ % sudo multipath -ll
san0 (----) dm-10 NFINIDAT,InfiniBox
size=6.0T features='0' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 11:0:0:11 sdb 8:16 active ready running
|- 13:0:0:11 sde 8:64 active ready running
|- 11:0:1:11 sdk 8:160 active ready running
|- 13:0:1:11 sdi 8:128 active ready running
|- 11:0:2:11 sdg 8:96 active ready running
`- 13:0:2:11 sda 8:0 active ready runningSorry, wasn't sure how to confirm that. Does this help?
mwilck
commented
1 month ago Sorry, I was mistaken. According to the SCSI spec, with reservation type 5 (Write exclusive / registrants only), only one I/T nexus is the reservation holder. Write access should be allowed for every registered I_T nexus (i.e. both nodes; you could test this by writing to the SCSI devices but that would of course be dangerous). So what you report above is correct, AFAICT.
Can you try running the above mpathpersist command with -v4?
I should add that I have little experience with cluster configurations, perhaps @bmarzins will see more clearly than me what's going wrong.
ajaincdsc
commented
1 month ago Thanks for the help! Here's the output, though I apologize for how long it is... There are 6 Seagate drives on the host that we've blacklisted from multipath, which show up in the output. I've left them in but can trim if preferred.
bmarzins
commented
1 month ago Martin was wondering if you could run this command
/usr/sbin/mpathpersist -o --preempt-abort --prout-type=5 --param-rk=1 --param-sark=2 -d /dev/mapper/san0
with -v4.
bmarzins
commented
1 month ago the output of
multipath -T
would also be helpful
bmarzins
commented
1 month ago Don't bother with the command outputs. I can reproduce this myself. It looks to be a legit mpathpersist bug. I'll work up a fix next week.
ajaincdsc
commented
1 month ago I know you said not to bother, but on the off chance it helps:
bmarzins
commented
1 month ago Sent a fix for this to dm-devel.
https://lore.kernel.org/dm-devel/20240722233616.2496277-1-bmarzins@redhat.com/T/#u
ajaincdsc
commented
1 month ago We built v0.9.4 and added this patch to the list of patches from debian's maintainers and are currently running it on 2x debian 12 hosts. The error looks to be resolved and so far no negative effects have reared their head (though its been only 2 days). Looks good overall, thank you!
I will leave this issue open for you to tie in to the next pull from the queue.
mwilck
commented
3 weeks ago Thanks a lot, @bmarzins!
It's strange that this hasn't caused issues earlier, the affected code is ancient.
mwilck
commented
3 weeks ago @bmarzins, I don't quite understand your explanation yet why we haven't seen this error before. "This appeared to work for many commands because of an error in checking if the correct reservation key was used." The code that compared mpp->reservation_key against the command parameters has been in place for a long time, and it should always have had the result observed by @ajaincdsc, unless either key or sa_key happened to be 0.
Is it just that? Many of the "simple" PROUT commands just need either RK or SARK, and other commands would always have failed, but nobody noticed?
bmarzins
commented
3 weeks ago Yep. The following commands wouldn't fail before, because the command doesn't set both the RK and SARK.
registering a key:
mpathpersist -o -G -S <sark> <devpath>
reserving a device:
mpathpersist -o -R -K <rk> -T <type_nr> <devpath>
clearing a reservation:
mpathpersist -o -L -K <rk> -T <type_nr> <devpath>
clearing a registration:
mpathpersist -o -G -K <rk> <devpath>
As long as nothing goes wrong, those may well be the only commands that ever get run. But doing something like preempting and aborting an existing reservation fails, since it requires both a RK (which should match mpp->reservation_key) and a SARK:
mpathpersist -o -A -T <type_nr> -K <rk> -S <sark> <devpath>
Also, this only fails for configurations where the key is set in the multipaths section. It works fine if reservation_key is set to "file" in the defaults section, which is much easier to configure, and what, I assume, most people are using.
mwilck
commented
3 weeks ago Right ... "file" would be set in the defaults section, not the multipaths section. That was the part I'd been missing. Thanks.
Hello,
I can't seem to narrow down this error. On running this command:
/usr/sbin/mpathpersist -o --preempt-abort --prout-type=5 --param-rk=1 --param-sark=2 -d /dev/mapper/san0I receive this error:
san0: configured reservation key doesn't match: 0x0However, 0x0 doesn't match any configured keys:
There are 2 hosts running a Pacemaker stack using
fence_mpathto generate the commands, with the keys 1 and 2 respectively. Other commands seem to work fine, I'm just unable to figure out why there's a failure. I have tried manually releasing the reservation, but it ends up at the same place eventually.I am running Debian 12 with multipath-tools 0.9.4
Any help is appreciated, thanks!