Firmware: multi hosts: fail to promotion from shareable to exclusive mode if another host is timeout

[Leo-Yan]

Firmware version: signed_CFW_MFW_r2229.lod

Host A: Acquire a mutex with shareable mode and stop to renew it

Host B: Acquire the same mutex with shareable mode; Wait for 60 seconds for host A timeout;

Mutex state for Host A:

400    00 00 00 00 00 00 01 04  00 00 01 73 13 b1 02 22
410    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 02
420    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
430    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
440    32 78 6a 38 4a 64 45 36  58 76 64 75 32 64 4d 75
450    77 37 34 6a 31 70 53 6a  57 52 6f 47 74 49 30 46
460    37 4e 4e 32 6a 43 69 68  64 32 63 55 41 73 72 66
470    5a 58 63 49 76 6b 52 79  61 58 66 45 77 61 69 57
480    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
490    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
4a0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
4b0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
4c0    3a 8f 3e b4 79 0c f5 b0  22 4e 82 20 07 3a 25 bc
4d0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 47 50
4e0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
4f0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
500    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
510    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
520    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
530    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
540    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
550    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
560    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
570    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
580    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
590    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
5a0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
5b0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
5c0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
5d0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
5e0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
5f0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00

Mutex state for Host B:

600    00 00 00 00 00 00 01 01  00 00 01 73 13 b1 02 22
610    00 00 00 00 00 00 e8 14  00 00 00 00 00 00 00 02
620    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
630    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
640    32 78 6a 38 4a 64 45 36  58 76 64 75 32 64 4d 75
650    77 37 34 6a 31 70 53 6a  57 52 6f 47 74 49 30 46
660    37 4e 4e 32 6a 43 69 68  64 32 63 55 41 73 72 66
670    5a 58 63 49 76 6b 52 79  61 58 66 45 77 61 69 57
680    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
690    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
6a0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
6b0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
6c0    f0 16 15 5c 41 0b ed b4  94 4d 4e e4 21 14 84 8c
6d0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 6f 39
6e0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
6f0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
700    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
710    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
720    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
730    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
740    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
750    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
760    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
770    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
780    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
790    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
7a0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
7b0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
7c0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
7d0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
7e0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
7f0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00

If host B uses refresh command to convert from shareable mode to exclusive mode, the drive firmware will report the the error as:

array: _scsi_write cdb
0000: 89 00 01 00 00 00 00 00 00 00 00 00 00 01 05 00
array: _scsi_write data
0000: 00 00 00 00 00 00 00 00 00 00 01 73 13 b2 60 2a
0010: 00 00 00 00 00 00 ea 60 00 00 00 00 00 00 00 00
0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0040: 32 78 6a 38 4a 64 45 36 58 76 64 75 32 64 4d 75
0050: 77 37 34 6a 31 70 53 6a 57 52 6f 47 74 49 30 46
0060: 37 4e 4e 32 6a 43 69 68 64 32 63 55 41 73 72 66
0070: 5a 58 63 49 76 6b 52 79 61 58 66 45 77 61 69 57
0080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00c0: f0 16 15 5c 41 0b ed b4 94 4d 4e e4 21 14 84 8c
00d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 6f 39
00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

2020-07-03 03:54:18 25150 [27788]: _scsi_read: status 0x22 2020-07-03 03:54:18 25150 [27788]: _scsi_read: masked status 0x11 2020-07-03 03:54:18 25150 [27788]: _scsi_read: host status 0x0 2020-07-03 03:54:18 25150 [27788]: _scsi_read: driver status 0x0

In this case, the host B should can convert the lock mode from shareable to exclusive successfully.

[alphonsus-kwok]

According to the IDM State-Cmd matrix, this is the use case S13 / S15 which will return Command Terminated.

[Leo-Yan]

Hi @alphonsus-kwok,

Understood. Sorry I didn't carefully review the defined status mapping for this case.

From my understanding, this issue refers to the use case S15, the owner has locked the mutex, but other hosts have been timeout. For this case, the owner can promotion the locking mode from shareable to exclusive.

To fix this case, the lock manager can use an ugly workaround by firstly releasing the mutex and then acquiring again with exclusive mode. But this might introduce atomic issue. The best approach is to fix in the drive firmware: the owner can hold its ownership and promote from shareable mode to exclusive mode.

[alphonsus-kwok]

Perhaps Breaklock command would be better. For use case HostA - shared timeout HostB - shared locked HostC - shared timeout then breaklock command sent by HostB (exclusive) will promote to Exclusive class.

For use case: HostA - shared timeout HostB - shared locked HostC - shared locked then breaklock command sent by HostB (exclusive) will return Command Terminated.

Let's wait for Tom's input also.

[Leo-Yan]

Hi @alphonsus-kwok,

I have tried to use breaklock command to workaround this issue, but based on the testing result, the break lock command also returns "Command Terminated" error.

[TProhofsky]

Sending a BreakLock Shared followed by a Refresh Exclusive should promote to the Rid to exclusive if all other shared locks have timed out. A BreakLock Exclusive should also do this in one command. See version 1.4b of the command response spreadsheet for confirmation.

[Leo-Yan]

Did a quick trying for the flow:

• Step 1: BreakLock Shared
• Step 2: Refresh Exclusive

When breaklock shared, the SCSI command still returns failure with status "COMMAND_TERMINATED".

As tried in my previous replying for "A BreakLock Exclusive", also returns failure with status "COMMAND_TERMINATED".

[alphonsus-kwok]

Breaklock will return "COMMAND_TERMINATED" as there is still an active locked mutex as per the IDM State-Cmd matrix cell W15.

But I saw a bug that after Refresh, the mutex will be lost. I will resolve it and give you an update.

[Leo-Yan]

Tested with two cases:

One case is IDM lock manager test cases: https://github.com/Seagate/propeller/blob/master/test/ilm_test.py#L736 and another case is LVM multi hosts testing: https://github.com/Seagate/lvm2-stx-private/blob/centos7_lvm2/test/shell/idm_multi_hosts_lv_sh_timeout_hostb.sh; both can pass.

So can confirm this issue has been fixed by the latest firmware 1759.