Closed tkenbo closed 3 months ago
Does this happen with latest nvme-cli and kernel? Older verison nvme-cli and kenrel will issue request for nvme list
Hi @tkenbo,
I had seen this issue, we added a condition to power cycle of the device once its inaccessible and then restart the process from where its left.
But, i think its a kind of bugs, is it possible to share your host properties like OS, Kernal version, nvmi-cli version etc. along with the python script, which you had mention.
I will try to reproduce and investigate more about this issue.
thanks'
Hi @AkashKumarSingh11032001,
Thank you for your response. We tested this on Rocky Linux 8.8 (kernel 4.18), 9.2 (kernel 5.13), 9.3 (kernel 5.14). And Gen3 Samsung SSD did not show this issue but all Gen4 SSD I tested showed it. I also tested nvme-cli, 2.21 and 2.9.1 (latest) and both showed the issue. I am struggling to install the latest kernel on my Rocky system, so much appreciated you looking into this. In our testing, if it fails, it would fail by a few hours longest, so running several hours should be long enough.
for whatever reason, I could not attach the script directly, so I put it as a snippet.
import subprocess
import time
import threading
import sys
fail = False
def list_loop():
global fail
print('list starts')
try:
while not fail:
result = subprocess.run('nvme list', shell=True, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE,encoding='utf-8')
time.sleep(0.1)
except subprocess.CalledProcessError as e:
print('list thread error')
print(e)
fail = True
print('list ends')
def reset_loop(dev):
global fail
print('reset starts')
try:
while not fail:
result = subprocess.run('nvme reset ' + dev, shell=True, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE,encoding='utf-8')
except subprocess.CalledProcessError as e:
print('reset thread error')
print(e)
fail = True
print('reset ends')
if __name__ == '__main__':
if len(sys.argv) == 1:
print('ERROR: need device node "python xxx.py /dev/nvme1"')
sys.exit(1)
print('starts on ' + sys.argv[1])
list = threading.Thread(target=list_loop)
reset = threading.Thread(target=reset_loop, args=(sys.argv[1],))
list.start()
reset.start()
list.join()
reset.join()
hi @igaw
I tried the followings.
With new kernel, I see "no usable path" message in the system log but the drive seems okay.
May 21 08:08:42 sm-110p-140 kernel: nvme nvme5: resetting controller
May 21 08:08:42 sm-110p-140 kernel: block nvme5n1: no usable path - requeuing I/O
May 21 08:08:42 sm-110p-140 kernel: block nvme5n1: no usable path - requeuing I/O
May 21 08:08:42 sm-110p-140 kernel: nvme nvme5: D3 entry latency set to 10 seconds
May 21 08:08:42 sm-110p-140 kernel: nvme nvme5: 52/0/0 default/read/poll queues
If this is a result of some fixes in kernel driver, do you have any idea which version of kernel has the change?
It looks like the driver recovers from the reset. The requeue message means that outstanding I/Os are not dropped instead requeued and issued as soon the device recovers. The last line indicates all is good again.
The fix I am referring consist in the kernel exposing additional sysfs entries and libnvme not issuing any commands when it finds all necessary sysfs entries for a topology scan.
You need a kernel which ships a1a825ab6a60 ("nvme: add csi, ms and nuse to sysfs") (kernel v6.8).
Though it could be another fix in the kernel which helps in your setup. Though from what you are describing it sounds like the hang is caused by issuing a command by nvme list
.
We have several test scripts to test nvme spec compliance and some of them intermittently fail for the error in title and the drive is inaccessible unless power cycle it.
This error tends to occur when we test multiple drives on one system concurrently and several investigations led us to prove this was a result of interference between "nvme reset" and "nvme list" command. When one of drives on the system is being reset by one test process, other processes send a list command potentially led the error. I created a simple python script that one thread to send a reset to a target drive repeatedly and the other thread to send list command. I tried this script on several Gen4 drives from various vendors and they all failed within 10 min to 1 hour. I also tried one of Gen3 drives but that did not fail.
When this happened, the system log always shows the following messages.
I was able to repeat this error on various systems and kernel 4.18 (Rocky8.8), 5.13(Rocky9.2) or 5.14 (Rocky9.3)
It'd much appreciated if anyone could enlighten me on this. I can provide a script I created if requested.