nvmetool-win: Sample program of accessing NVMe device using Windows' inbox NVMe driver
Abstract
This software demonstrates that issuing some NVMe commands from userland to NVMe device using Windows inbox NVMe driver (stornvme.sys)[1].
Specification that this software refered to is NVMe 1.3d[2] and NVMe 1.4b[6].
Some fields introduced in NVMe Base Specification revision 2.0[10] are supported.
For StorNVMe on Windows 10 version 1903 and later, Supported NVMe Admin and NVM command sets and associated opcodes are shown in [8], and supported features are shown in [9].
Note for this software
This software is a SAMPLE and DEMONSTRATION program to show how to access NVMe drive with Windows' inbox NVMe driver.
It is NOT intended to provide versatile tool with such functions like accepting arbitrary value for parameters, file input / output, non-interactive mode, support for vendor specific commands.
If you need such functions, you can create your own tool based on this software.
Thank you.
Command matrix
Table 1. Command matrix (Admin Command Set)
Legend for "Status" column: "x" => can issue, "--" => cannot issue, " "(blank) => not confirmed
| Opcode | Command name | Status | Note |
|---|---|---|---|
| 00h | Delete I/O Submission Queue | 1 | |
| 01h | Create I/O Submission Queue | 1 | |
| 02h | Get Log Page | See Table 4 | |
| 04h | Delete I/O Completion Queue | 1 | |
| 05h | Create I/O Completion Queue | 1 | |
| 06h | Identify | See Table 3 | |
| 08h | Abort | 1 | |
| 09h | Set Features | See Table 5 | |
| 0Ah | Get Features | See Table 5 | |
| 0Ch | Asynchronous Event Request | 1 | |
| 0Dh | Namespace Management | 2 | |
| 10h | Firmware Commit | ||
| 11h | Firmware Image Download | ||
| 14h | Device Self-test | x | |
| 15h | Namespace Attachment | 2 | |
| 18h | Keep Alive | Not listed in [8] | |
| 19h | Directive Send | 1 | |
| 1Ah | Directive Receive | 1 | |
| 1Ch | Virtualization Management | Not supported | |
| 1Dh | NVMe-MI Send | 2 | |
| 1Eh | NVMe-MI Receive | 2 | |
| 20h | Capacity Management | from 2.0 | |
| 24h | Lockdown | from 2.0 | |
| 7Ch | Doorbell Buffer Config | Not supported | |
| 80h | Format NVM | x | Only for data drive. Need to support Format NVM command and cryptographic erase is supported along Format NVM (bit 2 of the FNA field in Identify Controller data structure) |
| 81h | Security Send | ||
| 82h | Security Receive | x | Getting TCG Level 0 Discovery Data for example in this tool |
| 84h | Sanitize | 2 | |
| 86h | Get LBA Status | Not supported | |
| C0h -- FFh | Vendor Specific | Sample (Template) code exists |
Note 1: probably cannot issue this command because only "Internal Driver Use" is written in [8]
Note 2: can issue this command only in WinPE mode according to [8]
Note 3: To use sample (template) code for issuing Vendor Specific Commands, you need to edit the codes as you want to do
Table 2. Command matrix (NVM Command Set)
Legend for "Status" column: "x" => can issue, "--" => cannot issue, " "(blank) => not confirmed
| Opcode | Command name | Status | Note |
|---|---|---|---|
| 00h | Flush | x | |
| 01h | Write | x | |
| 02h | Read | x | |
| 04h | Write Uncorrectable | 1 | |
| 05h | Compare | Can issue only in WinPE mode [8] | |
| 08h | Write Zeroes | 1 | |
| 09h | Dataset Management | x | Only Deallocate is supported |
| 0Ch | Verify | 1 | |
| 0Dh | Reservation Register | 1 | |
| 0Eh | Reservation Report | 1 | |
| 11h | Reservation Acquire | 1 | |
| 15h | Reservation Release | 1 | |
| 19h | Copy | from 2.0 | |
| 80h -- FFh | Vendor Specific | Sample (Template) code exists |
Note 1: these command are stated as "Not supported" in [8]
Note 2: To use sample (template) code for issuing Vendor Specific Commands, you need to edit the codes as you want to do
Controller or Namespace Structure (CNS) for Identify command
CNS values from 10h to 13h are seemed to be able to issue only in WinPE mode, because these values are mandatory for controllers that support the Namespace Management capability in NVMe spec., but Namespace Management command can only be issued in WinPE mode according to [8].
CNS values 14h and 15h are seemed to be unable to issue, because these values are mandatory for controllers that support Virtualization Enhancements in NVMe spec., but Virtualization Management command is not supported according to [8].
Table 3. Controller or Namespace Structure (CNS)
Legend for "Status" column: "x" => can issue, "--" => cannot issue, " "(blank) => not confirmed
| CNS | Description | Status | Note |
|---|---|---|---|
| 00h | Namespace | x | NSID = 1 |
| 01h | Controller | x | CNTID = 0 |
| 02h | Active Namespace ID list | we can't specify NSID... and seems no data is returned... | |
| 03h | Namespace Identification Descriptor list | x | |
| 04h | NVM Set list | ||
| 05h | I/O Command Set specific Identify Namespace data structure | x | NSID = 1 |
| 06h | I/O Command Set specific Identify Controller data structure | CNTID = 0 | |
| 07h | Active Namespace ID list associated with the specified I/O Command Set | ||
| 08h | I/O Command Set Independent Identify Namespace data structure | NSID = 1 | |
| 09h -- 0Fh | Reserved | ||
| 10h | Allocated Namespace ID list | Probably can issue only in WinPE mode [8] | |
| 11h | Specified allocated namespace | Probably can issue only in WinPE mode [8] | |
| 12h | List of controllers attached to specified namespace | Probably can issue only in WinPE mode [8] | |
| 13h | List of controllers exist in the NVM subsystem | Probably can issue only in WinPE mode [8] | |
| 14h | Primary Controller Capabilities | Probably cannot issue [8] | |
| 15h | Secondary controller list | Probably cannot issue [8] | |
| 16h | Namespace Granularity List | ||
| 17h | UUID List | ||
| 18h | Domain List | ||
| 19h | Endurance Group List | ||
| 1Ah | I/O Command Set specific Allocated Namespace ID list | ||
| 1Bh | I/O Command Set specific Identify Namespace data structure | ||
| 1Ch | I/O Command Set data structure | ||
| 1Dh -- 1Fh | Reserved | Spec says CNS "18h to 1Fh" are reserved, I think it's typo. | |
| 20h -- FFh | Reserved |
Log Identifier (LID) for Get Log Page command
Table 4. Log Identifier (LID)
Legend for "Status" column: "x" => can issue, "--" => cannot issue, " "(blank) => not confirmed
| LID | Description | Status | Note |
|---|---|---|---|
| 00h | Supported Log Pages | -- | error code = 1117 (The request could not be performed because of an I/O device error.) |
| 01h | Error Information | x | Number of entries to be retrieved from devices depends on the value "ELPE" in Identify Controller data |
| 02h | SMART / Health Information | x | CNTID = 0, NSID = 0xFFFFFFFh |
| 03h | Firmware Slot Information | x | |
| 04h | Changed Namespace List | ||
| 05h | Command Supported and Effects | x | |
| 06h | Device Self-test | x | |
| 07h | Telemetry Host-initiated | x | Getting header only. see also Note 1. |
| 08h | Telemetry Controller-initiated | x | Getting header only. |
| 09h | Endurance Group Information | x | |
| 0Ah | Predictable Latency Per NVM Set | ||
| 0Bh | Predictable Latency Event Aggregate | ||
| 0Ch | Asymmetric Namespace Access | ||
| 0Dh | Persistent Event Log | ||
| 0Eh | LBA Status Information | ||
| 0Fh | Endurance Group Event Aggregate | -- | error code = 1117 (The request could not be performed because of an I/O device error.) |
| 10h | Media Unit Status | ||
| 11h | Supported Capacity Configuration List | ||
| 12h | Feature Identifiers Supported and Effects | ||
| 13h | NVMe-MI Commands Supported and Effects | ||
| 14h | Command and Feature Lockdown | ||
| 15h | Boot Partition | ||
| 16h | Rotational Media Information | ||
| 17h -- 6Fh | Reserved | ||
| 70h | Discovery | For NVMe over Fabrics (NVMeoF) | |
| 71h -- 7Fh | Reserved for NVMeoF | ||
| 80h | Reservation Notification | Probably cannot issue [8] | |
| 81h | Sanitize Status | Probably can issue only in WinPE mode [8] | |
| 82h -- BFh | Reserved | ||
| C0h -- FFh | Vendor Specific |
Note 1: If you want to set Create Telemetry Host-Initiated Data to 1, you need Windows 10 version 2004 (build 10041) or later. And to do this, this tool uses IOCTL_STORAGE_GET_DEVICE_INTERNAL_LOG described in [9]. But I don't know products (SSDs) that can respond this type of request (command)...
Feature Identifier (FID) for Get/Set Features command
According to [8], only Host Controlled Thermal Management is supported via Set Features command with IOCTL_STORAGE_SET_PROPERTY.
Table 5. Feature Identifier (FID)
Legend for "Status" column: "x" => can issue, "--" => cannot issue, " "(blank) => not confirmed
| FID | Description | Status (Get/Set) | Note |
|---|---|---|---|
| 01h | Arbitration | x / | |
| 02h | Power Management | x / | |
| 03h | LBA Range Type | -- / | error code = 1117 (The request could not be performed because of an I/O device error.) |
| 04h | Temperature Threashold | x / | |
| 05h | Error Recovery | x / | |
| 06h | Volatile Write Cache | x / -- | error code = 1117 (The request could not be performed because of an I/O device error.) |
| 07h | Number of Queues | -- / | error code = 1117 (The request could not be performed because of an I/O device error.) |
| 08h | Interrupt Coalescing | x / | |
| 09h | Interrupt Vector Configuration | / | |
| 0Ah | Write Atomicity Normal | x / | |
| 0Bh | Asynchronous Event Configuration | x / | |
| 0Ch | Autonomous Power State Transition | -- / | error code = 1117 (The request could not be performed because of an I/O device error.) |
| 0Dh | Host Memory Buffer | -- / | error code = 1117 (The request could not be performed because of an I/O device error.) |
| 0Eh | Timestamp | -- / | error code = 1117 (The request could not be performed because of an I/O device error.) |
| 0Fh | Keep Alive Timer | / | |
| 10h | Host Controlled Thermal Management | x / x | "SV = 1" is not supported |
| 11h | Non-operational Power State Config | x / | |
| 12h | Read Recovery Level Config | / | |
| 13h | Predictable Latency Mode Config | / | |
| 14h | Predictable Latency Mode Window | / | |
| 15h | LBA Status Information Report Interval | / | |
| 16h | Host Behavior Support | / | |
| 17h | Sanitize Config | / | |
| 18h | Endurance Group Event Configuration | / | |
| 19h | I/O Command Set Profile | / | |
| 1Ah | Spinup Control | / | |
| 1Bh -- 77h | Reserved | / | |
| 78h -- 7Ch | Refer to the NVMe-MI spec. | / | |
| 7Dh | Enhanced Controller Metadata | / | |
| 7Eh | Controller Metadata | / | |
| 7Fh | Namespace Metadata | / | |
| 80h | Software Progress Marker | x / | |
| 81h | Host Identifier | / | |
| 82h | Reservation Notification Mask | / | 1 |
| 83h | Reservation Persistence | / | 1 |
| 84h | Namespace Write Protect Config | / | |
| 85h -- BFh | Reserved | / | |
| C0h -- FFh | Vendor specific | / |
Note 1: FID 82h and 83h are seemed to be unable to issue, because commands related to Reservation are not supported according to [8].
Note for some commands
Read command
Read command is issued via SCSI Pass-through[3].
Issue Read command with Starting Logical Block Address (SLBA) = 0 and Number of Logical Blocks (NLB) = 1.
Write command
Write command is issued via SCSI Pass-through[3].
Issue Write command with SLBA = 0, NLB = 1, and data is "DEADBEEF".
Dataset Management command
Dataset Management command is issued via SCSI Pass-through[3] with IOCTL_STORAGE_MANAGE_DATA_SET_ATTRIBUTES [4][5] (only Deallocate bit is set).
LBA region to be deallocated is SLBA = 0 and NLB = 1 (Sector 0).
An NVMe drive that supports Dataset Management command is also needed.
Supporting Dataset Management command can be checked by bit 2 of "Optional NVM Command Support (ONCS)" field in Identify Controller Data Structure.
If the bit is set to 1, the controller support Dataset Management command.
For further information, see "5.15.3 Identify Controller data structure (CNS 01h)" in the NVMe specification[2].
An NVMe drive that return fixed data pattern for reading deallocated logical block is recommended for checking the result of deallocation.
It can be checked by bit 2:0 of "Deallocate Logical Block Features (DLFEAT)" field in Identify Namespace Data Structure.
If it is 001b, the drive returns all bytes cleared to 00h for reading deallocated logical block.
If it is 010b, the drive returns all bytes set to FFh for reading deallocated logical block.
For further information, see "5.15.2 Identify Namespace data structure (CNS 00h)" in the NVMe specification[2].
Caution
Issuing "Write", "Dataset Management (Deallocate)", and "Format NVM" command to devices under management by file systems (e.g. FAT, NTFS) will corrupt your system.
These commands should be issued with highest care.
"Read" command does not have such a side effect... :-)
Environment
Confirmed on the following software environment:
- Tested operating system and device driver (latest)
- Windows 11 Pro 64bit (Version 22H2, Build 22621.963)
- stornvme.sys (version 10.0.22621.755, WinBuild 160101.0800)
- Developing environment
- Microsoft Visual Studio Community 2022 (Version 17.4.3)
- Windows SDK 10.0.22621.0
- Windows Driver Kit 10.0.22605.1
Limitations
Only tested with the NVMe drive directly attached to PC via PCIe.
It is ok to access to M.2 drives and M.2 drives that attached to M.2-PCIe converter Add-In-Card (AIC).
Also, it may be ok with U.2 drives.
But it may not work over protocol translations such as usb-nvme.
To build and run program
- Double click
NVMeTool.slnto start Visual Studio - Build program with Build -> Build Solution (or Control+Shift+B)
- Run program (F5)
Note that, when you run (debug) in Visual Studio, confirm that the command line argument is set to appropriate drive ID for your system.
You can find the settings of command line arguments from Debug menu -> NVMeTool Debug Properties -> Configuration Properties -> Debugging -> Command Arguments.
Future Works
- Test commands, LIDs, FIDs, and CNSs in the NVMe specification and listed in [8]
- Test in WinPE mode
License
This software is released under the MIT License and Microsoft Public License (MS-PL), see LICENSE.
Reference
[1] Microsoft, "Working with NVMe drives", https://docs.microsoft.com/en-us/windows/desktop/fileio/working-with-nvme-devices, Retrieved May 14, 2019
[2] NVM Express, "NVM Express[TM] Base Specification", Revision 1.3d, March 20, 2019
[3] Microsoft, "SCSI Pass-Through Interface Tool", https://github.com/microsoft/Windows-driver-samples/tree/master/storage/tools/spti, Retrieved May 14, 2019.
[4] Mark E. Russinovich, et al., "Windows Internals, Part 2", 6th Edition, October 2012, Microsoft Press, ISBN: 978-0735665873
[5] Microsoft, _"IOCTL_STORAGE_MANAGE_DATA_SETATTRIBUTES IOCTL", https://docs.microsoft.com/ja-jp/windows/desktop/api/winioctl/ni-winioctl-ioctl_storage_manage_data_set_attributes, Retrieved May 14, 2019
[6] NVM Express, "NVM Express[TM] Base Specification", Revision 1.4b, Sept. 21, 2020
[7] Microsoft, "NVMe Features Supported by StorNVMe", https://docs.microsoft.com/en-us/windows-hardware/drivers/storage/nvme-features-supported-by-stornvme, Retrieved June 25, 2020
[8] Microsoft, "StorNVMe Command Set Support", https://docs.microsoft.com/en-us/windows-hardware/drivers/storage/stornvme-command-set-support, Retrieved June 25, 2020
[9] Microsoft, "NVMe Features Supported by StorNVMe", https://docs.microsoft.com/en-us/windows-hardware/drivers/storage/stornvme-feature-support, Retrieved June 29, 2020
[10] NVM Express, "NVM Express[TM] Base Specification", Revision 2.0, June 2, 2021