search

openzfsonwindows / openzfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
494 stars 18 forks source link

Driver verifier reports DRIVER_VERIFIER_IOMANAGER_VIOLATION during uninstall. #102

Closed datacore-rm closed 2 years ago

datacore-rm commented 2 years ago

Driver verifier reports the below violation during driver uninstall (zfsinstaller.exe uninstall..).

Issue:- zfsintaller.cpp/send_zfs_ioc_unregister_fs() { (1) DeviceIoControl(ToIoctldeviceobject, ZFS_IOC_UNREGISTER_FS...); Driver=> dispatcher()=>zfs_ioc_unregister_fs(){ delete ioctlDeviceObject; ioctlDeviceObject= NULL } (2) CloseHandle(ToIoctldeviceobject); =>IRP_MJ_CLOSE=>dispatcher() { if (DeviceObject == ioctlDeviceObject) //condition is false as ioctlDeviceObject = NULL in (1) so it calls storport dispatcher() which returns failure(STATUS_UNSUCCESSFUL) and does not complete this IRP_MJ_CLOSE IRP. } }

2: kd> !analyze -v

Debugging Details:

KEY_VALUES_STRING: 1

Key  : Analysis.CPU.Sec
Value: 3

Key  : Analysis.DebugAnalysisProvider.CPP
Value: Create: 8007007e on RM-LT1

Key  : Analysis.DebugData
Value: CreateObject

Key  : Analysis.DebugModel
Value: CreateObject

Key  : Analysis.Elapsed.Sec
Value: 3

Key  : Analysis.Memory.CommitPeak.Mb
Value: 80

Key  : Analysis.System
Value: CreateObject

ADDITIONAL_DEBUG_TEXT: Bugcheck data extracted from VfErrorBugcheckData.

BUGCHECK_CODE: c9

BUGCHECK_P1: 226

BUGCHECK_P2: fffff8015ef6c5e0

BUGCHECK_P3: ffffaf8ca5496ea0

BUGCHECK_P4: 0

DRIVER_VERIFIER_IO_VIOLATION_TYPE: 226

IRP_ADDRESS: ffffaf8ca5496ea0

DEVICE_OBJECT: ffffaf8ca587f290

DRIVER_OBJECT: ffffaf8ca5880e20

IMAGE_NAME: ZFSin.sys

MODULE_NAME: ZFSin

FAULTING_MODULE: fffff8015ecc0000 ZFSin

PROCESS_NAME: zfsinstaller.exe

STACK_TEXT:
ffff91033b9ef630 fffff80158da7c1f : 0000000000000226 ffffaf8ca5944010 ffffaf8ca5496ea0 0000000000000000 : nt!ViErrorFinishReport+0x8e ffff91033b9ef690 fffff801593ce3ad : ffffaf8cc9221950 ffffaf8cc9c9baa0 0101000000000000 ffffaf8ca587f290 : nt!ViErrorReport1+0x63 ffff91033b9ef730 fffff801593cf08e : ffffaf8cc9c9baa0 0000000000000000 ffffaf8cc9c9bb58 0000000000000000 : nt!IovpCallDriver2+0x1f1 ffff91033b9ef780 fffff801593c2f5a : ffffaf8ca5496ea0 ffffaf8ca587f290 ffffaf8cc9191080 fffff80158c9c5f9 : nt!VfAfterCallDriver+0x1da ffff91033b9ef7b0 fffff80158c9b9bb : ffffaf8cca577c00 0000000000000000 ffffaf8ca5496ea0 ffffaf8cc9c9baa0 : nt!IovCallDriver+0x286 ffff91033b9ef7f0 fffff80159094834 : 0000000000000001 ffffaf8cca577c00 ffffaf8ca5496ea0 0000000000000000 : nt!IofCallDriver+0x184e6b ffff91033b9ef830 fffff8015909d520 : 0000000000000001 0000000000000000 ffffaf8ca34f6220 0000000000000001 : nt!IopDeleteFile+0x124 ffff91033b9ef8b0 fffff80158b73009 : 0000000000000000 0000000000000000 0000000000000001 ffffaf8cca577c00 : nt!ObpRemoveObjectRoutine+0x80 ffff91033b9ef910 fffff801590bae70 : 00000000ffff8001 ffffaf8ca34f6220 ffff9b8500000000 0000000000007fff : nt!ObfDereferenceObjectWithTag+0xc9 ffff91033b9ef950 fffff801590bf71e : ffffaf8cc87ee080 0000000000000000 0000000000000000 0000002933afed68 : nt!ObCloseHandleTableEntry+0x270 ffff91033b9efa90 fffff80158c63305 : ffffaf8cc87ee080 0000000000000001 ffff91033b9efb80 ffffaf8cc9a59160 : nt!NtClose+0xde ffff91033b9efb00 00007ffed663f944 : 00007ffed26aea05 0000000000000000 0000000000000000 0000002933aff57c : nt!KiSystemServiceCopyEnd+0x25 0000002933aff4e8 00007ffed26aea05 : 0000000000000000 0000000000000000 0000002933aff57c 0000000000000000 : ntdll!NtClose+0x14 0000002933aff4f0 00007ff7df2cb0e1 : 0000000000000000 00007ff7df35530a 0000000000000000 0000000000000000 : KERNELBASE!CloseHandle+0x45 0000002933aff520 0000000000000000 : 00007ff7df35530a 0000000000000000 0000000000000000 0000000000000000 : zfsinstaller+0xb0e1

FAULTING_SOURCE_LINE: C:\mergezfsdestroy\openzfs\module\os\windows\zfs\zfs_vnops_windows.c

FAULTING_SOURCE_FILE: C:\mergezfsdestroy\openzfs\module\os\windows\zfs\zfs_vnops_windows.c

FAULTING_SOURCE_LINE_NUMBER: 5469

FAULTING_SOURCE_CODE:
5465: NTSTATUS 5466: dispatcher( 5467: In PDEVICE_OBJECT DeviceObject, 5468: Inout PIRP Irp)

5469: { 5470: BOOLEAN TopLevel = FALSE; 5471: BOOLEAN AtIrqlPassiveLevel; 5472: PIO_STACK_LOCATION IrpSp; 5473: NTSTATUS Status = STATUS_NOT_IMPLEMENTED; 5474: uint64_t validity_check;

SYMBOL_NAME: ZFSin!dispatcher+0

STACK_COMMAND: .thread ; .cxr ; kb

BUCKET_ID_FUNC_OFFSET: 0

FAILURE_BUCKET_ID: 0xc9_226_VRF_ZFSin!dispatcher

OS_VERSION: 10.0.17763.1

BUILDLAB_STR: rs5_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

FAILURE_ID_HASH: {fdbbfeaf-1cfe-b169-9ca0-ccb342e39d1d}

Followup: MachineOwner

2: kd> !irp ffffaf8ca5496ea0 Irp is active with 1 stacks 1 is current (= 0xffffaf8ca5496f70) No Mdl: No System Buffer: Thread ffffaf8cc87ee080: Irp stack trace.
cmd flg cl Device File Completion-Context

[IRP_MJ_CLOSE(2), N/A(0)] 0 0 ffffaf8ca587f290 ffffaf8cca577c00 00000000-00000000
\Driver\ZFSin Args: 00000000 00000000 00000000 00000000

Irp Extension present at 0xffffaf8ca5496fb8:

datacore-rm commented 2 years ago

Tested by moving the ioctldeviceobject cleanup to IRP_MJ_CLOSE and no DV violations reported.

ioctlDispatcher() { case IRP_MJ_CLOSE: Status = zfsdev_release((dev_t)IrpSp->FileObject, Irp);

    //uninstall
    extern kmutex_t zfsdev_state_lock;
    if (fsDiskDeviceObject == NULL) {
        mutex_enter(&zfsdev_state_lock);
        if (ioctlDeviceObject != NULL) {
        ObDereferenceObject(ioctlDeviceObject);
        IoDeleteDevice(ioctlDeviceObject);
        ioctlDeviceObject = NULL;
        }
        mutex_exit(&zfsdev_state_lock);
    }

}

lundman commented 2 years ago

Huh interesting - thanks for finding this!

datacore-rm commented 2 years ago

Any suggestions on the fix or for testing?

lundman commented 2 years ago

Is moving the release into MJ_CLOSE unattractive? I'm not sure I see any other way around it, if you need to wait for all handles to be released after having been signalled to unload. I used an ioctl() for unload as it is what I know, what do other drivers do to trigger unload?

datacore-rm commented 2 years ago

The current ioctl also checks if pools are exported. This way the changes are minimal. If ioctl device object is not deleted then I think driver unload function is not triggered. When ZFS root device is uninstalled from installer, IRP_MN_QUERY_REMOVE_DEVICE & IRP_MN_REMOVE_DEVICE is received.

lundman commented 2 years ago

I think it is acceptable to do as proposed, an indicator we are unloading, and on last CLOSE do so. Did you want you want to PR it, or happy with me copying your suggestion?