microsoft / wdkmetadata

Tooling to generate metadata for Win32 APIs in the Windows Driver Kit (WDK).
Other
96 stars 10 forks source link

Invalid handle types #31

Closed kennykerr closed 1 year ago

kennykerr commented 1 year ago

Further testing revealed some invalid handle types that make certain APIs inexpressible. I'm not too familiar with the WDK but it seems that all or some of these underscore-prefixed structs should be handles:

image

For example: https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-ioinitializeworkitem

Here the parameter is named PIO_WORKITEM and this typedef should be a pointer to an opaque structure defined by the driver. I would expect it to be defined something like this:

[NativeTypedef]
public struct PIO_WORKITEM
{
    public IntPtr Value;
}

And then the empty _IO_WORKITEM struct should be removed. I assume many of the other underscore-prefixed structs fall into the same category, but again I'm not that familiar with the WDK.

kennykerr commented 1 year ago

@mikebattista A simple way to smoke out these types, both in win32 and wdk metadata, is to look for structs that don't have any fields. There are quite a few across both the Win32 and WDK metadata and this could be used to quickly weed out invalid struct definitions in both repos. They're not always handle types but they're almost always an indicator of some invalid struct scraping. Here are just a few that are obviously wrong:

Searching for structs with zero fields will quickly bring them all in view.

kennykerr commented 1 year ago

Ignoring those that have a Guid attribute, here's what I find. Less than 200 types:


Windows.Wdk.Foundation.DEVICE_OBJECT_POWER_EXTENSION
Windows.Wdk.Foundation.ECP_LIST
Windows.Wdk.Foundation.EJOB
Windows.Wdk.Foundation.IO_TIMER
Windows.Wdk.Foundation.KPROCESS
Windows.Wdk.Foundation.KTHREAD
Windows.Wdk.Foundation.OBJECT_TYPE
Windows.Wdk.Graphics.Direct3D.OBJECT_ATTRIBUTES
Windows.Wdk.Storage.FileSystem._REAL_NOTIFY_SYNC
Windows.Wdk.Storage.FileSystem.ECP_HEADER
Windows.Wdk.Storage.FileSystem.SOCKADDR_STORAGE
Windows.Wdk.Storage.FileSystem.SspiAsyncContext
Windows.Wdk.System.SystemServices._ADAPTER_OBJECT
Windows.Wdk.System.SystemServices._AFFINITY_TOKEN
Windows.Wdk.System.SystemServices._BUS_HANDLER
Windows.Wdk.System.SystemServices._CALLBACK_OBJECT
Windows.Wdk.System.SystemServices._DEVICE_HANDLER_OBJECT
Windows.Wdk.System.SystemServices._DISK_PARTITION
Windows.Wdk.System.SystemServices._EPROCESS
Windows.Wdk.System.SystemServices._ETHREAD
Windows.Wdk.System.SystemServices._EX_RUNDOWN_REF_CACHE_AWARE
Windows.Wdk.System.SystemServices._EX_TIMER
Windows.Wdk.System.SystemServices._IO_REMOVE_LOCK_TRACKING_BLOCK
Windows.Wdk.System.SystemServices._IO_WORKITEM
Windows.Wdk.System.SystemServices._KINTERRUPT
Windows.Wdk.System.SystemServices._LOADER_PARAMETER_BLOCK
Windows.Wdk.System.SystemServices._PCW_BUFFER
Windows.Wdk.System.SystemServices._PCW_INSTANCE
Windows.Wdk.System.SystemServices._PCW_REGISTRATION
Windows.Wdk.System.SystemServices._SCSI_REQUEST_BLOCK
Windows.Wdk.System.SystemServices._SILO_MONITOR
Windows.Wdk.System.SystemServices.CREATE_DISK
Windows.Wdk.System.SystemServices.DMA_COMMON_BUFFER_VECTOR
Windows.Wdk.System.SystemServices.DRIVE_LAYOUT_INFORMATION
Windows.Wdk.System.SystemServices.DRIVE_LAYOUT_INFORMATION_EX
Windows.Wdk.System.SystemServices.IMAGE_NT_HEADERS32
Windows.Wdk.System.SystemServices.IMAGE_NT_HEADERS64
Windows.Wdk.System.SystemServices.IOMMU_DMA_DEVICE
Windows.Wdk.System.SystemServices.IOMMU_DMA_DOMAIN
Windows.Wdk.System.SystemServices.KENLISTMENT
Windows.Wdk.System.SystemServices.KRESOURCEMANAGER
Windows.Wdk.System.SystemServices.KTM
Windows.Wdk.System.SystemServices.KTRANSACTION
Windows.Wdk.System.SystemServices.PEB
Windows.Wdk.System.SystemServices.SET_PARTITION_INFORMATION_EX
Windows.Web.Http.Diagnostics.HttpDiagnosticsContract
Windows.Win32.Data.HtmlHelp.IITGroup
Windows.Win32.Data.HtmlHelp.IITQuery
Windows.Win32.Data.HtmlHelp.IITStopWordList
Windows.Win32.Devices.AllJoyn._alljoyn_abouticon_handle
Windows.Win32.Devices.AllJoyn._alljoyn_abouticonobj_handle
Windows.Win32.Devices.AllJoyn._alljoyn_abouticonproxy_handle
Windows.Win32.Devices.BiometricFramework._WINIBIO_ENGINE_CONTEXT
Windows.Win32.Devices.BiometricFramework._WINIBIO_SENSOR_CONTEXT
Windows.Win32.Devices.BiometricFramework._WINIBIO_STORAGE_CONTEXT
Windows.Win32.Devices.HumanInterfaceDevice._HIDP_PREPARSED_DATA
Windows.Win32.Globalization.UBiDi
Windows.Win32.Globalization.UBiDiTransform
Windows.Win32.Globalization.UBreakIterator
Windows.Win32.Globalization.UCaseMap
Windows.Win32.Globalization.UCharsetDetector
Windows.Win32.Globalization.UCharsetMatch
Windows.Win32.Globalization.UCollationElements
Windows.Win32.Globalization.UCollator
Windows.Win32.Globalization.UConstrainedFieldPosition
Windows.Win32.Globalization.UConverter
Windows.Win32.Globalization.UConverterSelector
Windows.Win32.Globalization.UCPMap
Windows.Win32.Globalization.UDateFormatSymbols
Windows.Win32.Globalization.UDateIntervalFormat
Windows.Win32.Globalization.UEnumeration
Windows.Win32.Globalization.UFieldPositionIterator
Windows.Win32.Globalization.UFormattedDateInterval
Windows.Win32.Globalization.UFormattedList
Windows.Win32.Globalization.UFormattedNumber
Windows.Win32.Globalization.UFormattedNumberRange
Windows.Win32.Globalization.UFormattedRelativeDateTime
Windows.Win32.Globalization.UFormattedValue
Windows.Win32.Globalization.UGenderInfo
Windows.Win32.Globalization.UHashtable
Windows.Win32.Globalization.UIDNA
Windows.Win32.Globalization.UListFormatter
Windows.Win32.Globalization.ULocaleData
Windows.Win32.Globalization.ULocaleDisplayNames
Windows.Win32.Globalization.UMutableCPTrie
Windows.Win32.Globalization.UNormalizer2
Windows.Win32.Globalization.UNumberFormatter
Windows.Win32.Globalization.UNumberingSystem
Windows.Win32.Globalization.UPluralRules
Windows.Win32.Globalization.URegion
Windows.Win32.Globalization.URegularExpression
Windows.Win32.Globalization.URelativeDateTimeFormatter
Windows.Win32.Globalization.UResourceBundle
Windows.Win32.Globalization.USearch
Windows.Win32.Globalization.USet
Windows.Win32.Globalization.USpoofChecker
Windows.Win32.Globalization.USpoofCheckResult
Windows.Win32.Globalization.UStringPrepProfile
Windows.Win32.Globalization.UStringSearch
Windows.Win32.Graphics.Direct3D11.CD3D11_VIDEO_DEFAULT
Windows.Win32.Graphics.DirectDraw._DD_DESTROYDRIVERDATA
Windows.Win32.Graphics.DirectDraw._DD_GETVPORTAUTOFLIPSURFACEDATA
Windows.Win32.Graphics.DirectDraw._DD_SETMODEDATA
Windows.Win32.Graphics.DirectDraw._DDFXROP
Windows.Win32.Graphics.OpenGL.GLUnurbs
Windows.Win32.Graphics.OpenGL.GLUquadric
Windows.Win32.Graphics.OpenGL.GLUtesselator
Windows.Win32.Media.KernelStreaming.IKsAllocator
Windows.Win32.Media.KernelStreaming.IKsAllocatorEx
Windows.Win32.Media.KernelStreaming.IKsPin
Windows.Win32.Networking.Clustering._HCHANGE
Windows.Win32.Networking.Clustering._HCLUSCRYPTPROVIDER
Windows.Win32.Networking.Clustering._HCLUSENUM
Windows.Win32.Networking.Clustering._HCLUSENUMEX
Windows.Win32.Networking.Clustering._HCLUSTER
Windows.Win32.Networking.Clustering._HGROUP
Windows.Win32.Networking.Clustering._HGROUPENUM
Windows.Win32.Networking.Clustering._HGROUPENUMEX
Windows.Win32.Networking.Clustering._HGROUPSET
Windows.Win32.Networking.Clustering._HGROUPSETENUM
Windows.Win32.Networking.Clustering._HNETINTERFACE
Windows.Win32.Networking.Clustering._HNETINTERFACEENUM
Windows.Win32.Networking.Clustering._HNETWORK
Windows.Win32.Networking.Clustering._HNETWORKENUM
Windows.Win32.Networking.Clustering._HNODE
Windows.Win32.Networking.Clustering._HNODEENUM
Windows.Win32.Networking.Clustering._HNODEENUMEX
Windows.Win32.Networking.Clustering._HREGBATCH
Windows.Win32.Networking.Clustering._HREGBATCHNOTIFICATION
Windows.Win32.Networking.Clustering._HREGBATCHPORT
Windows.Win32.Networking.Clustering._HREGREADBATCH
Windows.Win32.Networking.Clustering._HREGREADBATCHREPLY
Windows.Win32.Networking.Clustering._HRESENUM
Windows.Win32.Networking.Clustering._HRESENUMEX
Windows.Win32.Networking.Clustering._HRESOURCE
Windows.Win32.Networking.Clustering._HRESTYPEENUM
Windows.Win32.Networking.Ldap.LDAPSearch
Windows.Win32.Networking.WindowsWebServices.WS_CHANNEL
Windows.Win32.Networking.WindowsWebServices.WS_ERROR
Windows.Win32.Networking.WindowsWebServices.WS_HEAP
Windows.Win32.Networking.WindowsWebServices.WS_LISTENER
Windows.Win32.Networking.WindowsWebServices.WS_MESSAGE
Windows.Win32.Networking.WindowsWebServices.WS_METADATA
Windows.Win32.Networking.WindowsWebServices.WS_OPERATION_CONTEXT
Windows.Win32.Networking.WindowsWebServices.WS_POLICY
Windows.Win32.Networking.WindowsWebServices.WS_SECURITY_CONTEXT
Windows.Win32.Networking.WindowsWebServices.WS_SECURITY_TOKEN
Windows.Win32.Networking.WindowsWebServices.WS_SERVICE_HOST
Windows.Win32.Networking.WindowsWebServices.WS_SERVICE_PROXY
Windows.Win32.Networking.WindowsWebServices.WS_XML_BUFFER
Windows.Win32.Networking.WindowsWebServices.WS_XML_READER
Windows.Win32.Networking.WindowsWebServices.WS_XML_WRITER
Windows.Win32.Networking.WinSock.RIO_BUFFERID_t
Windows.Win32.Networking.WinSock.RIO_CQ_t
Windows.Win32.Networking.WinSock.RIO_RQ_t
Windows.Win32.NetworkManagement.Dhcp.DHCP_CLIENT_SEARCH_UNION
Windows.Win32.NetworkManagement.Dhcp.DHCP_OPTION_ELEMENT_UNION
Windows.Win32.NetworkManagement.Dhcp.DHCP_OPTION_SCOPE_UNION6
Windows.Win32.NetworkManagement.Dhcp.DHCP_SUBNET_ELEMENT_UNION
Windows.Win32.NetworkManagement.Dhcp.DHCP_SUBNET_ELEMENT_UNION_V4
Windows.Win32.NetworkManagement.Dhcp.DHCP_SUBNET_ELEMENT_UNION_V6
Windows.Win32.Security.Authentication.Identity._HMAPPER
Windows.Win32.Storage.Imapi._MSGSESS
Windows.Win32.Storage.IscsiDisc._ADAPTER_OBJECT
Windows.Win32.Storage.Vss.IVssExamineWriterMetadata
Windows.Win32.System.AddressBook._WABACTIONITEM
Windows.Win32.System.Com.IContext
Windows.Win32.System.Com.IEnumContextProps
Windows.Win32.System.Com.StructuredStorage.PMemoryAllocator
Windows.Win32.System.Iis._IIS_CRYPTO_BLOB
Windows.Win32.System.RemoteManagement.WSMAN_API
Windows.Win32.System.RemoteManagement.WSMAN_COMMAND
Windows.Win32.System.RemoteManagement.WSMAN_OPERATION
Windows.Win32.System.RemoteManagement.WSMAN_SESSION
Windows.Win32.System.RemoteManagement.WSMAN_SHELL
Windows.Win32.System.Rpc._NDR_ASYNC_MESSAGE
Windows.Win32.System.Rpc._NDR_CORRELATION_INFO
Windows.Win32.System.Rpc._NDR_PROC_CONTEXT
Windows.Win32.System.Rpc.NDR_ALLOC_ALL_NODES_CONTEXT
Windows.Win32.System.Rpc.NDR_POINTER_QUEUE_STATE
Windows.Win32.System.Search.IRowsetExactScroll
Windows.Win32.System.Services._SC_NOTIFICATION_REGISTRATION
Windows.Win32.System.SystemServices.AtlThunkData_t
Windows.Win32.System.SystemServices.TEB
Windows.Win32.System.SystemServices.TP_CLEANUP_GROUP
Windows.Win32.System.SystemServices.TP_POOL
Windows.Win32.System.WindowsProgramming._D3DHAL_CALLBACKS
Windows.Win32.System.WindowsProgramming._D3DHAL_GLOBALDRIVERDATA
Windows.Win32.System.WinRT._RO_REGISTRATION_COOKIE
Windows.Win32.System.Wmi.MI_Module_Self
Windows.Win32.UI.Shell._APPCONSTRAIN_REGISTRATION
Windows.Win32.UI.Shell._APPSTATE_REGISTRATION
Windows.Win32.UI.Shell.PropertiesSystem.SERIALIZEDPROPSTORAGE
Windows.Win32.Web.InternetExplorer.HTMLPersistEvents
Windows.Win32.Web.InternetExplorer.LayoutRectEvents
Windows.Win32.Web.MsHtml.HostDialogHelper
mikebattista commented 1 year ago

Thanks for the list. Many of these are defined as typedef struct _HGROUP *HGROUP. How should those be handled?

Should we treat them as native typedefs? What's the value type for it, IntPtr?

kennykerr commented 1 year ago

Yes, that's a standard pattern for an opaque handle. So in that example, the typedef is HGROUP with underlying IntPtr.

mikebattista commented 1 year ago

We workaround these by manually defining and excluding things.

@tannergooding can ClangSharp be enlightened here? Right now it just outputs empty structs.

At least could it output a suggested remapping for all of these opaque handles when log-potential-typedef-remappings is specified?

mikebattista commented 1 year ago

If I run ClangSharp with exclude-empty-records I get the below for the SDK which looks pretty close to your list. We still need to workaround these manually or at least remap to IntPtr. It would be great if ClangSharp could suggest these kinds of remappings.

_ADAPTER_OBJECT
_alljoyn_abouticon_handle
_alljoyn_abouticonobj_handle
_alljoyn_abouticonproxy_handle
_APPCONSTRAIN_REGISTRATION
_APPSTATE_REGISTRATION
_DD_DESTROYDRIVERDATA
_DD_GETVPORTAUTOFLIPSURFACEDATA
_DD_SETMODEDATA
_EPROCESS
_HCHANGE
_HCLUSCRYPTPROVIDER
_HCLUSENUM
_HCLUSENUMEX
_HCLUSTER
_HGROUP
_HGROUPENUM
_HGROUPENUMEX
_HGROUPSET
_HGROUPSETENUM
_HMAPPER
_HNETINTERFACE
_HNETINTERFACEENUM
_HNETWORK
_HNETWORKENUM
_HNODE
_HNODEENUM
_HNODEENUMEX
_HREGBATCH
_HREGBATCHNOTIFICATION
_HREGBATCHPORT
_HREGREADBATCH
_HREGREADBATCHREPLY
_HRESENUM
_HRESENUMEX
_HRESOURCE
_HRESTYPEENUM
_MSGSESS
_NDR_ASYNC_MESSAGE
_NDR_CORRELATION_INFO
_NDR_PROC_CONTEXT
_WINIBIO_ENGINE_CONTEXT
_WINIBIO_SENSOR_CONTEXT
_WINIBIO_STORAGE_CONTEXT
AUTHZ_CAP_CHANGE_SUBSCRIPTION_HANDLE__
CD3D11_VIDEO_DEFAULT
GLUnurbs
GLUquadric
GLUtesselator
HDEVQUERY__
HDRVDIALOGINSTANCE__
HIORING__
HTAPICALL__
HTAPILINE__
HTAPIPHONE__
IITGroup
IKsAllocatorEx
IVssExamineWriterMetadata
LDAPSearch
MachineGlobalObjectTableRegistrationToken__
MI_Module_Self
NDR_ALLOC_ALL_NODES_CONTEXT
NDR_POINTER_QUEUE_STATE
NET_ADDRESS_INFO
PACKAGE_VIRTUALIZATION_CONTEXT_HANDLE__
PACKAGEDEPENDENCY_CONTEXT__
PMemoryAllocator
RIO_BUFFERID_t
RIO_CQ_t
RIO_RQ_t
SERIALIZEDPROPSTORAGE
SPSTATEHANDLE__
TrackerHandle__
UBiDi
UBiDiTransform
UBreakIterator
UCaseMap
UCharsetDetector
UCharsetMatch
UCollationElements
UCollator
UConstrainedFieldPosition
UConverter
UConverterSelector
UCPMap
UDateIntervalFormat
UEnumeration
UFieldPositionIterator
UFormattedDateInterval
UFormattedList
UFormattedNumber
UFormattedRelativeDateTime
UFormattedValue
UGenderInfo
UIDNA
UListFormatter
ULocaleData
ULocaleDisplayNames
UMutableCPTrie
UNormalizer2
UNumberFormatter
UNumberingSystem
UPluralRules
URegion
URegularExpression
URelativeDateTimeFormatter
UResourceBundle
USet
USpoofChecker
USpoofCheckResult
UStringPrepProfile
UStringSearch
WS_CHANNEL
WS_ERROR
WS_HEAP
WS_LISTENER
WS_MESSAGE
WS_METADATA
WS_OPERATION_CONTEXT
WS_POLICY
WS_SECURITY_CONTEXT
WS_SECURITY_TOKEN
WS_SERVICE_HOST
WS_SERVICE_PROXY
WS_XML_BUFFER
WS_XML_READER
WS_XML_WRITER
WSMAN_API
WSMAN_COMMAND
WSMAN_OPERATION
WSMAN_SESSION
WSMAN_SHELL
mikebattista commented 1 year ago

The metadata now builds with exclude-empty-records on so any introduction of empty structs going forward will cause a build break.