Wrong layout of Shell32.STRRET structure in x64 process

First some background, as this is probably not very common use case. I'm writing Namespace extension for windows explorer. As part of it I have own implementation for IShellFolder. Implementation of GetDisplayNameOf() method returns STRRET structure.

I'm replacing some of our native structure declarations with one's provided by Vanara.PInvoke.* libraries and faced issue when I've attempted to replace our declaration of STRRET with one, declared in Vanara.PInvoke.Shell32.STRRET.

Did some investigation and it looks like the issue is caused by the wrong layout of the Vanara structure when it is created in x64 process (in my case inside explorer.exe)

I have this structure declared as

[StructLayout(LayoutKind.Sequential, Size = 264)]
public struct StrRet
{
    public STRRET_TYPE uType;
    public IntPtr pOleStr;
}

(it is rather simple because I needed only pOleStr member of union for my needs)

Vanara has more generic declaration with all members declared with explicit offset (simplified, as I'm focusing only on layout)

[StructLayout(LayoutKind.Explicit, Size = 264)]
public struct STRRET
{
    [FieldOffset(0)]
    public STRRET_TYPE uType;

    [FieldOffset(4)]
    public StrPtrUni pOleStr; 

    [FieldOffset(4)]
    public uint uOffset; // Offset into SHITEMID

    [FieldOffset(4)]
    public StrPtrAnsi cStr;
}

When I've tried to replace my structure with Vanara - shell stopped rendering display names of objects in my NSE. Once I switch to my structure everything works fine again.

I've written a simple code to check the layout of structures and it appears that in x64 process, in my structure actual offset of pOleStr field is 8, not 4: (you need to remove Prefer32 check mark in console project build properties)

var strS = new StrRet();
var strE = new STRRET();
byte* addr = (byte*) &strS;
Console.WriteLine($"Ptr size: {IntPtr.Size}");

Console.WriteLine("Sequential:");
Console.WriteLine("Size:      {0}", sizeof(StrRet));
Console.WriteLine("pOleStr Offset: {0}", (byte*)&strS.pOleStr - addr);

addr = (byte*) &strE;
Console.WriteLine("Explicit:");
Console.WriteLine("Size:      {0}", sizeof(STRRET));
Console.WriteLine("pOleStr Offset: {0}", (byte*)&strE.pOleStr - addr);

Results:

Ptr size: 8
Sequential:
Size:      264
pOleStr Offset: 8
Explicit:
Size:      264
pOleStr Offset: 4

So I have to stick with our in-house declaration of STRRET structure for the time been. And I propose to eventually fix the declaration of this structure in Vanara.Shell32 library to properly support x64 processes

One possible way to resolve this in Vanara project and support all members of union is to implement union as a substructure, to avoid LayoutKind.Explicit on STRRET. Something like below:

[StructLayout(LayoutKind.Sequential)]
public struct STRRET2
{
    public STRRET_TYPE uType;

    public STRRET_UNION union;
}

[StructLayout(LayoutKind.Explicit, Size = 260)]
public struct STRRET_UNION
{
    [FieldOffset(0)] public IntPtr pOleStr;

    [FieldOffset(0)]
    public uint uOffset; // Offset into SHITEMID

    [FieldOffset(0)]
    public IntPtr cStr;
}

Thanks for great set of libraries, and that you keep evolving! Hope this would help to make them even better.

dahall / Vanara

Wrong layout of Shell32.STRRET structure in x64 process #453