Can't start .NET Core 2.2.1 host: "ESP was not properly saved across function call"

[RayKoopa]

• using the .NET Core Runtime 2.2.1 x86 Binaries
• new VC++ x86 project in VS2017
• running Windows 10.0.17763.253 x64

I try to run a .NET Core host from a new project (porting my old COM hosting API code to the newer CoreClrHost.h functionality), but when I call coreclr_initialize_ptr, it fails with the following Visual Studio message:

Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. 
This is usually a result of calling a function declared with one calling convention with a
function pointer declared with a different calling convention.

I can't seem to ignore this message, debugging loops getting access violation exceptions afterwards.

Weirder, I can reproduce this issue with the official SampleHost, but there it seems like I can ignore this message and continue executing the code, just to get the message again at core_clr_create_delegate. Is this supposed to happen?

If of interest, I steamed down my own code to the following (latest C++ features):

#include <filesystem>
#include <string>
#include <Windows.h>
#include "CoreClrHost.h"

using namespace std;
using namespace std::experimental::filesystem;

// ---- Helper methods extracted for this sample ----

std::wstring Utf8ToUtf16(std::string const& utf8)
{
    if (utf8.empty())
        return std::wstring();
    int utf16Length = MultiByteToWideChar(CP_UTF8, 0, &utf8[0], (int)utf8.size(), NULL, 0);
    std::wstring utf16(utf16Length, 0);
    MultiByteToWideChar(CP_UTF8, 0, &utf8[0], (int)utf8.size(), &utf16[0], utf16Length);
    return utf16;
}

std::string Utf16ToUtf8(std::wstring const& utf16)
{
    if (utf16.empty())
        return std::string();
    int utf8Length = WideCharToMultiByte(CP_UTF8, 0, &utf16[0], (int)utf16.size(), NULL, 0, NULL, NULL);
    std::string utf8(utf8Length, 0);
    WideCharToMultiByte(CP_UTF8, 0, &utf16[0], (int)utf16.size(), &utf8[0], utf8Length, NULL, NULL);
    return utf8;
}

string ConcatenatePaths(vector<string> const& paths)
{
    string concatenatedPaths;
    for (string path : paths)
    {
        concatenatedPaths.append(canonical(path).string());
        concatenatedPaths.append(";");
    }
    return concatenatedPaths;
}

vector<string> GetAssembliesFromFolder(string const& folderPath)
{
    vector<string> assemblies;
    for (auto& item : directory_iterator(folderPath))
    {
        path path = item.path();
        string extension = path.extension().string();
        transform(extension.begin(), extension.end(), extension.begin(), ::tolower);
        if (extension == ".dll")
            assemblies.push_back(path.string());
    }
    return assemblies;
}

// ---- The important stuff ----

int main()
{
    wchar_t moduleFileName[MAX_PATH];
    GetModuleFileName(NULL, moduleFileName, MAX_PATH);
    string exeFolder = path(Utf16ToUtf8(moduleFileName)).parent_path().string();
    string runtimeFolder = (exeFolder / "runtime").string();

    // Load the CoreCLR library and retrieve the required functions.
    auto coreClr = LoadLibraryEx(
        Utf8ToUtf16((runtimeFolder / "coreclr.dll").string()).c_str(),
        NULL,
        0);
    if (!coreClr)
        return -1;

    auto fnInitialize = (coreclr_initialize_ptr)GetProcAddress(
        (HMODULE)coreClr,
        "coreclr_initialize");
    if (!fnInitialize)
        return -1;

    // Configure the AppDomain settings.
    auto platformAssembliesPaths = ConcatenatePaths(GetAssembliesFromFolder(runtimeFolder));
    const char* propertyKeys[] = { "TRUSTED_PLATFORM_ASSEMBLIES" };
    const char* propertyValues[] = { platformAssembliesPaths.c_str() };

    // Start the runtime and the AppDomain.
    void* host;
    unsigned int domainId;
    // Trying to run this makes VS spit out above message and I can't ignore it.
    if (fnInitialize(
        exeFolder.c_str(),
        "My AppDomain",
        sizeof(propertyKeys) / sizeof(char*),
        propertyKeys,
        propertyValues,
        &host,
        &domainId) < 0)
    {
        return -1;
    }
}

I checked the paths, the loaded library and procedure addresses, they're all correct, only the call at the end fails. Is there anything special I have to set up in my project to make this work?

[janvorli]

It seems that your project uses a different calling convention than the one that coreclr uses. We use the stdcall calling convention. Could you please try to add __stdcall to the CORECLR_HOSTING_API macro in the coreclrhost.h and see if it fixes the problem?

#define CORECLR_HOSTING_API(function, ...) \
    extern "C" int __stdcall function(__VA_ARGS__); \
    typedef int (*function##_ptr)(__VA_ARGS__)

[RayKoopa]

I added __stdcall to CoreClrHost.h as specified, but the issues remain.

Apparently, my new VC++ project uses __cdecl, being the default for new projects in VS2017. I changed it to __stdcall in the project properties, and now it seems to work.

I'm just surprised it doesn't work with the explicit calling convention in the CoreClrHost.h... thanks for the help!