AllocConsole and freopen mess up input/output operations

[sokolas]

I'm writing a Win32 application in C++ (with WinMain entry point) that should also allocate and display a console, and I can't get any console input/output working in Lua. Here's what I have:

#include <windows.h>
#include "sol.hpp"

void redirect()
{
    if (AllocConsole()) {
        SetConsoleCP(CP_UTF8);
        SetConsoleOutputCP(CP_UTF8);
        FILE* file = nullptr;
        freopen_s(&file, "CONOUT$", "wt", stdout);
        freopen_s(&file, "CONOUT$", "wt", stderr);
        freopen_s(&file, "CONIN$", "rt", stdin);
        SetConsoleTitle(L"Debug Console");
        SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), FOREGROUND_GREEN | FOREGROUND_BLUE | FOREGROUND_RED);
        SetConsoleTextAttribute(GetStdHandle(STD_ERROR_HANDLE), FOREGROUND_GREEN | FOREGROUND_BLUE | FOREGROUND_RED);
    }
}

int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance,
    PSTR lpCmdLine, int nCmdShow)
{
    redirect();

    std::cout << "hello world from C++\n"; // works fine, I get output in the allocated console

    sol::state lua;
    lua.open_libraries();
    int c = lua.script("print('fsdfs'); return 10"); // doesn't output anything
    std::cout << c << std::endl; // prints "10" so I assume Lua is working
    lua.script("io.write('abcd');"); // doesn't output anything
    lua.script("local f = io.open('CONOUT$', 'w'); f:write('1234');"); // doesn't output anything

    return 0;
}

If I change WinMain to main and subsystem from windows to console I get the output working.

I'm using Sol3 from the latest release (3.3.0), Visual Studio 2022 (17.7.0) community edition, and LuaJIT 2.1 compiled with that version of MSVC as a shared DLL. Both DLL and exe are compiled for x64.

[sokolas]

My bad, it has nothing to do with Sol, it was because LUaJIT and the main exe were compiled with different release/debug configurations.