Deadlock in win_iocp_io_context::shutdown in threaded DLL global variable destruction

[gcerretani]

Versions

Windows 10 Visual Studio 2019 ASIO 1.18.2 (Boost 1.76), but tested also with 1.12.1 (Boost 1.67).

How to replicate it

I have a DLL with this interface

// libtest.h
#pragma once
extern "C" void test();

and this source code

//libtest.cpp
#include "libtest.h"

#include <boost/asio.hpp>
#include <thread>

#include <Windows.h>

void run(boost::asio::io_context& ctx) {
    using work_guard_type = boost::asio::executor_work_guard<boost::asio::io_context::executor_type>;
    work_guard_type work_guard(ctx.get_executor());
    ctx.run();
}

struct foo {
    static foo& instance() { // singleton
        static foo instance;
        return instance;
    }
private:
    foo() :
        _ctx(),
        _thrd([&c = _ctx] { run(c); }) {}
    ~foo() {
        _thrd.join();
    }

    boost::asio::io_context _ctx;
    std::thread _thrd;
};

void test() {
    foo::instance();
}

BOOL APIENTRY DllMain(HMODULE hModule, DWORD  ul_reason_for_call, LPVOID lpReserved) {
    switch (ul_reason_for_call) {
    case DLL_PROCESS_ATTACH:
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}

ASIO is used as header only library, defining with these preprocessors definitions:

• _WIN32_WINNT=0x0603
• BOOST_ALL_NO_LIB
• BOOST_ERROR_CODE_HEADER_ONLY
• BOOST_SYSTEM_NO_DEPRECATED

Then, just create an application that invokes test():

// main.cpp
#include "libtest.h"
int main() {
    test();
}

The library global destruction remains locked on a call to GetQueuedCompletionStatus in win_iocp_io_context::shutdown. The reason seems that the global variable destruction happens just after DllMain invoked with DLL_PROCESS_DETACH. At that point, according to the ExitProcess documentation:

• All of the threads in the process, except the calling thread, terminate their execution without receiving a DLL_THREAD_DETACH notification. [...] Note that returning from the main function of an application results in a call to ExitProcess.

This forced thread shutdown seems to leave _ctx in a unconsistent state, and the ~io_context() get blocked in a call to GetQueuedCompletionStatus() because internal field outstanding_work_ is not zero.

I'm not sure this issue is related #431, even they have some points in common.

My use case

For extended information, I explain you my use case. Actually, my application has a more complex code, with a boost::asio::ip::tcp::socket member of the foo class, connected to a remote client. I have a another DLL API function that disconnects the socket and properly destroies the context. I get this problem if the user forget to invoke this API function. In this case, currently I use a workaround that consists in invoke boost::asio::basic_socket::close(boost::system::error_code& ec) on the destructor. In this case, close fails with ec.value() == WSANOTINITIALISED ant, in this case, I completely abort the DLL deinitialization with a call to std::_Exit from ~foo(). Probably it is not the most elegant solution, but at least it seems to work.

[chevonc]

I'm hitting this same issue as well. Any suggestions to solve without a hacky work-around would be appreciated @chriskohlhoff

[chevonc]

@gcerretani took a deeper look into your approach, and I came across the following:

When handling DLL_PROCESS_DETACH, a DLL should free resources such as heap memory only if the DLL is being unloaded dynamically (the lpReserved parameter is NULL).

So, it seems like it's possible to use the DLL_PROCESS_DETACH and check lpReserved to determine if the process is terminating

[gcerretani]

Thanks a lot @chevonc for this further analysis on Win32 documentations, it seems that also Microsoft suggest to skip deinitialization in this case.

It can be useful also for others, so I add here the full citation, from DllMain documentation:

When a DLL is unloaded from a process as a result of an unsuccessful load of the DLL, termination of the process, or a call to FreeLibrary, the system does not call the DLL's entry-point function with the DLL_THREAD_DETACH value for the individual threads of the process. The DLL is only sent a DLL_PROCESS_DETACH notification. DLLs can take this opportunity to clean up all resources for all threads known to the DLL.

When handling DLL_PROCESS_DETACH, a DLL should free resources such as heap memory only if the DLL is being unloaded dynamically (the lpReserved parameter is NULL). If the process is terminating (the lpvReserved parameter is non-NULL), all threads in the process except the current thread either have exited already or have been explicitly terminated by a call to the ExitProcess function, which might leave some process resources such as heaps in an inconsistent state. In this case, it is not safe for the DLL to clean up the resources. Instead, the DLL should allow the operating system to reclaim the memory.

If you terminate a process by calling TerminateProcess or TerminateJobObject, the DLLs of that process do not receive DLL_PROCESS_DETACH notifications. If you terminate a thread by calling TerminateThread, the DLLs of that thread do not receive DLL_THREAD_DETACH notifications.

[gcerretani]

In other words, it does not seems an ASIO bug, because, according to Windows, it's up to the user to skip the resource clean up in this cases.