native: Ability to set a WSI programmatically

When a program uses DXVK-Native, currently it must set the environment variable DXVK_WSI_DRIVER=SDL2 or similar, as documented in the README. This can be done by a wrapper script, or by calling setenv() or putenv() at the beginning of main(), for example something like this in a C++ executable that uses SDL2:

int main(int argc, char** argv)
{
#ifndef _WIN32
  if (::setenv("DXVK_WSI_DRIVER", "SDL2", 1) < 0) {                               
    std::cerr << "setenv:" << std::strerror(errno) << std::endl;                  
    return 1;                                                                     
  }                                                                               
#endif

  ...

However, setting environment variables like this inherits down through the process hierarchy: for example if a game launcher uses SDL2 and DXVK-Native, then every game launched by it would inherit DXVK_WSI_DRIVER=SDL2, even if the game has nothing to do with SDL2 and is actually using GLFW. This seems error-prone: the fact that the parent process (launcher) uses SDL2 for its HWND implementation is a fact about the parent process only, and generally not a true fact about its child processes.

One solution that might be suggested is to unset the environment variable after initializing DXVK, but that usually isn't safe, because environ is shared state with no specific locking, therefore it is unsafe to edit the environment in one thread while another thread might be calling getenv(). Many libraries call getenv() in functions that might be called from a non-main thread, for example anything that uses gettext() for localization, with the result that the only place that the environment can safely be altered is right at the beginning of main(), before the first non-main thread has been created.

Would it be possible to add a DXVK-specific way to set the WSI in use, as non-inheritable process-global state, analogous to SDL_SetHint()? Perhaps something like this?

int main (int argc, char **argv)
{
  HWND window;

#ifndef _WIN32
  dxvkInitWSI("SDL2");
#endif

  ... carry on with initialization ...

#ifdef _WIN32
  window = ... create a Win32 window ...
#else
  window = ... create a SDL2 window ...
#endif

  ... continue with application code ...
}

Or alternatively, a way to tell DXVK "my HWND is actually a SDL2 SDL_Window" via parameters to CreateDeviceEx or CreateSwapChainForHwnd() or equivalent, or by calling a DXVK-specific method on the Device, or something of that sort (whatever layer is the most sensible one), so that you could do something like this?

  Com<ID3D11Device> device;

  HRESULT status = D3D11CreateDevice(...);

#ifndef _WIN32
  device->dxvkSetWSI("SDL2");
#endif

  ... continue with application code ...

cc @flibitijibibo

doitsujin / dxvk

native: Ability to set a WSI programmatically #4093