search

ziglang / zig

General-purpose programming language and toolchain for maintaining robust, optimal, and reusable software.
https://ziglang.org
MIT License
34.38k stars 2.51k forks source link

ideas to improve windows header files and libc #9998

Open marler8997 opened 2 years ago

marler8997 commented 2 years ago

Here's what MinGW provides for Zig:

(Let me know if I'm missing something here)

The problems I've been having with MinGW

Every software has bugs, but MinGW's main problem here is that it's difficult, time consuming and slow to fix them. I think if this wasn't the case, I would have no issue continuing to use MinGW and submitting patches.

So what can we do? A big chunk of work is going to be getting C/C++ headers for the win32 APIs. Because Microsoft has released the https://github.com/microsoft/win32metadata project, and we are already using it to generate Zig bindings, we can enhance that project to generate C/C++ headers as well. I anticipate the initial work to support this would be just a few weeks.

That leaves us with the 2nd chunk of work, an implementation of libc for Windows. This work is already planned (https://github.com/ziglang/zig/issues/2879). With this it may be feasible to drop MinGW with work that's already planned and leveraging existing tools.

Thoughts, concerns?

P.S. Long term goal here is to completely replace the need for Visual Studio using this proposal and https://github.com/ziglang/zig/issues/8973

SpexGuy commented 2 years ago

There are at least a couple other things mingw provides:

  1. .def files listing symbols in win32 dll files so that we can link with them
  2. build support via zig c++ for libraries that support mingw as a target

The first is something we could probably generate from the win32 metadata, but there may be hidden dragons in 2. Particularly, C libraries that assume that windows && !MSVC means mingw, and use mingw-specific header declarations.

marler8997 commented 2 years ago

Yeah def files are trivial to generate with win32metadata. I didn't really consider C++, there could definitely be some gotchas there.

michal-z commented 2 years ago

@marler8997 Wouldn't it be easier to maintain our own fork of Mingw-w64?

Unfortunately https://github.com/microsoft/win32metadata also has bugs. Currently there are some critical errors for some DX APIs.

Also, looking at: https://github.com/mingw-w64/mingw-w64/commits/master it seems that development is pretty active. Have you tried reporting issues there?

andrewrk commented 2 years ago

This is quite an ambitious and interesting topic. I like the energy. Let's go over all the pieces here individually; there is a lot to consider.

it seems to have alot of bugs

Can you elaborate on this? Seems like you have some insight that I am not privy to.

it's difficult to contribute to (patches over email, non-responsive, can take months to integrate changes)

I share your preference for more "modern" systems for patches than email. However, if I'm being honest, it can take months for Zig pull requests to get merged too, sometimes. We have a few open since May of this year and that's on me, I really need to spend the time to figure it out and get those PRs merged or reviewed or otherwise addressed. But point being, can we do better than MinGW on this one metric? Maybe not.

I've also had some really pleasant interactions with Martin Storsjö when I asked for help on the MinGW-w64 on IRC. I do think that upstream is willing to work with us, and I personally see that relationship as valuable. This particular point, I do not see as compelling. However...

MinGW's goals don't always align with Zig's, so sometimes it doesn't make sense for them to fix things that Zig needs

I'm pretty sure I agree with you, but I'm curious what in particular you are considering.

IMO, it's not "compatible enough" with MSVC so most projects need to add explicit support for MinGW. MinGW doesn't care to be more compatible and is OK with projects needing to put in work to support it alongside MSVC.

I'm also curious to hear more details about this. I've used mingw-w64 in the past for various projects, and found its binary compatibility with MSVC to be flawless. Maybe I just got "lucky" but either way, it would help to have some more information on this axis to help move this issue along.

Every software has bugs, but MinGW's main problem here is that it's difficult, time consuming and slow to fix them. I think if this wasn't the case, I would have no issue continuing to use MinGW and submitting patches.

So, I'm actually finding myself kind of disagreeing with your reasoning here, but agreeing with your conclusion regardless. Idea being that if we can fulfill the Windows target features, as you listed them (implementation of libc for windows, .def files for windows, C/C++ headers for windows), without depending on this third party project, that would be an improvement.

Let's address each use case independently:

implementation of libc for windows

Indeed, there is a plan to do this already: #2879

Even if we stick with mingw-w64, the plan is to reduce the number of files that we depend on. For example, at the very least, all the math functions could be common and shared between all of zig's libc targets.

Even if this issue were not accepted, any progress towards replacing components of mingw-w64 libc with zig libc files would be welcome.

.def files for windows

Yeah def files are trivial to generate with win32metadata.

Neat! That's good to know.

C/C++ headers for windows

Because Microsoft has released the https://github.com/microsoft/win32metadata project, and we are already using it to generate Zig bindings, we can enhance that project to generate C/C++ headers as well. I anticipate the initial work to support this would be just a few weeks.

This is a cool idea - I hadn't considered that this is possible. Do you think it has a realistic chance of working, in the sense that it could generate a viable windows.h file that would be compatible with existing software? It might be worth doing a proof-of-concept for this to explore whether this is feasible.

@michal-z's point is important:

Unfortunately https://github.com/microsoft/win32metadata also has bugs. Currently there are some critical errors for some DX APIs.

That's good to know. One of the benefits mingw-w64 provides is that it has people using it and curating fixes to stuff like this. So we would be taking on that labor burden if we tried to do it with win32metadata ourselves. It could still be done, but that's something to consider. If win32metadata has bugs, that can be worked around, by us having "patch" data on top of it, that applies before lowering win32metadata to .h file(s). This is similar to the strategy that we do with tools/update_cpu_features.zig which applies some adjustments on top of LLVM's CPU features table data.

Conclusion

Seems like we should start with migrating logic from mingw-w64 to zig libc and then re-evaluate after that.

P.S. Long term goal here is to completely replace the need for Visual Studio using this proposal and #8973

Does the existing mingw-w64 integrations not already solve this problem? If not, can we list some explicit problems with status quo?

Anyway, seems like we have some stuff to explore. My goals are:

I think I forgot a couple points, but hopefully this is enough to keep the issue moving forward.

marler8997 commented 2 years ago

Andrew thanks for taking the time to respond with all your thoughts. I think we're mostly in agreement here. I think I'm going to go ahead with enhancing my zigwin32gen project to generate C/C++ headers first. This will be an easy (and fun) task for me that will give us alot of information about the feasibility of decreasing dependence on MinGW, and getting an idea of what benefits may come from it as well. In simple terms, this task is generating all the files in lib/libc/include/any-windows-any.

Below you'll find some of my thoughts and experiences. Since I think we're already in agreement, it doesn't seem necessary but I've included it anyway since you asked for it.

Thoughts and Experiences (unnecessary reading)

The "seems to have alot of bugs" statement comes from trying to use "zig cc" on various projects and getting header file errors from the mingw headers. For example I've come across this one a few times:

2021-08-04T11:54:57.4224911Z /snap/zig/3678/lib/libc/mingw/secapi/vsprintf_s.c:39:10: warning: implicit declaration of function '__ms_vsnprintf' is invalid in C99 [-Wimplicit-function-declaration]
2021-08-04T11:54:57.4225714Z   return __ms_vsnprintf (_DstBuf, _Size, _Format, _ArgList);

I'm guessing a portion of the errors I've seen over time were because of "zig cc" and not mingw itself. It just "seems" to me that it tends to have bugs for simple things that it shouldn't have, non-complicated use cases. Note that my use of the word "seems" here is intentional and comes from my limited experience. But even if mingw isn't all that buggy, Zig's usage of it (manual inclusion of particular files) is, and its usage is brittle since it's not supported by MinGW. I don't think MinGW is going to put in effort into supporting Zig's use case (and why would they) so this "bugginess" which results from the combination of Zig is the reality of our situation.

There's a lot of work left to do on the Zig windows toolchain, and when I see a problem with mingw itself, I know it's going to take along time to fix. For example, I submitted a 2-line change to fix how they were including a particular header. This change was needed in order to allow projects to have header files named "rpc.h", something which MSVC allows. One particular developer was against the change (an example of not caring much about MSVC compatibility), but Martin said the change was "ok" but it was in a header file shared with Wine, so Martin asked another contributor to forward the discussion to them. It's now been 6 weeks since then and the change hasn't been merged and there's been no follow up on whether the discussion was forwarded to Wine. I pinged them 3 days ago so maybe I'll hear back soon. If I was submitting a change like this to Zig, I wouldn't be scratching my head as to what's going on 6 weeks later.

I'll also point out that fixes to MinGW are dependent on the mingw release schedule. This could be mitigated with a pre-release patch system, but I'm weighing that work against what it would take to release or decrease dependence on mingw.

Marler(me): MinGW's goals don't always align with Zig's, so sometimes it doesn't make sense for them to fix things that Zig needs

Andrew: I'm pretty sure I agree with you, but I'm curious what in particular you are considering.

I should explain my motivation here. For some reason, the idea that Zig could be a replacement for Visual Studio is what excites me. My goal is to get Zig to become a drop-in replacement for msvc in as many projects as possible. I think dependence on MinGW is going to hold us back here. MinGW seems to have lower aspirations. For example, when I submitted a patch to fix an issue with some WSA inline function, rather than try to match what MSVC was doing which would have resulted in the most compatible solution, they were more concerned about modifying their headers and potentially breaking projects that depended on mingw's alternative behavior. I think this is a valid approach for MinGW to have, but I don't think it's the right approach for Zig. If we want to be a drop-in replacement, then compatibility with MSVC would be more important than supporting projects who have become dependent on our particular toolchain idiosyncrasies.

marler8997 commented 2 years ago

@michal-z

Wouldn't it be easier to maintain our own fork of Mingw-w64?

Short term definitely. Long term, almost certainly not. Long term we should already have a common libc we can leverage, and zigwin32gen/win32metadata can also be leveraged for most everything else. I'm not sure about libc++, but we could still leverage MinGW for that if we were so inclined.

Unfortunately https://github.com/microsoft/win32metadata also has bugs. Currently there are some critical errors for some DX APIs.

True, but they're "usually" quick to fix them and when we don't want to wait for them, I've already got systems in place to patch them.

Patching data like win32metadata is easier than patching a codebase like MinGW.

y0shir commented 2 years ago

That doesn't mean dropping optional compatibility with MinGW? Just because microsoft compiler is the "dominant" one doesn't mean that there isn't many great projects that depend on MinGW in some form.

slimsag commented 2 years ago

@marler8997

I am discovering mingw-w64 headers are missing key updates to the Direct3D 12 headers, and the latest D3D12 headers from the Windows SDK are incompatible with mingw-w64's COM definitions etc. so I cannot intermix them. This is preventing me from building Dawn / WebGPU on Windows.

I saw your project above and I found your genc PR https://github.com/marlersoft/zigwin32gen/pull/10 but I am curious how far along this work is, and if you still plan to work on it further?

andrewrk commented 2 years ago

@slimsag can you check mingw-w64 master branch and see if those headers are present and have the desired updates?

git clone git://git.code.sf.net/p/mingw-w64/mingw-w64

List of changed files:

 mingw-w64-crt/Makefile.am                          |   357 +-
 mingw-w64-crt/Makefile.in                          | 35120 ++++++++-----------
 mingw-w64-crt/configure                            |    65 +
 mingw-w64-crt/configure.ac                         |    31 +
 mingw-w64-crt/crt/charmax.c                        |     2 +-
 mingw-w64-crt/crt/crtdll.c                         |     4 +-
 mingw-w64-crt/crt/crtexe.c                         |    30 +-
 mingw-w64-crt/crt/mingw_helpers.c                  |     2 +-
 mingw-w64-crt/crt/pseudo-reloc.c                   |    21 +-
 mingw-w64-crt/crt/tlssup.c                         |     6 +-
 mingw-w64-crt/gdtoa/strtopx.c                      |    14 +
 ...-0.def => api-ms-win-crt-convert-l1-1-0.def.in} |     8 +-
 .../lib-common/api-ms-win-crt-math-l1-1-0.def.in   |     2 +-
 mingw-w64-crt/lib-common/computecore.def           |    63 +
 mingw-w64-crt/lib-common/computenetwork.def        |    61 +
 mingw-w64-crt/lib-common/computestorage.def        |    19 +
 mingw-w64-crt/lib-common/crtdll.mri                |     5 +
 mingw-w64-crt/lib-common/msvcr100.mri              |     5 +
 mingw-w64-crt/lib-common/msvcr110.mri              |     5 +
 mingw-w64-crt/lib-common/msvcr120.mri              |     5 +
 mingw-w64-crt/lib-common/msvcr120_app.mri          |     6 +
 mingw-w64-crt/lib-common/msvcr120d.mri             |     5 +
 mingw-w64-crt/lib-common/msvcr80.mri               |     5 +
 mingw-w64-crt/lib-common/msvcr80_64.mri            |     6 +
 mingw-w64-crt/lib-common/msvcr90.mri               |     5 +
 mingw-w64-crt/lib-common/msvcr90d.mri              |     5 +
 mingw-w64-crt/lib-common/msvcrt-os.mri             |     6 +
 mingw-w64-crt/lib-common/mswsock.def               |     2 +
 mingw-w64-crt/lib-common/ntdll.def.in              |    60 +
 mingw-w64-crt/lib-common/ntquery.def               |    17 +
 mingw-w64-crt/lib-common/query.def                 |    17 +
 mingw-w64-crt/lib-common/tbs.def                   |     3 +
 mingw-w64-crt/lib-common/ucrtbase.def.in           |     8 +-
 mingw-w64-crt/lib-common/ucrtbase.mri              |     5 +
 mingw-w64-crt/lib-common/ws2_32.def.in             |     1 +
 mingw-w64-crt/lib32/Makefile.am                    |     3 +
 mingw-w64-crt/lib32/authz.def                      |    36 +-
 mingw-w64-crt/lib32/fontsub.def                    |     4 +
 mingw-w64-crt/lib32/ksuser.def                     |    11 +-
 mingw-w64-crt/lib32/mswsock.def                    |    46 +-
 mingw-w64-crt/lib32/ntdll.def                      |    55 +
 mingw-w64-crt/lib32/ntquery.def                    |    17 +
 mingw-w64-crt/lib32/query.def                      |    17 +
 mingw-w64-crt/lib32/tbs.def                        |    15 +
 mingw-w64-crt/lib32/user32.def                     |   191 +-
 mingw-w64-crt/lib32/ws2_32.def                     |    14 +
 mingw-w64-crt/lib64/Makefile.am                    |     4 +
 mingw-w64-crt/lib64/query.def                      |  1447 -
 mingw-w64-crt/lib64/winhvplatform.def              |    37 +
 mingw-w64-crt/libarm32/Makefile.am                 |     1 +
 mingw-w64-crt/libarm32/query.def                   |    51 -
 mingw-w64-crt/libarm64/Makefile.am                 |     5 +
 mingw-w64-crt/libsrc/activeds-uuid.c               |    15 +-
 mingw-w64-crt/libsrc/bits.c                        |    17 +
 mingw-w64-crt/libsrc/ksuser.c                      |     1 +
 mingw-w64-crt/math/cephes_emath.c                  |  1283 -
 mingw-w64-crt/math/cephes_emath.h                  |   719 -
 mingw-w64-crt/math/x86/asinh.c                     |    55 +-
 mingw-w64-crt/math/x86/asinhf.c                    |    17 +-
 mingw-w64-crt/math/x86/asinhl.c                    |    15 +-
 mingw-w64-crt/math/x86/atanh.c                     |     2 +-
 mingw-w64-crt/math/x86/atanhf.c                    |     2 +-
 mingw-w64-crt/math/x86/atanhl.c                    |     2 +-
 mingw-w64-crt/misc/longjmp.S                       |     3 +
 mingw-w64-crt/misc/mingw_wcstod.c                  |    18 +-
 mingw-w64-crt/misc/mingw_wcstold.c                 |    96 +-
 mingw-w64-crt/misc/setjmp.S                        |     2 +
 mingw-w64-crt/misc/strtold.c                       |   398 -
 mingw-w64-crt/misc/wcstold.c                       |    64 +-
 mingw-w64-crt/stdio/mingw_pformat.c                |  1886 +-
 mingw-w64-crt/stdio/strtof.c                       |    18 +-
 mingw-w64-crt/stdio/ucrt_vfscanf.c                 |     2 +-
 mingw-w64-headers/Makefile.am                      |    10 +
 mingw-w64-headers/Makefile.in                      |    10 +
 mingw-w64-headers/crt/_mingw.h.in                  |     6 +-
 mingw-w64-headers/crt/_mingw_mac.h                 |     4 +-
 mingw-w64-headers/crt/corecrt.h                    |     5 -
 mingw-w64-headers/crt/ctype.h                      |     2 +-
 mingw-w64-headers/crt/intrin.h                     |     4 +
 mingw-w64-headers/crt/locale.h                     |     6 +
 mingw-w64-headers/crt/setjmp.h                     |     8 +-
 mingw-w64-headers/crt/wchar.h                      |     2 +-
 mingw-w64-headers/ddk/include/ddk/ata.h            |  1732 +
 mingw-w64-headers/ddk/include/ddk/ntifs.h          |     2 +
 mingw-w64-headers/ddk/include/ddk/wdm.h            |  1144 +-
 mingw-w64-headers/include/activation.h             |     2 +-
 mingw-w64-headers/include/activaut.h               |     2 +-
 mingw-w64-headers/include/activdbg.h               |     2 +-
 mingw-w64-headers/include/activdbg100.h            |     2 +-
 mingw-w64-headers/include/activprof.h              |     2 +-
 mingw-w64-headers/include/activscp.h               |     2 +-
 mingw-w64-headers/include/adhoc.h                  |     2 +-
 mingw-w64-headers/include/afunix.h                 |    19 +
 mingw-w64-headers/include/alg.h                    |     2 +-
 mingw-w64-headers/include/amstream.h               |     2 +-
 mingw-w64-headers/include/amvideo.h                |     2 +-
 mingw-w64-headers/include/asyncinfo.h              |     2 +-
 mingw-w64-headers/include/audioclient.h            |     2 +-
 mingw-w64-headers/include/audioendpoints.h         |     2 +-
 mingw-w64-headers/include/audiopolicy.h            |     2 +-
 mingw-w64-headers/include/austream.h               |     2 +-
 mingw-w64-headers/include/bdaiface.h               |     2 +-
 mingw-w64-headers/include/bits.h                   |  1540 +-
 mingw-w64-headers/include/bits.idl                 |   418 +
 mingw-w64-headers/include/bits1_5.h                |   589 +-
 mingw-w64-headers/include/bits1_5.idl              |   109 +
 mingw-w64-headers/include/bits2_0.h                |   815 +-
 mingw-w64-headers/include/bits2_0.idl              |   100 +
 mingw-w64-headers/include/bits2_5.h                |   270 +
 mingw-w64-headers/include/bits2_5.idl              |   102 +
 mingw-w64-headers/include/bits3_0.h                |   704 +
 mingw-w64-headers/include/bits3_0.idl              |    71 +
 mingw-w64-headers/include/bits5_0.h                |   581 +
 mingw-w64-headers/include/bits5_0.idl              |    59 +
 mingw-w64-headers/include/comadmin.h               |     2 +-
 mingw-w64-headers/include/comcat.h                 |     2 +-
 mingw-w64-headers/include/commoncontrols.h         |     2 +-
 mingw-w64-headers/include/computecore.h            |    82 +
 mingw-w64-headers/include/computedefs.h            |   119 +
 mingw-w64-headers/include/computenetwork.h         |   142 +
 mingw-w64-headers/include/computestorage.h         |    40 +
 mingw-w64-headers/include/control.h                |     2 +-
 mingw-w64-headers/include/ctfutb.h                 |     2 +-
 mingw-w64-headers/include/d3d10.h                  |     2 +-
 mingw-w64-headers/include/d3d10_1.h                |     2 +-
 mingw-w64-headers/include/d3d10effect.h            |     4 +-
 mingw-w64-headers/include/d3d10sdklayers.h         |     2 +-
 mingw-w64-headers/include/d3d11.h                  |     2 +-
 mingw-w64-headers/include/d3d11_1.h                |     2 +-
 mingw-w64-headers/include/d3d11_2.h                |     2 +-
 mingw-w64-headers/include/d3d11_3.h                |     2 +-
 mingw-w64-headers/include/d3d11_4.h                |     2 +-
 mingw-w64-headers/include/d3d11on12.h              |     2 +-
 mingw-w64-headers/include/d3d11sdklayers.h         |     2 +-
 mingw-w64-headers/include/d3d12.h                  |     2 +-
 mingw-w64-headers/include/d3d12sdklayers.h         |     2 +-
 mingw-w64-headers/include/d3d12shader.h            |     2 +-
 mingw-w64-headers/include/d3dcommon.h              |     2 +-
 mingw-w64-headers/include/dbgprop.h                |     2 +-
 mingw-w64-headers/include/dcommon.h                |     2 +-
 mingw-w64-headers/include/dcompanimation.h         |     2 +-
 mingw-w64-headers/include/ddstream.h               |     2 +-
 mingw-w64-headers/include/devicetopology.h         |     2 +-
 mingw-w64-headers/include/dimm.h                   |     2 +-
 mingw-w64-headers/include/dinput.h                 |     4 +-
 mingw-w64-headers/include/dinputd.h                |     2 +-
 mingw-w64-headers/include/directmanipulation.h     |     2 +-
 mingw-w64-headers/include/dispex.h                 |     2 +-
 mingw-w64-headers/include/dmodshow.h               |     2 +-
 mingw-w64-headers/include/docobj.h                 |     2 +-
 mingw-w64-headers/include/docobjectservice.h       |     2 +-
 mingw-w64-headers/include/documenttarget.h         |     2 +-
 mingw-w64-headers/include/downloadmgr.h            |     2 +-
 mingw-w64-headers/include/drmexternals.h           |     2 +-
 mingw-w64-headers/include/dsound.h                 |    12 +-
 mingw-w64-headers/include/dvdif.h                  |     2 +-
 mingw-w64-headers/include/dwrite.h                 |     2 +-
 mingw-w64-headers/include/dwrite_1.h               |     2 +-
 mingw-w64-headers/include/dwrite_2.h               |     2 +-
 mingw-w64-headers/include/dwrite_3.h               |     2 +-
 mingw-w64-headers/include/dxgi.h                   |     2 +-
 mingw-w64-headers/include/dxgi1_2.h                |     2 +-
 mingw-w64-headers/include/dxgi1_3.h                |     2 +-
 mingw-w64-headers/include/dxgi1_4.h                |     2 +-
 mingw-w64-headers/include/dxgi1_5.h                |     2 +-
 mingw-w64-headers/include/dxgi1_6.h                |     2 +-
 mingw-w64-headers/include/dxgicommon.h             |     2 +-
 mingw-w64-headers/include/dxgidebug.h              |     2 +-
 mingw-w64-headers/include/dxgiformat.h             |     2 +-
 mingw-w64-headers/include/dxgitype.h               |     2 +-
 mingw-w64-headers/include/dxva.h                   |   269 +
 mingw-w64-headers/include/dxva2api.h               |     2 +-
 mingw-w64-headers/include/dxvahd.h                 |     2 +-
 mingw-w64-headers/include/endpointvolume.h         |     2 +-
 mingw-w64-headers/include/eventtoken.h             |     2 +-
 mingw-w64-headers/include/evr.h                    |     2 +-
 mingw-w64-headers/include/evr9.h                   |   411 +-
 mingw-w64-headers/include/evr9.idl                 |   125 +
 mingw-w64-headers/include/exdisp.h                 |     2 +-
 mingw-w64-headers/include/filter.h                 |     2 +-
 mingw-w64-headers/include/fontsub.h                |    64 +
 mingw-w64-headers/include/fsrm.h                   |     2 +-
 mingw-w64-headers/include/fsrmenums.h              |     2 +-
 mingw-w64-headers/include/fsrmquota.h              |     2 +-
 mingw-w64-headers/include/fsrmreports.h            |     2 +-
 mingw-w64-headers/include/fsrmscreen.h             |     2 +-
 mingw-w64-headers/include/fusion.h                 |     2 +-
 mingw-w64-headers/include/fwptypes.h               |     2 +-
 mingw-w64-headers/include/hstring.h                |     2 +-
 mingw-w64-headers/include/icftypes.h               |     2 +-
 mingw-w64-headers/include/icodecapi.h              |     2 +-
 mingw-w64-headers/include/iketypes.h               |     2 +-
 mingw-w64-headers/include/inputscope.h             |     2 +-
 mingw-w64-headers/include/inspectable.h            |     2 +-
 mingw-w64-headers/include/ioringapi.h              |   140 +
 mingw-w64-headers/include/locationapi.h            |     2 +-
 mingw-w64-headers/include/mediaobj.h               |     2 +-
 mingw-w64-headers/include/medparam.h               |     2 +-
 mingw-w64-headers/include/mfapi.h                  |    64 +-
 mingw-w64-headers/include/mfidl.h                  |   542 +-
 mingw-w64-headers/include/mfidl.idl                |    56 +-
 mingw-w64-headers/include/mfobjects.h              |     2 +-
 mingw-w64-headers/include/mfplay.h                 |     2 +-
 mingw-w64-headers/include/mfreadwrite.h            |     2 +-
 mingw-w64-headers/include/mftransform.h            |     2 +-
 mingw-w64-headers/include/mmdeviceapi.h            |     2 +-
 mingw-w64-headers/include/mmreg.h                  |     6 +
 mingw-w64-headers/include/mmstream.h               |     2 +-
 mingw-w64-headers/include/mscoree.h                |     2 +-
 mingw-w64-headers/include/msctf.h                  |     2 +-
 mingw-w64-headers/include/mshtmhst.h               |     2 +-
 mingw-w64-headers/include/mshtml.h                 |     2 +-
 mingw-w64-headers/include/msinkaut.h               |     2 +-
 mingw-w64-headers/include/msinkaut_i.c             |     2 +-
 mingw-w64-headers/include/msopc.h                  |     2 +-
 mingw-w64-headers/include/mstcpip.h                |   241 +-
 mingw-w64-headers/include/msxml.h                  |     2 +-
 mingw-w64-headers/include/napcertrelyingparty.h    |     2 +-
 mingw-w64-headers/include/napcommon.h              |     2 +-
 mingw-w64-headers/include/napenforcementclient.h   |     2 +-
 mingw-w64-headers/include/napmanagement.h          |     2 +-
 mingw-w64-headers/include/napprotocol.h            |     2 +-
 mingw-w64-headers/include/napservermanagement.h    |     2 +-
 mingw-w64-headers/include/napsystemhealthagent.h   |     2 +-
 .../include/napsystemhealthvalidator.h             |     2 +-
 mingw-w64-headers/include/naptypes.h               |     2 +-
 mingw-w64-headers/include/netcfgn.h                |     2 +-
 mingw-w64-headers/include/netcfgx.h                |     2 +-
 mingw-w64-headers/include/netfw.h                  |     2 +-
 mingw-w64-headers/include/netlistmgr.h             |     2 +-
 mingw-w64-headers/include/ntdddisk.h               |     6 +
 mingw-w64-headers/include/ntioring_x.h             |    54 +
 mingw-w64-headers/include/ntsecapi.h               |  1147 +-
 mingw-w64-headers/include/ntstatus.h               |     1 +
 mingw-w64-headers/include/nvme.h                   |  2835 ++
 mingw-w64-headers/include/oaidl.h                  |     2 +-
 mingw-w64-headers/include/objectarray.h            |     2 +-
 mingw-w64-headers/include/objidl.h                 |     2 +-
 mingw-w64-headers/include/objidlbase.h             |     2 +-
 mingw-w64-headers/include/ocidl.h                  |     2 +-
 mingw-w64-headers/include/oleacc.h                 |     2 +-
 mingw-w64-headers/include/oleidl.h                 |     2 +-
 mingw-w64-headers/include/optary.h                 |     2 +-
 mingw-w64-headers/include/pchannel.h               |    16 +-
 mingw-w64-headers/include/portabledevicetypes.h    |     2 +-
 mingw-w64-headers/include/processthreadsapi.h      |    97 +
 .../include/proofofpossessioncookieinfo.h          |   173 +
 .../include/proofofpossessioncookieinfo.idl        |    70 +
 mingw-w64-headers/include/propidl.h                |     2 +-
 mingw-w64-headers/include/propsys.h                |     2 +-
 mingw-w64-headers/include/propvarutil.h            |   104 +-
 mingw-w64-headers/include/psdk_inc/_dbg_common.h   |    97 +-
 mingw-w64-headers/include/psdk_inc/_ip_mreq1.h     |     9 +
 mingw-w64-headers/include/psdk_inc/intrin-impl.h   |   119 +
 mingw-w64-headers/include/qedit.h                  |     2 +-
 mingw-w64-headers/include/qnetwork.h               |     2 +-
 mingw-w64-headers/include/rdpencomapi.h            |     2 +-
 mingw-w64-headers/include/regbag.h                 |     2 +-
 mingw-w64-headers/include/relogger.h               |     2 +-
 mingw-w64-headers/include/rtworkq.h                |     2 +-
 mingw-w64-headers/include/sapi51.h                 |     2 +-
 mingw-w64-headers/include/sapi53.h                 |     2 +-
 mingw-w64-headers/include/sapi54.h                 |     2 +-
 mingw-w64-headers/include/sdkddkver.h              |     3 +-
 mingw-w64-headers/include/sensorsapi.h             |     2 +-
 mingw-w64-headers/include/servprov.h               |     2 +-
 mingw-w64-headers/include/shldisp.h                |     2 +-
 mingw-w64-headers/include/shobjidl.h               |     2 +-
 mingw-w64-headers/include/shtypes.h                |     2 +-
 mingw-w64-headers/include/spatialaudioclient.h     |     2 +-
 mingw-w64-headers/include/spellcheck.h             |     2 +-
 mingw-w64-headers/include/strmif.h                 |     2 +-
 .../include/structuredquerycondition.h             |     2 +-
 mingw-w64-headers/include/taskschd.h               |     2 +-
 mingw-w64-headers/include/textstor.h               |     2 +-
 mingw-w64-headers/include/thumbcache.h             |     2 +-
 mingw-w64-headers/include/tlbref.h                 |     2 +-
 mingw-w64-headers/include/tlogstg.h                |     2 +-
 mingw-w64-headers/include/tpcshrd.h                |     2 +-
 mingw-w64-headers/include/tsvirtualchannels.h      |  1060 +
 mingw-w64-headers/include/tsvirtualchannels.idl    |   102 +
 mingw-w64-headers/include/tuner.h                  |     2 +-
 mingw-w64-headers/include/uianimation.h            |     2 +-
 mingw-w64-headers/include/uiautomationclient.h     |   198 +-
 mingw-w64-headers/include/uiautomationclient.idl   |   107 +
 mingw-w64-headers/include/uiautomationcore.h       |   269 +-
 mingw-w64-headers/include/uiautomationcore.idl     |    67 +-
 mingw-w64-headers/include/uiviewsettingsinterop.h  |     2 +-
 mingw-w64-headers/include/unknwn.h                 |     2 +-
 mingw-w64-headers/include/unknwnbase.h             |     2 +-
 mingw-w64-headers/include/urlhist.h                |     2 +-
 mingw-w64-headers/include/urlmon.h                 |     2 +-
 mingw-w64-headers/include/vmr9.h                   |     2 +-
 mingw-w64-headers/include/vss.h                    |     2 +-
 mingw-w64-headers/include/wbemads.h                |     2 +-
 mingw-w64-headers/include/wbemcli.h                |     2 +-
 mingw-w64-headers/include/wbemdisp.h               |     2 +-
 mingw-w64-headers/include/wbemprov.h               |     2 +-
 mingw-w64-headers/include/wbemtran.h               |     2 +-
 mingw-w64-headers/include/wdstptmgmt.h             |     2 +-
 mingw-w64-headers/include/winbase.h                |    17 -
 mingw-w64-headers/include/wincodec.h               |     2 +-
 mingw-w64-headers/include/wincodecsdk.h            |     2 +-
 mingw-w64-headers/include/wincon.h                 |    30 +-
 mingw-w64-headers/include/windns.h                 |   538 +-
 .../include/windows.foundation.collections.h       |     2 +-
 mingw-w64-headers/include/windows.foundation.h     |    74 +-
 .../include/windows.media.speechsynthesis.h        |    16 +-
 .../include/windows.security.cryptography.h        |    16 +-
 mingw-w64-headers/include/windows.storage.h        |    86 +-
 .../include/windows.storage.streams.h              |     2 +-
 mingw-w64-headers/include/windows.system.h         |     2 +-
 .../include/windows.system.threading.h             |    46 +-
 mingw-w64-headers/include/windowscontracts.h       |     2 +-
 mingw-w64-headers/include/winerror.h               |     1 +
 mingw-w64-headers/include/winevt.h                 |    21 +-
 mingw-w64-headers/include/winhvplatform.h          |    37 +
 mingw-w64-headers/include/winhvplatformdefs.h      |   731 +-
 mingw-w64-headers/include/wininet.h                |    98 +-
 mingw-w64-headers/include/winineti.h               |     1 +
 mingw-w64-headers/include/winioctl.h               |   475 +-
 mingw-w64-headers/include/winnt.h                  |   123 +-
 mingw-w64-headers/include/winsock2.h               |     1 +
 mingw-w64-headers/include/winsvc.h                 |   115 +-
 mingw-w64-headers/include/winuser.h                |     5 +-
 mingw-w64-headers/include/wmcodecdsp.h             |     2 +-
 mingw-w64-headers/include/wmcontainer.h            |     2 +-
 mingw-w64-headers/include/wmdrmsdk.h               |     2 +-
 mingw-w64-headers/include/wmp.h                    |     2 +-
 mingw-w64-headers/include/wmprealestate.h          |     2 +-
 mingw-w64-headers/include/wmpservices.h            |     2 +-
 mingw-w64-headers/include/wmsbuffer.h              |   315 +-
 mingw-w64-headers/include/wmsbuffer.idl            |    22 +
 mingw-w64-headers/include/wmsdkidl.h               |   381 +-
 mingw-w64-headers/include/wmsdkidl.idl             |    45 +-
 mingw-w64-headers/include/wmsecure.h               |   472 +
 mingw-w64-headers/include/wmsecure.idl             |    62 +
 mingw-w64-headers/include/wpcapi.h                 |     2 +-
 mingw-w64-headers/include/ws2ipdef.h               |     5 -
 mingw-w64-headers/include/wsdattachment.h          |     2 +-
 mingw-w64-headers/include/wsdbase.h                |     2 +-
 mingw-w64-headers/include/wsdclient.h              |     2 +-
 mingw-w64-headers/include/wsddisco.h               |     2 +-
 mingw-w64-headers/include/wsdhost.h                |     2 +-
 mingw-w64-headers/include/wsdxml.h                 |     2 +-
 mingw-w64-headers/include/wsmandisp.h              |     2 +-
 mingw-w64-headers/include/wtypes.h                 |     2 +-
 mingw-w64-headers/include/wtypesbase.h             |     2 +-
 mingw-w64-headers/include/wuapi.h                  |   159 +-
 mingw-w64-headers/include/wuapi.idl                |    25 +
 mingw-w64-headers/include/xapo.h                   |     2 +-
 mingw-w64-headers/include/xaudio2.h                |     2 +-
 mingw-w64-headers/include/xaudio2fx.h              |     2 +-
 mingw-w64-headers/include/xmllite.h                |     2 +-
 mingw-w64-headers/include/xpsdigitalsignature.h    |     2 +-
 mingw-w64-headers/include/xpsobjectmodel.h         |     2 +-
 mingw-w64-headers/include/xpsobjectmodel_1.h       |     2 +-
 mingw-w64-headers/include/xpsprint.h               |     2 +-
 mingw-w64-headers/include/xpsrassvc.h              |     2 +-
 mingw-w64-headers/wine-import.sh                   |    11 +
 mingw-w64-tools/genpeimg/src/genpeimg.c            |    81 +-
 mingw-w64-tools/genpeimg/src/img.h                 |     3 +-
 mingw-w64-tools/genpeimg/src/img_pe.c              |    21 +-
 mingw-w64-tools/widl/VERSION                       |     2 +-
 mingw-w64-tools/widl/configure                     |    20 +-
 mingw-w64-tools/widl/include/winbase.h             |     7 +-
 mingw-w64-tools/widl/include/winerror.h            |    24 +
 mingw-w64-tools/widl/include/winnt.h               |  1229 +-
 mingw-w64-tools/widl/src/header.c                  |    24 +-
 mingw-w64-tools/widl/src/widl.c                    |   105 +-
 370 files changed, 36845 insertions(+), 28703 deletions(-)
slimsag commented 2 years ago

@andrewrk Yeah, this was the first thing I checked actually. Unfortunately, latest mingw-w64 headers do not fix this. I will see about filing issues upstream on mingw-w64

Here are some concrete examples of what I've found to be missing in the latest upstream mingw-w64-headers compared to the headers shipped with e.g. a recent Windows SDK (10.0.19041.0):

I suspect that there are many more definitions than just these missing

michal-z commented 2 years ago

Please note that latest DirectX 12 headers from Microsoft are here: https://github.com/microsoft/DirectX-Headers (those are standalone and MIT licensed).

DirectX 12 headers that come with Windows SDK have problems with functions that return structure by value (when compiled as C code).

mingw-w64 should use headers from Microsoft git repo if possible.

michal-z commented 2 years ago

d3d12.h that comes with mingw-w64 (https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-headers/include/d3d12.h) is very old (latest header provides ID3D12Device8 the one that come with mingw-w64 provides only ID3D12Device1).

@slimsag I think that your best bet is to take headers from https://github.com/microsoft/DirectX-Headers and put them into Dawn source tree.

slimsag commented 2 years ago

@michal-z The primary problem isn't that the d3d12.h is outdated, I can workaround that by including updated d3d headers as you noted. The big problem is those last two bullet points I mentioned:

These are direct dependencies of Dawn and the mingw-w64 headers are incompatible with them.

andrewrk commented 2 years ago

Note that we can carry patches to our mingw-w64 headers in the zig repo as long as the patches have been accepted upstream as well.

michal-z commented 2 years ago

@slimsag Regarding macro definitions, I think you can do something like:

#define _Maybenull_
#define __in_ecount_opt(x)

in dxcapi.h.

But, yeah, UWP headers are problematic.

andrewrk commented 2 years ago

Got one piece of info from #mingw-w64 on OFTC:

<jon_y> andrewrk: directx headers are from Wine
<jon_y> you can contribute changes to sal.h
slimsag commented 2 years ago

Lengthy write-up incoming, hoping this is useful to whoever else stumbles across this issue and attempts to solve it. After spending ~2 weeks on this, I don't feel I've made much progress and feel a bit ill at how large of a problem I now realize this may be for Zig w.r.t. cross compiling for Windows.

Hopefully some of this information is useful for us finding a long-term solution to this. It feels a bit like I've traveled inside a broken Fukushima reactor and back again, I'm planning to find some band-aids for myself and I probably won't dive back in again for now.

Here goes..

Windows header file sources

Authoritative sources for Windows API headers

We can find these in two places:

  1. The IDL files that Microsoft releases as part of the Windows SDK, which are redistributed under an MIT license here: https://github.com/microsoft/win32metadata/tree/master/generation/WinSDK/RecompiledIdlHeaders
    • But note the authoritative source is actually the Windows source code inside of Microsoft, the win32metadata repository merely has a script which copies them from the Windows SDK into the repo.
    • https://github.com/microsoft/DirectX-Headers similarly is a redistributed copy, Microsoft allows constributions to this repository to .h and .idl files, but ultimately they periodically synchronize the changes in this repository with the IDL files in the Windows internal repository and regenerate the .h files from the .idl sources found in the Windows SDK.
  2. Wine, they have a hand crafted (emphasis on that) recreation of Microsoft's IDL files. They're not the same at all as the IDL sources Microsoft distributes due to licensing concerns in the past.

MinGW-w64 may have some other headers (sal.h?) which it does not acquire from Wine, I didn't investigate this, but I strongly suspect almost all headers mingw-w64 distributes are generated from Wine's IDL files. Could be wrong though.

Non-authoritative sources for Windows API headers

Using IDL files

How IDL -> .h happens

There exist two IDL compilers which parse .idl files and generate .h files from them:

  1. MIDL, Microsoft IDL compiler, distributed with Visual Studio. Closed source.
  2. WIDL, Wine IDL compiler, found in tools/widl of the Wine source code.

IDL files are actually incredibly similar to the C header files themselves, what the generator does is quite minimal translation of syntax (but more on this below.)

Incompatibilities

Any header compiled with Microsoft's MIDL compiler, i.e. the .h files in:

Are fundamentally incompatible with mingw-w64 and gcc outright, and fundamentally incompatible with clang/zig when targeting a GNU ABI.

Header files generated using the Wine WIDL compiler (those distributed by Zig, those distributed by mingw-w64, and those in the Wine repository) are compatible with with all compilers (more on this below.)

Atrocities of COM / win32 headers

The COM calling convention differs from the C calling convention, in specific msvc has undocumented behavior in which aggregates are returned by pointer using a hidden 2nd parameter in COM C/C++ functions and methods. Clang and Zig emulate this undocumented msvc behavior when targetting msvc ABI, but not gnu ABI.

the IDL files distributed by Microsoft have attributes which denote which COM methods require a hidden parameter in order to store the return value.

How this is handled is where Wine and Microsoft headers differ greatly. When it comes to the C definitions, they are identical. For example Wine and Microsoft headers both define:

      D3D12_CPU_DESCRIPTOR_HANDLE *( STDMETHODCALLTYPE *GetCPUDescriptorHandleForHeapStart )( 
          ID3D12DescriptorHeap * This,
          D3D12_CPU_DESCRIPTOR_HANDLE * RetVal);

However, when it comes to C++ definitions Microsoft defines just:

virtual D3D12_CPU_DESCRIPTOR_HANDLE STDMETHODCALLTYPE GetCPUDescriptorHandleForHeapStart( void) = 0;

Despite what is written above, msvc and msvc-emulating compilers detect that the return value of this function is an aggregate and inject a second parameter (the return pointer.)

Wine on the other hand defines this same C++ virtual method as:

#ifdef WIDL_EXPLICIT_AGGREGATE_RETURNS
    virtual D3D12_CPU_DESCRIPTOR_HANDLE* STDMETHODCALLTYPE GetCPUDescriptorHandleForHeapStart(D3D12_CPU_DESCRIPTOR_HANDLE *__ret) = 0;
    D3D12_CPU_DESCRIPTOR_HANDLE STDMETHODCALLTYPE GetCPUDescriptorHandleForHeapStart()
    {
        D3D12_CPU_DESCRIPTOR_HANDLE __ret;
        return *GetCPUDescriptorHandleForHeapStart(&__ret);
    }
#else
    // We crash here!
    virtual D3D12_CPU_DESCRIPTOR_HANDLE STDMETHODCALLTYPE GetCPUDescriptorHandleForHeapStart() = 0;
#endif

When WIDL_EXPLICIT_AGGREGATE_RETURNS is defined, the headers are compatible with gcc, mingw-w64, and clang (when targetting gnu ABI.) Otherwise, the definition matches that found in the Microsoft headers and the headers become fundamentally broken for non-msvc-emulating compilers.

Relevant links:

Today, GCC/MinGW-w64 still cannot emulate msvc's COM calling convention behavior. Only clang can: https://github.com/llvm/llvm-project/commit/85a0f8fe6c5c8ab35790c40d078d4fa103a5a54a#diff-48f9c7403ada1feaf517d77cd12f97c5bd239d6247190543d74b463c7da242ccR1058 (here you can see the ARM logic they use to detect an aggregate requiring a hidden return-by-paramete, which is much more complex than just "uses the COM calling convention".)

@marler32 and @michal-z ran into this for example in https://github.com/microsoft/win32metadata/issues/636 and thought the Microsoft header definitions may be wrong, unfortunately not - msvc just has undocumented behavior of injecting a hidden parameter. Microsoft actually cannot alter the header definitions to match C calling convention, as msvc would still inject the secret parameter.

Me and @Andoryuuta found only one single semi-official reference to this undocumented behavior in msvc, as almost a footnote of this Raymond Chen article in 2014: https://devblogs.microsoft.com/oldnewthing/20040114-00/?p=41053

"The return value is placed in rax. If the return value is larger than 64 bits, then a secret first parameter is passed which contains the address where the return value should be stored. "

The harsh reality of all of this is that any Zig project which includes Microsoft headers and uses the default GNU ABI will compile just fine, but get a segfault when they invoke a COM method returning an aggregate as the hidden return parameter will be null.

Additionally, it means any C++ project compiled with Zig or MinGW that does not explicitly define WIDL_EXPLICIT_AGGREGATE_RETURNS themselves will find the same segfault. Truly horrific.

Failed approaches I investigated

The headers distributed in win32metadata are not sufficient

@andrewrk suggested we might be able to drop the GNU ABI support and only support msvc ABI. On that note, I checked to see if we can just use the .h headers distributed in https://github.com/microsoft/win32metadata/tree/master/generation/WinSDK/RecompiledIdlHeaders (which are MSVC compatible) and only use clang's msvc-emulating ABI target

The problem with this approach is that win32metadata is not complete: the headers included in this repository are generated by the Microsoft MIDL compiler distributed with Visual Studio and because of this they have a dependency on vcruntime.h and all the other headers distributed as part of Visual Studio, which are not open source.

  1. Even if we had a way to include those headers legally, it seems our libcxx is incompatible with those headers (at least when I tried swapping our any-windows-any headers dir for all of those): https://gist.github.com/slimsag/da3a5ffbba121289917eae5c49c5365a - I am not smart enough to resolve this.
  2. For cross-compilation, we still have no msvc runtime libraries we can redistribute. We need to depend on mingw/wine for libraries still, I'm unsure if we will run into more incompatibilities here. For example, I have seen indications MinGW-w64 and clang both may emit __mingw prefix'd references when invoking COM methods in gnu ABI targets.

Could we run the Wine WIDL compiler on Microsoft's IDL files?

Another thought I had was: what if we could merely run Wine's WIDL compiler on Microsoft's IDL files? Effectively giving us super-up-to-date headers for MinGW-w64 that could theoretically even be used by MinGW and Wine themselves.

I ran quite far with this experiment, and I believe it may be possible however I ran into two hurdles:

  1. The IDL files in https://github.com/microsoft/win32metadata/tree/master/generation/WinSDK/RecompiledIdlHeaders depend on what seems to be just two files not distributed as OSS: stdole32.tlb and stdole2.tlb which you can see are used via importlib here. Maybe we could get Microsoft to distribute these under a permissive license? Unsure.
  2. The Wine WIDL compiler does not support a whole litany of more modern syntax features that the Microsoft IDL files use. I have categorized these here but basically I think we'd either be looking at writing our own IDL compiler (which I think would not be too hard) or contributing some major changes to Wine's.

This was the approach I got furthest with. In fact, with relatively few patches I was able to use the WIDL compiler to compile most of the DirectX headers from Microsoft. I have not tested these yet, but I suspect they'll work and be compatible with MinGW/clang/zig with GNU ABIs using the WIDL_EXPLICIT_AGGREGATE_RETURNS definition hack. See recent commits in https://github.com/slimsag/win32metadata (21277d9f9c8860c75c9aee7319fe51d67a1301b7 and before)

Can we stick with wine/mingw-w64 headers and contribute to them?

This is the path of least effort, and what we're currently doing. I will say however that after what I have found here.. I feel quite confident the Wine headers are currently, and I suspect will remain for quite some time, very outdated.

Unfortunately, merging win32metadata IDL file changes into Wine's IDL files would be extremely non-trivial, too, because Wine's are hand-crafted much in the same way Microsoft's are - they differ quite a lot in syntax, ordering of definitions, and even which files symbols are defined in sometimes. Attempting to merge the two would be a seriously tedious undertaking.

andrewrk commented 2 years ago

Thank you for this really, really helpful writeup.

Could we run the Wine WIDL compiler on Microsoft's IDL files?

Despite your apparent emotional defeat, my take on this is that you successfully pioneered the way forward, and I think the Zig project can carry the torch from here. :fire:

michal-z commented 2 years ago

This looks like handcrafted, Wine-based IDL with latest interfaces (up to ID3D12Device9), maybe we could use this one? https://github.com/HansKristian-Work/vkd3d-proton/blob/master/include/vkd3d_d3d12.idl

This is basically nice and clean IDL without Microsoft specific annotations and it is also up to date. I think we should run Wine WIDL compiler on this.

slimsag commented 2 years ago

@michal-z Ah, yes, I forgot to mention vk3d-proton. I did look into this as well. I do believe they are compatible with Wine's WIDL. However, they are actually quite stripped down and do not include D3D11 or prior, only good if you care about D3D12 (which in my case is a problem as Dawn also supports D3D11)

It also doesn't solve the larger issue of mingw-w64 headers being quite out of date, I think that is worth solving.

As for a temporary workaround: it's a good solution, also note I did manage to get the WIDL compiler to build the full latest d3d headers with minor patches - buried in my message above:

In fact, with relatively few patches I was able to use the WIDL compiler to compile most of the DirectX headers from Microsoft. I have not tested these yet, but I suspect they'll work and be compatible with MinGW/clang/zig with GNU ABIs using the WIDL_EXPLICIT_AGGREGATE_RETURNS definition hack. See recent commits in https://github.com/slimsag/win32metadata (21277d9f9c8860c75c9aee7319fe51d67a1301b7 and before)

michal-z commented 2 years ago

@slimsag Using d3d11.idl from Wine and latest d3d12.idl from Proton should work, right? This would give us headers to build Dawn I suppose.

slimsag commented 2 years ago

@michal-z The workaround I quoted above (should, I think / hope) give us headers to build Dawn, I just haven't tested it yet.

But no, mixing the Wine and Proton headers does not generally work. Multiple reasons for this: (1) proton headers strip out non-d3d12 things from files shared between d3d12/d3d11, e.g. in dxgibase, d3dcommon, etc. (2) proton headers use vk3d_ prefix (easy to fix) but also in constant names, etc. (3) proton headers redeclare some symbols, e.g. vkd3d_unknown.idl redeclares a subset of what is in the upstream unknwn.idl and unknwnbase.idl which leads to conflicts.

Basically if you want to do that, you need to manually port/copy over the portions of Proton's headers that you want into the Wine headers which is non-trivial (but slightly easier than copying from Windows SDK IDL -> Wine IDL, you don't have to remove annotations etc.)

smourier commented 2 years ago

Note the "secret first parameter" trick is officially documented here in the "x64 calling convention return values": https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170#return-values

A scalar return value that can fit into 64 bits, including the m64 type, is returned through RAX. Non-scalar types including floats, doubles, and vector types such as m128, m128i, m128d are returned in XMM0. The state of unused bits in the value returned in RAX or XMM0 is undefined.

User-defined types can be returned by value from global functions and static member functions. To return a user-defined type by value in RAX, it must have a length of 1, 2, 4, 8, 16, 32, or 64 bits. It must also have no user-defined constructor, destructor, or copy assignment operator. It can have no private or protected non-static data members, and no non-static data members of reference type. It can't have base classes or virtual functions. And, it can only have data members that also meet these requirements. (This definition is essentially the same as a C++03 POD type. Because the definition has changed in the C++11 standard, we don't recommend using std::is_pod for this test.) Otherwise, the caller must allocate memory for the return value and pass a pointer to it as the first argument. The remaining arguments are then shifted one argument to the right. The same pointer must be returned by the callee in RAX.

EvineDev commented 2 years ago

Extending from the x64 calling convention, there is 2 new rules in returning an struct / class from a method.

  1. Class / Struct of 8 bytes or less are always returned by pointer. Primitive types <= 8 bytes are still returned in registers.
  2. The Class / Struct this pointer takes precedence over the return object pointer. So you'll get this in rcx and return_pointer in rdx

Now, this is news to me. It's clearly not documented on the msdn page on calling conventions. And I don't see any reason for these changes to the abi.

I made a godbolt snippet where I explored the issue: https://godbolt.org/z/1xE6YnEnx

  1. fn_ret_obj returns in registers as the abi guide describes.
  2. fn_ret_int returns in registers as the abi guide describes.
  3. method_ret_obj returns by pointer. In contrast from what you would expect.
  4. method_ret_int returns in registers. This is what you would expect with a natural extension to the abi.
michal-z commented 2 years ago

Agner Fog explains calling conventions in detail here: https://www.agner.org/optimize/calling_conventions.pdf See: Section 7.1 Passing and returning objects Table 7. Methods for returning structure, class and union objects

All possible cases are described in above document.

EvineDev commented 2 years ago

A lot great information there, but I don't see it describing returning a simple structure, class or union from a method.

If I follow Table 7, I would expect returning a 4 byte struct from a method to do so via registers. But in my godbolt link the object is returned by pointer. See the method_ret_obj disassembly.

michal-z commented 2 years ago

Methods always return objects by pointer (even small objects). This is described by Microsoft and quoted by @smourier :

Otherwise, the caller must allocate memory for the return value and pass a pointer to it as the first argument. The remaining arguments are then shifted one argument to the right. The same pointer must be returned by the callee in RAX.

Above quote applies to methods. User-defined types can be returned in a register only in a global functions and static member functions.

slimsag commented 2 years ago

Yeah, so the Microsoft "x64 calling convention return values" quote above is correct. However, I will say it is worded quite confusingly and I can explain what you are seeing @unXedDani.

In your godbolt link, the C function returning an object matches the simple structure, class or union row in the Agner Fog doc and is Returned in integer registers which matches the asm.

I am not able to find a matching P row in the Agner Fog document that @michal-z posted which matches the behavior of the C++ method returning an object, however I could be misreading the table.

It does however match the Microsoft x64 calling convention quote, sort of, anyway. We have three arguments:

  1. rcx, C++ this pointer
  2. rdx, pointer to return struct
  3. r8, the int num parameter

What is misleading about the quote is that it implies the first argument woudl be the pointer to return struct:

Otherwise, the caller must allocate memory for the return value and pass a pointer to it as the first argument. The remaining arguments are then shifted one argument to the right. The same pointer must be returned by the callee in RAX.

This quote does not consider the C++ this pointer to be an argument. Once you realize that, it's accurate.

Also want to point out that the calling convention changes for aarch64. More on this below.

Here are two sources for how to determine what calling convention to use:

My interpretation of both of these sources is that:

  1. Wine's WIDL implementation: determining if the return value should be via pointer is as simple as looking at whether or not the IDL type definition is a struct, union, coclass, interface, or runtimeclass. I couldn't tell you why Wine does not have special handling for the case of aarch64 here (maybe because it's dealing with IDL files and not C++ code, so this is unnecessary? Alternatively it could be a real bug with Wine on aarch64)
  2. Clang's implementation: Specifically of note is that whether or not the return value should be via pointer is dependent upon: (a) whether the return type is POD (non-aarch64) OR (b) whether the return type is a C++14 'aggregate' (aarch64 only) OR (c) is a C++ instance method.

If we do write our own IDL compiler, we could leverage the info in #1 above quite well.

mstorsjo commented 2 years ago

FWIW, I'll try to fill in with a bit more details on a couple parts in the earlier discussion here...

The "seems to have alot of bugs" statement comes from trying to use "zig cc" on various projects and getting header file errors from the mingw headers. For example I've come across this one a few times:

2021-08-04T11:54:57.4224911Z /snap/zig/3678/lib/libc/mingw/secapi/vsprintf_s.c:39:10: warning: implicit declaration of function '__ms_vsnprintf' is invalid in C99 [-Wimplicit-function-declaration]
2021-08-04T11:54:57.4225714Z   return __ms_vsnprintf (_DstBuf, _Size, _Format, _ArgList);

I don't see that warning when building mingw-w64-crt (testing latest master version though, but I don't remember seeing that warning in older releases either).

I'm guessing a portion of the errors I've seen over time were because of "zig cc" and not mingw itself. It just "seems" to me that it tends to have bugs for simple things that it shouldn't have, non-complicated use cases. Note that my use of the word "seems" here is intentional and comes from my limited experience.

I wouldn't say there aren't bugs - there's plenty (like any major project) - but the overall buggyness is pretty far from how you describe it.

If you see that kind of "bugs for simple things that it shouldn't have, non-complicated use cases" when you reproduce it with a regular GCC/Clang based mingw tool distribution, then please do let me/someone know. If the bugs can't be reproduced that way, then please don't describe a project as "seeming to have a lot of bugs" if the bugs are introduced by the way you configure/build it differently.

But even if mingw isn't all that buggy, Zig's usage of it (manual inclusion of particular files) is, and its usage is brittle since it's not supported by MinGW. I don't think MinGW is going to put in effort into supporting Zig's use case (and why would they) so this "bugginess" which results from the combination of Zig is the reality of our situation.

If we're talking about bugs introduced from how Zig builds mingw-w64-crt while bypassing its buildsystem, by just hardcoding 10% of what it does... Reimplementing things to use the lower level C stuff from win32metadata instead of mingw-w64 probably is an interesting goal in itself, but by the time that setup is on a similar feature level as what you have today, you will have spent way more time and code than you'd do by just replicating more of mingw-w64's build system.

Here's what MinGW provides for Zig:

  • C/C++ headers for the win32 APIs (means we can compile C/C++ code without depending on Visual Studio)
  • An implementation of libc for Windows

(Let me know if I'm missing something here)

mingw-w64 doesn't quite contain an implementation of a libc. Just to make the setting clearer... The original intent was to link code against msvcrt.dll (which is assumed to be present in the OS). To achieve this, you'd mostly just need suitable header declarations and import libraries (which can be generated from a plain symbol listing, i.e. the def files in mingw-w64). However, for a couple reasons, there's a fair bit of libc-like code that is bundled in mingw-w64 (mostly in libmingwex.a):

Now in modern environments, if targeting UCRT, the dynamically linked C runtime is C99 compliant already, so one pretty much only needs overriding functions that touch long double. (In practice, mingw-w64 might still be overriding functions that strictly aren't needed, because it still supports both msvcrt.dll and UCRT, and it's not always trivial to untangle things - the exact set of functions that need to be overridden differs a little between CRT alternatives and architectures).

A more detailed listing of what mingw-w64 provides (for Zig) is this:

Now regarding headers, WinSDK generally has been non-redistributable. But I do see that a lot of files from WinSDK are bundled in https://github.com/microsoft/win32metadata. That does indeed open up a lot of doors for doing things differently. (But as noted in https://github.com/ziglang/zig/issues/9998#issuecomment-997088467, those headers aren't enough - in MSVC mode you still need the headers that are distributed along with MSVC itself, plus the libraries that contain glue code around the UCRT import library.)

Also - while that main repo itself seems clearly licensed as MIT, it does seem a bit ambiguous whether the bundled WinSDK headers really are knowingly relicensed this way - I opened https://github.com/microsoft/win32metadata/issues/766 to get clarification of this matter.

implementation of libc for windows

Indeed, there is a plan to do this already: #2879

Even if we stick with mingw-w64, the plan is to reduce the number of files that we depend on. For example, at the very least, all the math functions could be common and shared between all of zig's libc targets.

Even if this issue were not accepted, any progress towards replacing components of mingw-w64 libc with zig libc files would be welcome.

This does seem like a worthwhile goal in itself. Also depending on whether you're linking against msvcrt.dll or UCRT, and whether you care about 80 bit long double or not, you could skip bundling a lot of code without even reimplementing anything at all. But if you'd want to link against a function from msvcrt.dll/UCRT that mingw-w64 normally replaces with a statically linked function, you'd need to patch the def files accordingly.

So this seems like a thing you can work on gradually whichever way you want, but you may need to tweak headers and def files for cases where you link things differently than upstream.

mstorsjo commented 2 years ago

Non-authoritative sources for Windows API headers

  • Any .h file you find is actually generated from the IDL file from either Windows SDK or Wine. .h files are never the authoritative source.

It would be good to clarify here that this only concerns the COM based headers/APIs. For plain Win32 API and the CRT stuff, the .h files really are the authoritative sources.

  1. Even if we had a way to include those headers legally, it seems our libcxx is incompatible with those headers (at least when I tried swapping our any-windows-any headers dir for all of those): https://gist.github.com/slimsag/da3a5ffbba121289917eae5c49c5365a - I am not smart enough to resolve this.

The errors here are quite cryptic, but the root cause is on a higher level I think. libcxx only implements the body parts of the C++ standard library; it needs to run on top of a C++ ABI library. In mingw mode (and on unix), the C++ compilers use the Itanium C++ ABI. MSVC mode uses an entirely different, MSVC specific C++ ABI. For mingw mode (and on unix), the C++ ABI implementation can be provided by either libstdc++/libsupc++, or libcxxabi. For MSVC mode, the only implementation of the MSVC C++ ABI I know of, is part of the non-opensource parts in MSVC (static libs or msvcp*.dll).

  1. For cross-compilation, we still have no msvc runtime libraries we can redistribute. We need to depend on mingw/wine for libraries still,

At this level (CRT glue code etc) there's no Wine code involved, it's all in mingw-w64, FWIW. But yeah, the symbol listings in win32metadata aren't quite enough for linking against UCRT (it gives you around 90% of what you need), as you'd still need the equivalent of msvcrt.lib or libcmt.lib which contains a bunch of glue code too.

I'm unsure if we will run into more incompatibilities here. For example, I have seen indications MinGW-w64 and clang both may emit __mingw prefix'd references when invoking COM methods in gnu ABI targets.

Hmm, I don't think there's anything in Clang automatically emitting references to __mingw_* anything unless it was present in the source?

As you've noticed, switching between MinGW and MSVC mode is kinda hard, because it controls both code generation/ABI and the source language accepted (GCC and Clang in MinGW mode can't cope with MSVC/WinSDK headers and MSVC can't cope with MinGW headers). So generally you can't just switch individual pieces at will, but if switching, you need to bring the full consistent library stack along with it.

As for Clang in MSVC mode, I don't think it can cope with all of the lower level bits in the MinGW headers either, but I haven't really tried either. (It was noted that widl generated headers do work in MSVC mode - that's good at least!) FWIW, I don't think it's necessarily out of the question to accept patches in mingw-w64 to make the headers more tolerable in MSVC mode too. It's clearly not the core aim of the project, but if it doesn't hurt normal GCC/Clang use, it could work too. (The MinGW headers do contain some ancient remnants of what seems like intending for them to work with MSVC too, but I don't think it actually has worked for the past 10-15 years at least.)

mstorsjo commented 2 years ago

My interpretation of both of these sources is that:

  1. Wine's WIDL implementation: determining if the return value should be via pointer is as simple as looking at whether or not the IDL type definition is a struct, union, coclass, interface, or runtimeclass. I couldn't tell you why Wine does not have special handling for the case of aarch64 here (maybe because it's dealing with IDL files and not C++ code, so this is unnecessary? Alternatively it could be a real bug with Wine on aarch64)

I'd guess that nobody has exercised those corners of Wine on aarch64 yet. But also, I think in practice, the difference between being non-POD or C++14 aggregate only is visible in more corner cases (not exactly familiar with the details of what constitutes a C++14 aggregate though). If you look at the same Godbolt testcase as above, but for aarch64 targets, you get the same difference for the method_ret_obj function: https://godbolt.org/z/fKjzGWceq

  1. Clang's implementation: Specifically of note is that whether or not the return value should be via pointer is dependent upon: (a) whether the return type is POD (non-aarch64) OR (b) whether the return type is a C++14 'aggregate' (aarch64 only) OR (c) is a C++ instance method.

I think this one is the correct one, but I don't know exactly in which cases the distinction differs.

But a much bigger issue regarding arm/aarch64, is that WIDL_EXPLICIT_AGGREGATE_RETURNS isn't enabled in mingw-w64 headers right now, but based on what I see right now, it really should.

Do you happen to have a minimal C++ standalone testcase that sets up a COM object and calls such a method, that I could try out for arm/aarch64 targets?

slimsag commented 2 years ago

But a much bigger issue regarding arm/aarch64, is that WIDL_EXPLICIT_AGGREGATE_RETURNS isn't enabled in mingw-w64 headers right now, but based on what I see right now, it really should.

Yup! I think Zig (in GNU mode only) and MinGW-w64 should both be defining this by default, as it stands out of the box programs will compile but then segfault.

Do you happen to have a minimal C++ standalone testcase that sets up a COM object and calls such a method, that I could try out for arm/aarch64 targets?

Unfortunately not, what I have currently is quite entangled in a mess of other stuff. We will need to create such a minimal testcase from scratch.

slimsag commented 2 years ago

Hmm, I don't think there's anything in Clang automatically emitting references to _mingw* anything unless it was present in the source?

Seems you're right! I think I got confused here.

mstorsjo commented 2 years ago

But a much bigger issue regarding arm/aarch64, is that WIDL_EXPLICIT_AGGREGATE_RETURNS isn't enabled in mingw-w64 headers right now, but based on what I see right now, it really should.

Yup! I think Zig (in GNU mode only) and MinGW-w64 should both be defining this by default, as it stands out of the box programs will compile but then segfault.

Indeed. Not sure if Zig is defining this manually (or whether it should), as mingw-w64 headers already do define it automatically where they think it's relevant: https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-headers/crt/_mingw_mac.h#L356-L361 (But based on the reading of specs above, this workaround should be enabled everywhere in GNU mode, regardless of architecture.)

slimsag commented 2 years ago

That's strange; I am pretty confident I ran into this with Zig in GNU mode, compiling C++ on an x86_64 platform - maybe that header was not getting included for me somehow. I'll see if I can debug further.

mstorsjo commented 2 years ago

That's strange; I am pretty confident I ran into this with Zig in GNU mode, compiling C++ on an x86_64 platform - maybe that header was not getting included for me somehow. I'll see if I can debug further.

Hmm, I think most of the mingw-w64 provided headers would end up pulling in those base headers. I guess it's plausible that it was missed if testing something specific to the IDL generated headers though - or if Zig just does something different. (Or if I'm misreading the headers somehow, and that define isn't normally defined? A minimal buildable and runnable testcase would be awesome here.)

andrewrk commented 2 years ago

To clarify Zig's end goal regarding libc, our goal is to provide static libc for Windows, with no dependency on msvcrt.dll and no dependency on UCRT either.

But a much bigger issue regarding arm/aarch64, is that WIDL_EXPLICIT_AGGREGATE_RETURNS isn't enabled in mingw-w64 headers right now, but based on what I see right now, it really should.

10360

Edit: ah, I see that the mingw-w64 headers are intended to define this with preprocessor defines; let's explore sending a patch to mingw-w64 before merging this workaround.

slimsag commented 2 years ago

I think my memory may be wrong about encountering this with Zig in a GNU target, sorry. I'd assume for now there is no issue with the WIDL_EXPLICIT_AGGREGATE_RETURNS define, and I'll report back if I can repro.

mstorsjo commented 2 years ago

To clarify Zig's end goal regarding libc, our goal is to provide static libc for Windows, with no dependency on msvcrt.dll and no depedency on UCRT either.

That's a quite nice goal in itself (and presumably worthwhile for static binaries on all platforms).

Are you planning to provide standalone C headers for this libc then too, or just rely on the platform's existing headers (mingw-w64 or UCRT, musl, etc)? I presume the latter. For math functions, it should be quite straightforward I think.

For things like printf etc, it's less straightforward though, to provide a cross-platform libc implementation. If potential future MSVC header compatibility is a target, the UCRT headers only have printf as an inline function calling e.g. __stdio_common_vfprintf, so if you want your libc to be usable with UCRT headers, that's the function you'd need to provide.

andrewrk commented 2 years ago

Ideally:

And then yep, we would just provide __stdio_common_vfprintf for Windows targets and whatever other special things are required to make it work. Of course we would want to share as much common implementations as possible, and as you noted this would be the most straightforward for math functions, but it would also be perfectly fine to have some Windows-specific code (and some Linux-specific code too).

I also wouldn't be against maintaining a patchset against UCRT headers if there were some improvements that could be made. The benefits of such patches would be weighed against the maintenance burden.

PathogenDavid commented 2 years ago

Hello all, I stumbled across this issue from it being posted on Hacker News.

Another source of the Windows SDK that I've not seen mentioned in this thread is the Microsoft.Windows.SDK.CPP NuGet package.

You can download the package directly using the link on the right, .nupkg files are just zip files in disguise so just unzip it with whatever. Note that the import libraires are found in separate architecture-specific packages such as Microsoft.Windows.SDK.CPP.x64.

These packages are unfortunately not permissively licensed so distribution might be problematic, but maybe Zig could just automatically download them as needed? If the main goal is to avoid having developers install Visual Studio or the Visual Studio Build Tools this is probably the closest you could get to an official solution.

(Apologies if I've misunderstood the goals here. I've not actually even had a chance to use Zig yet so I definitely don't have a great idea of what its needs are 😅)

jbkempf commented 2 years ago

Now regarding headers, WinSDK generally has been non-redistributable

I have to emphase this. The Windows SDK is very far from being MIT or redistributable; both headers and IDLs are copyrighted heavily.

The Win32metadata gives a tool to generate, but if you look at the headers generated, they are not at all MIT.

iainnicol commented 2 years ago

The Windows SDK is very far from being MIT or redistributable

Very true. However, as @mstorsjo and I independently discovered, .h and .idl files are checked into the MIT licenced win32metadata repo, and these files are incredibly similar to those in the Windows SDK.

I have to politely but strongly disagree with mstorsjo here: I think it's very clear the files are MIT licenced, given the licence notice on the repo, but why draw unnecessary attention, in case corporate changes it mind?

It's also worth noting Microsoft plans to (but has not yet) open source most of vcruntime/vcstartup (minus some proprietary intel intrinsics). This is as part of their open source STL project. Also, the the cppwinrt files are MIT licenced, in the actual Windows SDK itself. Together these facts mean we could, in theory, put together a large subset of the Windows SDK, that's free software and MSVC-ABI compatible.

jbkempf commented 2 years ago

However, as @mstorsjo and I independently discovered, .h and .idl files are checked into the MIT licenced win32metadata repo, and these files are incredibly similar to those in the Windows SDK.

I don’t think this is correct. The main license of the repo might be MIT but the files are not.

Examples:

The main project of win32metadata is the tooling to process the files, which is what the MIT is about.

I think it's very clear the files are MIT licenced

It's very clear they are not. See "all right reserved"

mstorsjo commented 2 years ago

The Windows SDK is very far from being MIT or redistributable

Very true. However, as @mstorsjo and I independently discovered, .h and .idl files are checked into the MIT licenced win32metadata repo, and these files are incredibly similar to those in the Windows SDK.

I have to politely but strongly disagree with mstorsjo here: I think it's very clear the files are MIT licenced, given the licence notice on the repo, but why draw unnecessary attention, in case corporate changes it mind?

Keep in mind that it's totally possible to have a mixture of licenses within one repo (e.g. BSD/MIT/LGPL/whatever). And these files very specifically say "All rights reserved", not "distributable according to the toplevel license file".

Given the unclear situation, I wouldn't start relying on this until the situation is clarified. And if it's a case where this is more or less accidental and they could realize their mistake and change their mind, then we definitely shouldn't rely on that.

iainnicol commented 2 years ago

I don’t think this is correct. The main license of the repo might be MIT but the files are not.

Good examples. But the question is, when there's a contradiction, what takes precedence?

It's very clear they are not. See "all right reserved"

I'll note “All rights reserved” is also used with the BSD licences. It doesn't mean there can't be a separate permissive licence.

if it's a case where this is more or less accidental and they could realize their mistake and change their mind, then we definitely shouldn't rely on that.

On the other hand, if these files have been validly permissively licenced, that can't be undone.

I should probably clarify: I appreciate the points yous are making. I'm not saying it's 100% clear cut, more that it's arguable. I can also see why you'd prefer something was 100% clear, not just arguable. I also wonder what legal counsel would say. Regardless, hopefully there is some positive clarity.

jbkempf commented 2 years ago

In case of conflict, of course the information of the file prevails.

This has always been the case in all Open Source distributions compliance.

All rights reserved is pretty clear, not arguable…

On the other hand, if these files have been validly permissively licenced, that can't be undone.

Copyright and authorships are valid whatever the license header say. Those rights are unalienable, by default.

Else I could take any file from anyone and slap a MIT header on it…

PathogenDavid commented 2 years ago

@89z cl.exe is not a part of the Windows SDK, it's a part of the MSVC toolchain. If you're looking for a foss alternative to the toolchain you can use the clang-cl and lld-link. Or if you're wanting the MSVC toolchain without Visual Studio you can install Build Tools for Visual Studio.

PathogenDavid commented 2 years ago

@89z You're definitely not understanding what this thread is about:

Here's what MinGW provides for Zig:

* C/C++ headers for the win32 APIs (means we can compile C/C++ code without depending on Visual Studio)

* An implementation of libc for Windows

MinGW provides both a GNU-based toolchain and an alternative SDK for building Windows software. Zig is using MinGW-w64, which represents only the latter. Those NuGet packages are the Microsoft-official equivalent of MinGW-w64.

PathogenDavid commented 2 years ago

@89z No idea, at a glance it looks like they're using lld. Either way it's not the subject of this thread.