Open bradfitz opened 3 years ago
That's a great idea. https://docs.microsoft.com/en-us/uwp/winrt-cref/winmd-files is the file format.
I'd be in favor of making x/win32
or x/windows/win32
or similar for this -- something totally separate from the incredibly cluttered and disorganized x/sys/windows
. It'll probably take us a few tries to get it right, and I assume that Microsoft will periodically issue corrections to the metadata, which we'll use to regenerate. In that sense, we wouldn't want to strive at an API compatibility guarantee beyond what Microsoft provides. If Microsoft fixes a function signature, then we'd want that to reflect in x/win32 without additional hassle or compatibility burden. This in turn means there might be a little bit more churn than the usual x/
repos, at least in the beginning, but in exchange, we get a complete API picture that's much better than what we have now.
CC @jstarks in case you're interested too.
@zx2c4
we wouldn't want to strive at an API compatibility guarantee beyond what Microsoft provides. If Microsoft fixes a function signature, then we'd want that to reflect in x/win32 without additional hassle or compatibility burden.
At that point would it make sense to rewrite x/sys/windows
in terms of small compatibility shims over x/sys/win32
?
Yes, this sounds like a great idea. I agree that starting fresh, rather than trying to replat x/sys/windows
(at least initially), is the right approach.
Note that there are fidelity issues with win32metadata today--for example, C unions are not yet supported correctly. But I'm told the team is working on it.
I bet @sotteson1 and @kennykerr would be happy to give advice and take feedback.
At that point would it make sense to rewrite x/sys/windows in terms of small compatibility shims over x/sys/win32?
That would potentially be a lot of work but wouldn't be impossible. It'd probably easier, though, to just deprecate x/sys/windows and encourage people to gradually move over to x/windows/win32 (or x/windows/winrt).
Note that there are fidelity issues with win32metadata today--for example, C unions are not yet supported correctly. But I'm told the team is working on it.
C unions are the big annoying thing for Go too, and we'd probably want to come up with an overall approach at handling these, whether it's just using unsafe.Pointer
when possible, or exploding one struct into multiple, or something else. A lot of fun decisions to make, I guess. C# doesn't quite have unions either, I don't think, which perhaps is where the trouble is coming from on the win32metadata side. I'll be interested to learn what you guys come up with there, as perhaps it's a model we could copy for Go.
Note that there are fidelity issues with win32metadata today--for example, C unions are not yet supported correctly. But I'm told the team is working on it.
Note that the metadata fully supports C-style unions. Some of the early language support don't yet support them, but the metadata is ready for consumption/trial.
Ah, good, I misunderstood the source of the union bug in the Rust crate. Thanks Kenny.
Does this mean golang's CGO will link with MSVC compiler in future? or I am missing something.
If I am correct, golang on windows works with MingGW & not MSVC
Does this mean golang's CGO will link with MSVC compiler in future? or I am missing something.
If I am correct, golang on windows works with MingGW & not MSVC
CGO and MSVC are unrelated concerns to this one, as far as I can see here.
Yes, I agree with most of what you said. We'll try not to repeat the mistakes of x/sys/windows, of which there are many many, when generating from winmd. Fully generating it means we can easily tweak the output format until it looks like something acceptable. And relaxing the compatibility requirements -- mentioned in https://github.com/golang/go/issues/43838#issuecomment-765585121 -- means that we'd be able to actually perfect the output, rather than getting stuck with inconsistent signatures, like the one you mentioned.
I cannot begin to tell you how much I want this feature. :100:
Note that the file referenced in Brad's first comment here is now located at https://github.com/microsoft/windows-rs/blob/master/crates/gen/default/Windows.Win32.winmd
To follow up, that file can be fetched from this package: https://www.nuget.org/packages/Microsoft.Windows.SDK.Win32Metadata
Download and unzip the file (nuget packages are just zip files) and you should find Windows.Win32.winmd
in the root directory.
I also just discovered today that WinMD files are valid PE files, so someone could potentially use pkg/debug/pe
to implement a parser for WinMD files.
Regarding parsing the WinMD files, Jonathan Marler created a project to generate JSON representation from the win32 metadata, which was used to create the win32 binding for Zig. There's also a talk on Youtube explaining the project.
The generated JSON files are here: https://github.com/marlersoft/win32json/tree/main/api
https://docs.microsoft.com/en-us/uwp/winrt-cref/winmd-files is the file format.
There is a full ECMA-335 specification https://www.ecma-international.org/wp-content/uploads/ECMA-335_6th_edition_june_2012.pdf.
See II.22 Metadata logical format: tables
.
https://github.com/tdakkota/win32metadata
Wrote metadata parser in Go.
There is a simple tool to generate some Go defintions from metadata:
$ go install github.com/tdakkota/win32metadata/cmd/printsig
$ printsig -file ./md/testdata/.windows/winmd/Windows.Win32.winmd -method ShellExecuteW
func ShellExecuteW(
p0 HWND,
hwnd PWSTR,
lpOperation PWSTR,
lpFile PWSTR,
lpParameters PWSTR,
lpDirectory int32,
) HINSTANCE
type PWSTR *uint16
type HINSTANCE int
type HWND int
Awesome work @tdakkota! After looking at the repo, I'm considering what would be the next steps to move this issue forward. I notice that printsig
does a straightforward translation of win32 datatypes to Go datatypes (e.g. PWSTR
is a *uint16
). Does the metadata say anything about which parameters are supposed to be strings? And if Go bindings are to be idiomatic, should we do UTF-8 <-> UTF-16 conversion on our end?
ECMA-335 defines some flags for types (II.23.1.15 Flags for types [TypeAttributes]
) and there are AnsiClass
and UnicodeClass
attributes, but it seems Windows metadata does not use it.
There is also a ELEMENT_TYPE_STRING
, but it used only in WinRT.
For example:
printsig -file ./md/testdata/.windows/winmd/Windows.WinRT.winmd -method RenameAsync
func RenameAsync(
desiredName string,
) IAsyncAction
type IAsyncAction struct { // printsig does not support managed structures and inheritance yet
}
windows-rs adds PSTR
and PWSTR
conversions manually
zigwin32gen too
I think if you want a pretty Go idiomatic API, you need a well-known types mapping.
There is also a ELEMENT_TYPE_STRING, but it used only in WinRT.
Interesting. That sounds like something worth bringing up upstream. I agree that in the meantime, having a mapping of well-known types is the right approach.
Edit: I filed an issue upstream at https://github.com/microsoft/win32metadata/issues/589.
The generated JSON files are here: https://github.com/marlersoft/win32json/tree/main/api
I've created a project go-win32api that provides win32 API bindings in go, which are generated based the the mentioned JSON files. Anyone interested could go and take a look.
It would be great if the WinRT API was added as well but that may be out of scope for this issue. They release those over NuGet as well: https://www.nuget.org/packages/Microsoft.Windows.SDK.Contracts
I'm (also!) interested in helping with this if possible, but I'm not totally sure what the current plan is after making the new repo. Sorry if it's simply too early in the process for these kinds of questions. 😄
Based on how x/sys/windows
works, it seems like the new repo (say, x/win32
) would contain the generated bindings as well as the tools that generate the bindings. This would let people use the tools to generate extra bindings if they aren't included in x/win32
, like what you can do now with manually written //sys
and the mkwinsyscall
generator. Is that generally what's in mind?
On the dependency side, it seems to me that parsing the winmd
files directly would be preferable to using an intermediate json
, just because it would keep the critical update path a little shorter if there's an important (e.g. security) release from microsoft/win32metadata
. But, maybe this isn't an important consideration because of the nature of the calls and win32 API?
I work for Microsoft, and a couple of us have done some work on another implementation that we can publish on the microsoft
org and commit to maintaining. However, it seems to me to be better for everyone if we end up with an implementation in x/win32
, so ideally, I'd like to help get there if I can.
Also, please see https://github.com/saltosystems/winrt-go which has seen some active development recently, and that we are currently using in the https://github.com/tinygo-org/bluetooth project.
Great to see some movement on this!
Thanks for mentioning us @deadprogram and kudos for the fantastic work done by @jagobagascon there!
Now that we're past the Dec holidays, wanted to ping this. I imagine that because creating some kind of x/win32
repository is part of the consensus (it seems to me), it'll take work/investment from the Go team to make that happen, even before the further questions enter the picture.
Is someone driving this proposal? This GitHub issue has been around for about a year now, so I wonder if priorities and perspectives may have shifted and a fresh, perhaps more detailed proposal is needed.
Beware that the data portions of the github.com/microsoft/win32metadata repository are.. not open source, and the derivative versions built from those sources are.. questionably licensed:
https://github.com/microsoft/win32metadata/issues/766
https://github.com/microsoft/win32metadata/issues/761
https://github.com/microsoft/win32metadata/blob/main/licenses/sdk_license.txt
There is no derivative issue here (Microsoft owns all the goop in there). The license on the bits you care about (the .winmd) is MIT. I'll poke some folks again to get that clarified in the README.
fyi, there is also https://github.com/microsoft/go-winmd now
Yeah, just to clarify, https://github.com/microsoft/go-winmd is the repo for the project I mentioned here:
I work for Microsoft, and a couple of us have done some work on another implementation that we can publish on the
microsoft
org and commit to maintaining. However, it seems to me to be better for everyone if we end up with an implementation inx/win32
, so ideally, I'd like to help get there if I can.
There are a number of open issues on the repo, https://github.com/microsoft/go-winmd/issues, and we haven't been able to work on this recently to get it to a point where we think it's stable and produces truly usable syscall signatures.
After a little more work, what we're planning to do as our next step is create a new proposal to create a x/win32
repo that contains the code that's currently in the microsoft/go-winmd repo along with syscalls generated by that tooling.
Microsoft released this win32 metadata project: https://github.com/microsoft/win32metadata
It or its output is basically this 9.5MB machine-readable file describing all the Win32 APIs: https://github.com/microsoft/windows-rs/blob/master/crates/winmd/default/Windows.Win32.winmd
Maybe x/sys/windows could use it to auto-generate bindings more easily/accurately.
Might be fun. Might be tedious. Might be worth it.
/cc @alexbrainman @ianlancetaylor @zx2c4