dtolnay / cxx

Safe interop between Rust and C++
https://cxx.rs
Apache License 2.0
5.91k stars 338 forks source link

Using the 'stdcall' calling convention (instead of the default 'cdecl') on i686-pc-windows-msvc? #1067

Open nbigaouette opened 2 years ago

nbigaouette commented 2 years ago

Thanks for this awesome project!

I'd like to use it in a i686-pc-windows-msvc project that uses the __stdcall calling convention on the C++ side (instead of the default __cdecl).

Unfortunately for me, cxx expands functions as unsafe extern "C" fn __my_function_name() (which I believe is __cdecl) instead of the unsafe extern "stdcall" fn __my_function_name().

I hope that cxx could be adapted to support the stdcall ABI but I'm not even sure it's possible.

What are my possibilities here?

Thanks a lot!

wravery commented 1 year ago

Would it make more sense to expand to extern "system"? The note in the nomicon seems to indicate that it would pick the appropriate "stdcall" default for i686-pc-windows-msvc, and use "C" for everything else.

wravery commented 1 year ago

It's not as simple as switching to extern "system". Unless I set CXXFLAGS="/Gz" in the environment, building i686-pc-windows-msvc fails because apparently MSVC doesn't set the default calling convention to __stdcall. If I do set that, I still get some export name mismatches in the tests, which appear to differ in the argument size they're expecting:

libcxx_test_suite-6d8f09ecab552ff9.rlib(cxx_test_suite-6d8f09ecab552ff9.442x16000risslmd.rcgu.o) : error LNK2019: unresolved external symbol _tests$cxxbridge1$c_take_callback$callback$0@0 referenced in function __ZN14cxx_test_suite3ffi15c_take_callback17h3372b6d47d75f792E
            Hint on symbols that are defined and could potentially match:
              _tests$cxxbridge1$c_take_callback$callback$0@16
          libcxx_test_suite-6d8f09ecab552ff9.rlib(cxx_test_suite-6d8f09ecab552ff9.442x16000risslmd.rcgu.o) : error LNK2019: unresolved external symbol _tests$cxxbridge1$c_take_callback_ref$callback$0@0 referenced in function __ZN14cxx_test_suite3ffi19c_take_callback_ref17hec515c6d4656d4e4E
            Hint on symbols that are defined and could potentially match:
              _tests$cxxbridge1$c_take_callback_ref$callback$0@8
          libcxx_test_suite-6d8f09ecab552ff9.rlib(cxx_test_suite-6d8f09ecab552ff9.442x16000risslmd.rcgu.o) : error LNK2019: unresolved external symbol _tests$cxxbridge1$c_take_callback_ref_lifetime$callback$0@0 referenced in function __ZN14cxx_test_suite3ffi28c_take_callback_ref_lifetime17h4468f144148151c6E
            Hint on symbols that are defined and could potentially match:
              _tests$cxxbridge1$c_take_callback_ref_lifetime$callback$0@8
          libcxx_test_suite-6d8f09ecab552ff9.rlib(cxx_test_suite-6d8f09ecab552ff9.442x16000risslmd.rcgu.o) : error LNK2019: unresolved external symbol _tests$cxxbridge1$c_take_callback_mut$callback$0@0 referenced in function __ZN14cxx_test_suite3ffi19c_take_callback_mut17he1f08170dbf13875E
            Hint on symbols that are defined and could potentially match:
              _tests$cxxbridge1$c_take_callback_mut$callback$0@8
wravery commented 1 year ago

Ah, I found where it's getting the @0 for the argument size:

                    #[link_name = #c_trampoline]
                    fn trampoline();

The compiler is able to figure out that it should be adding @<size of arguments in bytes> when exporting as extern "system", but since it's casting to a * c_void the declaration leaves out the arguments (and return type).