yasm/MS CL linkage fails for x64 target (with fix)

[GoogleCodeExporter]

What is the expected behavior? What do you see instead?

After configure --target=x86_64-win64-vs8
and assembling using yasm -f WIN64
I expect a link to succeed, as the 32 bit target does.

What I see instead are many linker errors, as the C and assembler symbol 
conventions are mismatched. Representative error:

x86_systemdependent.obj : error LNK2019: unresolved external symbol 
vp8_bilinear_predict8x8_ssse3 referenced in function vp8_arch_x86_common_init

What version are you using? On what operating system?

commit 58083cb34db84349d8f138e1ee59f4aee77e6624
Author: Fritz Koenig <frkoenig@google.com>
Date:   Thu Nov 11 08:19:37 2010 -0800

Cross-building Linux/WINE using Microsoft Visual Express 2008 toolchain.
Microsoft (R) C/C++ Optimizing Compiler Version 15.00.21022.08 for x64

Please provide any additional information below.

The following patch corrects the issue by suppressing the leading underscore 
from symbols, when yasm's win64 output format is being used. This matches the C 
linkage used by MS CL for the x64 target.

diff --git a/vpx_ports/x86_abi_support.asm b/vpx_ports/x86_abi_support.asm
index a872b28..b1a7464 100644
--- a/vpx_ports/x86_abi_support.asm
+++ b/vpx_ports/x86_abi_support.asm
@@ -88,6 +88,8 @@
 %define sym(x) x
 %elifidn __OUTPUT_FORMAT__,x64
 %define sym(x) x
+%elifidn __OUTPUT_FORMAT__,win64
+%define sym(x) x
 %else
 %define sym(x) _ %+ x
 %endif

Original issue reported on code.google.com by t...@telegraphics.com.au on 15 Nov 2010 at 4:17

[GoogleCodeExporter]

Fix looks ok.. Are there *any* differences between win64 and x64 in this 
context? I'm not clear on how you're hitting issue anyway -- when building for 
vs8, msvs invokes yasm -f $(PlatformName), which has always been x64 in my 
experience. Is this a difference between the Express and full versions maybe?

Original comment by jkoles...@google.com on 15 Nov 2010 at 8:51

[GoogleCodeExporter]

The yasm documentation does not mention -f x64. My build uses -f WIN64 which is 
documented in man yasm_objfmts:

WIN64
       The Win64 object format produces object files compatible with Microsoft compilers
       that target the 64-bit "x64" Windows platform. This format is very similar to the
       win32 object format, but produces 64-bit objects.

(This is the case for both yasm 0.8.0.2194 and, from my OS X build, yasm 
1.1.0.2352.)

I am not using VS IDE but rather the tools via CLI (in this case from the 
Visual Express distribution).

Original comment by t...@telegraphics.com.au on 15 Nov 2010 at 10:19

[GoogleCodeExporter]

Can you try with -f x64? I'm thinking that this may explain your other issue 
with crashes on 64 bit windows, since x86_abi_support.asm isn't the only place 
where we do something special for x64:

$ grep --include=*.asm -Rl x64 *
vp8/encoder/x86/dct_sse2.asm
vpx_ports/emms.asm
vpx_ports/x86_abi_support.asm

Didn't look in the docs, but this option should be fine in both the versions of 
yasm you mention:

$ yasm -f help
Available yasm object formats:
    dbg         Trace of all info passed to object format module
    bin         Flat format binary
    dosexe      DOS .EXE format binary
    elf         ELF
    elf32       ELF (32-bit)
    elf64       ELF (64-bit)
    coff        COFF (DJGPP)
    macho       Mac OS X ABI Mach-O File Format
    macho32     Mac OS X ABI Mach-O File Format (32-bit)
    macho64     Mac OS X ABI Mach-O File Format (64-bit)
    rdf         Relocatable Dynamic Object File Format (RDOFF) v2.0
    win32       Win32
    win64       Win64
    x64         Win64
    xdf         Extended Dynamic Object

Original comment by jkoles...@google.com on 17 Nov 2010 at 2:02

[GoogleCodeExporter]

John,

Solved both issues, thankyou. I wonder if the requirement for -f x64 with yasm 
needs stronger emphasis somewhere, given that the redundant options exist.

yasm man pages obviously need updating too. :/

Original comment by t...@telegraphics.com.au on 17 Nov 2010 at 4:42

[GoogleCodeExporter]

Original comment by iss...@webmproject.org on 16 Mar 2011 at 2:51

[GoogleCodeExporter]

Original comment by agra...@google.com on 17 Mar 2011 at 4:28

• Changed state: Accepted
• Added labels: Priority-Low, Type-Other
• Removed labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

No further change necessary at this time.

Original comment by jkoles...@google.com on 17 Mar 2011 at 4:43

• Changed state: Invalid