search

mesonbuild / meson

The Meson Build System
http://mesonbuild.com
Apache License 2.0
5.61k stars 1.63k forks source link

Wrong cpu detection for 'x86_64' windows #6128

Closed sz-CHEN closed 4 years ago

sz-CHEN commented 5 years ago

In ‘x64 Native Tools Command Prompt for VS 2017’, the cpu detected by meson is 'x86' ？

**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.16
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
D:\opt\glib-2.62.2\_build>meson .. --buildtype=release --prefix="D:\opt\glib\"
The Meson build system
Version: 0.52.0
Source dir: D:\opt\glib-2.62.2
Build dir: D:\opt\glib-2.62.2\_build
Build type: native build
Project name: glib
Project version: 2.62.2
C compiler for the host machine: cl (msvc 19.16.27034)
C linker for the host machine: link 19.16.27034
C++ compiler for the host machine: cl (msvc 19.16.27034)
C++ linker for the host machine: link 19.16.27034
Host machine cpu family: x86
Host machine cpu: x86

And I tried in vs2013 , 2015, got the same result.

scivision commented 5 years ago

I don't see this with VS2019--I didn't try with VS2017.


The Meson build system
Version: 0.52.0
Source dir: C:\meson\test cases\common\1 trivial
Build dir: c:\temp\11
Build type: native build
Project name: trivial test
Project version: undefined
C compiler for the host machine: cl (msvc 19.23.28106.4)
C linker for the host machine: link 19.23.28106.4
Host machine cpu family: x86_64
Host machine cpu: x86_64
``
sz-CHEN commented 5 years ago

Is there any way to overwrite host_machine?

scivision commented 5 years ago

The way I know of to control host_machine is by cross compiling. However that shouldn't be necessary here.

Is the correct CPU family detected from Powershell or regular windows command prompt instead of Visual Studio prompt?

Does anyone else reading this see the same issue from VS2017 or older?

lazka commented 5 years ago

glib CI uses VS2017 and it works there, but it's also using meson 0.49.2: https://gitlab.gnome.org/GNOME/glib/-/jobs/489274

scivision commented 5 years ago

hmm, is this related to #6134...

jpakkane commented 5 years ago

If you want to debug this, adding choice print statements in environment.py function detect_cpu and detect_cpu_family is fairly simple.

Also note that this may be related to the Python you are using. If you have a 32 bit Python on a 64 bit machine, it might misdetect the CPU in use. Something to keep in mind.

scivision commented 5 years ago

Looking inside environment.py, and using VS2019 command prompt on 64-bit Windows with Python 3.8 32-bit, this issue could happen if environment variable PROCESSOR_ARCHITEW6432 is not defined

@sz-CHEN what do you get from your Visual Studio command prompt for

echo %PROCESSOR_ARCHITEW6432%
scivision commented 5 years ago

More specifically, here's what one gets on a 64-bit Windows system using Python 64-bit vs. 32-bit. It does not matter if using Visual Studio command prompt or Powershell or whatever:

# 64-bit Python
>>> os.environ['PROCESSOR_ARCHITECTURE']
'AMD64'

# 32-bit Python
>>> os.environ['PROCESSOR_ARCHITECTURE']
'x86'
scivision commented 5 years ago

@sz-CHEN you can determine if you're using 32-bit or 64-bit Python by:

python -c "import sys; print(sys.maxsize > 2**32)"

reference

scivision commented 5 years ago

The above note may be the key to fix this issue. We are now waiting for feedback from @sz-CHEN to run these checks above.

sz-CHEN commented 5 years ago

@scivision Thanks for your help, the following is what I have tried.

**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.16
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community>D:

D:\>cd opt\glib-2.62.2

D:\opt\glib-2.62.2>python -c "import sys; print(sys.maxsize > 2**32)"
True

D:\opt\glib-2.62.2>python
Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> os.environ['PROCESSOR_ARCHITECTURE']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'os' is not defined
>>> import os
>>> os.environ['PROCESSOR_ARCHITECTURE']
'AMD64'
>>> quit()

D:\opt\glib-2.62.2>echo %PROCESSOR_ARCHITEW6432%
%PROCESSOR_ARCHITEW6432%

D:\opt\glib-2.62.2>echo %PROCESSOR_ARCHITECTURE%
AMD64

It is 64bit python, but the variable %PROCESSOR_ARCHITEW6432% was not defined but % PROCESSOR_ARCHITECTURE% is defined as 'AMD64'. And I tried set %PROCESSOR_ARCHITEW6432% as 'AMD64', but got the same result.

Then, I modified the 'environment.py', add print("compiler.target:"+compiler.target) in function detect_windows_arch, and found that: compiler.target:x86, and the compiler is defined from def detect_machine_info(compilers: typing.Optional[CompilersDict] = None) -> MachineInfo: called as machines.build = detect_machine_info()

C compiler for the host machine: cl (msvc 19.16.27034)
C linker for the host machine: link 19.16.27034
C++ compiler for the host machine: cl (msvc 19.16.27034)
C++ linker for the host machine: link 19.16.27034
compiler.target:x86
compiler.target:x86
Host machine cpu family: x86
Host machine cpu: x86
sz-CHEN commented 5 years ago

I tried print the compilers in different function.

OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
detect_cpu_family
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
detect_windows_arch
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
arch: amd64
compiler.target:x86
detect_cpu
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
detect_windows_arch
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
arch: amd64
compiler.target:x86
scivision commented 5 years ago

To summarize sz-CHEN's findings:

I think it may be helpful to trace line by line to see which exact line makes this x86 decision. Does any dev have vs2017 to try or sz-CHEN. You can insert a breakpoint() statement to help

mczarnek commented 5 years ago

Thank you for looking into this scivision, this is indeed the issue I've found too causing issue #6134

I ran this within meson itself: message('host_machine.cpu_family():'+host_machine.cpu_family()) gstreamer uses this to determine the version of wine to use, when it believes I'm using x86, this causes meson to pick the wine32 version of wine

Otherwise, same issue with python 2.7.17

>>> os.environ['PROCESSOR_ARCHITECTURE']
'AMD64'
>>> os.environ['PROCESSOR_ARCHITEW6432']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\os.py", line 425, in __getitem__
    return self.data[key.upper()]
KeyError: 'PROCESSOR_ARCHITEW6432'
scivision commented 5 years ago

OK @mczarnek I posted some questions in that issue

For @sz-CHEN, can you also copy and paste what Meson outputs from configuring a trivial meson.build containing only project('foo', 'c')

I suppose an issue may be that you only have Visual Studio compilers--well let's just try it and see what happens from regular (non-visual studio) command prompt.

lb90 commented 5 years ago

Hi! I have the same problem compiling GLib with VS 2019 (using x64 Native Tools Command Prompt for VS 2019). I have tried debugging detect_cpu_family in environment.py with pdb.set_trace(). Here's what turned out:

lb90 commented 5 years ago

In fact the first if succeds.

I have compiler.id == 'msvc' and compiler.target == 'x86'. But is it right to have compiler.target as 'x86'? Afterall I am not cross compiling. But sure, msvc can compile x86_64 and x86

scivision commented 5 years ago

Yes that's the question I have, I need someone more experienced with Meson and VS to know what the outcome should be--is it that VS has the "wrong" target configuration or that Meson should understand https://github.com/mesonbuild/meson/issues/6134#issuecomment-550465629

scivision commented 5 years ago

@nirbheek do you have insight or know who might into this issue? It seems with any version of Visual Studio, Meson thinks an x86 target should be built when a 64-bit target is desired.

lb90 commented 5 years ago

If I debug on 0.51.2 compiler.target is 'x64'

lb90 commented 5 years ago

The problem is this: https://github.com/mesonbuild/meson/blob/4c96aa34cb415ca654c0571280386880e9d14ca3/mesonbuild/environment.py#L993

The RE doesn't match. So match is None and target gets set to 'x86'.

lb90 commented 5 years ago

But why doesn't it match?

(Pdb) lookat
'Microsoft (R) C/C++ Optimizing Compiler versione 19.23.28106.4 per x64\nCopyright (C) Microsoft Corporation. Tutti i diritti  sono riservati.\n\n'
(Pdb) lookat.split('\n')[0]
'Microsoft (R) C/C++ Optimizing Compiler versione 19.23.28106.4 per x64'
lb90 commented 5 years ago

Ahhh OMG. The problem in the RE is that 'for' (an english word) is not present in other languages :)

lb90 commented 5 years ago

Here's the relevant commit: https://github.com/mesonbuild/meson/commit/a107eab2949cf9750ed88a634118b31666c16c7a

scivision commented 5 years ago

Assuming English language environment has been an issue previously, maybe on some issues still open.

It seems time to specifically have a non-English CI environment. Maybe a couple (Latin & non-Latin alphabets)

scivision commented 5 years ago

@sz-CHEN if you switch back to Meson 0.51.2 does that work, but 0.52.0 is broken? are you on a non-English language Windows computer? (like https://github.com/mesonbuild/meson/issues/6128#issuecomment-550490425)

lb90 commented 5 years ago

Is there a cl.exe command line option to force output in en_US?

scivision commented 5 years ago

set VSLANG=1033

or change codepage

If that helps, the question would be how to do that from Meson...

scivision commented 5 years ago

I think Meson's output is always in English, so depending on general opinion, if either of those work for this issue, maybe that would help avoid this issue in other places in Meson.

lb90 commented 5 years ago

set VSLANG=1033

Great! That works :)

lb90 commented 5 years ago

BTW, I think we should throw an exception here:

https://github.com/mesonbuild/meson/blob/4c96aa34cb415ca654c0571280386880e9d14ca3/mesonbuild/environment.py#L997

scivision commented 5 years ago

@jpakkane the solution here for a non-English locale appears to be setting environment variable to force Visual Studio to English output.

Another solution (besides removing the word "for" in the Meson regex code) would be per-language regex.

Since Meson's output is always English (?) maybe detecting this situation (non-English locale, use of MSVC) and setting this environment variable to force English MSVC output from the internal Python code is one possible fix?

scivision commented 5 years ago

yes I think an exception would be a better idea there. We'll surely have less confused users who might otherwise just give up in those cases.

sz-CHEN commented 5 years ago

I'm on a Chinese langauage Windows computer. After set VSLANG=1033 it works. Thanks for all of you.

jfinkhaeuser commented 5 years ago

Setting VSLANG=1033 does not seem to change my cl output, either on the Windows cmd or using a bash e.g. from MSYS or something.

I should add that I have this problem on the VS Community edition 19.23.28106.4

I think the regular expression may be too brittle. Why not change it to this?

 match = re.search('.* (x86|x64|ARM|ARM64)$', lookat.split('\n')[0])
jfinkhaeuser commented 5 years ago

Ah, I see master is already ahead.

lb90 commented 5 years ago

Hi Jens! Just for information purpose, could you post here the first line of cl.exe /? ?

jfinkhaeuser commented 5 years ago

Sure - took a while to get there (different/work computer and all that), but it's attached. Language is German, no matter the value of VSLANG.

output

I think the different detection patter in master is doing it right.

lb90 commented 5 years ago

Hi @jfinkhaeuser! Were you able to find a workaround in the meantime?

lb90 commented 5 years ago

I don't know if that does matters but I have the Visual Studio English Language Pack installed. In Visual Studio Installer, go to the 'Language Pack' tab and check the 'English' checkbox. Maybe this helps

jfinkhaeuser commented 5 years ago

That might be something I'll have to try. I usually configure everything in English, but don't really run VS - I just need the compiler :) I'll have a look.

For the time being, it's not super urgent - we need Windows builds mainly for running tests; the target platforms are mobile. So for now, we've just switched to 32-bit builds. But longer term, it'd be good to see this fixed.

Which really boils down to: is there an ETA for the detection code in master?

lb90 commented 4 years ago

@nirbheek using the VSLANG envvar is not a reliable solution. Can we change the regular expression in some way to work for all locales?

jfinkhaeuser commented 4 years ago

The code in master uses a different regex, that should be ok.

lb90 commented 4 years ago

Jens are you sure? Because I had this issue using meson from git.

lb90 commented 4 years ago

Here's the link: https://github.com/mesonbuild/meson/blob/master/mesonbuild/environment.py#L996

jfinkhaeuser commented 4 years ago

That might've been changed since last I looked. I saw the entire if 'Microsoft' in out or 'Microsoft' in err: block factored out into a function that just looked for a line ending in one of the CPU patterns, without the leading for.

Sorry for the confusion!

pandyah5 commented 11 months ago

I am facing a similar problem, I am on a Windows 11 64-bit system and have a 64-bit Python (3.11.4) installation. However, meson detects my system as x86 instead of x86_64. This in turn prompts it to look for x86 python which I don't have installed. Any suggestions on how to fix it?

PS: set VSLANG=1033 does not work for me. Also, I am using a developer command prompt for VS 2022

The Meson build system
  Version: 1.3.0
  Source dir: D:\Hetav_Documents\UofT\Academic Planning\ECE 444\OSS\matplotlib
  Build dir: D:\Hetav_Documents\UofT\Academic Planning\ECE 444\OSS\matplotlib\build\cp311
  Build type: native build
  Project name: matplotlib
  Project version: 3.9.0.dev0
  C compiler for the host machine: cl (msvc 19.36.32537 "Microsoft (R) C/C++ Optimizing Compiler Version 19.36.32537 for x86")
  C linker for the host machine: link link 14.36.32537.0
  C++ compiler for the host machine: cl (msvc 19.36.32537 "Microsoft (R) C/C++ Optimizing Compiler Version 19.36.32537 for x86")
  C++ linker for the host machine: link link 14.36.32537.0
  Host machine cpu family: x86
  Host machine cpu: x86
  Program python found: YES (D:\Hetav_Documents\UofT\Academic Planning\ECE 444\OSS\matplotlib-env\Scripts\python.exe)
  Need python for x86, but found x86_64
  Run-time dependency python found: NO (tried sysconfig)

  ..\..\meson.build:24:14: ERROR: Python dependency not found
lb90 commented 11 months ago

Hi @pandyah5! The VS Developer Command Prompt is always x86. If you want to target x86_64 then use the VS x64 Native Tools Command Prompt.

Should the issue persist, please do the following:

  1. Open Start > Microsoft Visual Studio > VS x64 Native Tools Command Prompt
  2. Type cl.exe /? and press Enter
  3. Post the output here

Setting VSLANG=1033 works only if the English / US language pack is installed. That's because VS uses a translation method based on string IDs, most probably relying on message tables in satellite DLLs, as usual for Windows-based software. That's different from typical UNIX software w/ libintl, where the base language is hardcoded in the program and thus always available.

In order to install the language pack, open the Visual Studio Installer, select "Modify" and then click on the "Language Pack" tab

pandyah5 commented 11 months ago

That solved my issue, thanks a ton @lb90 !