search

hypothetical-inc / gaffer

Gaffer is an open source application framework designed specifically for creating tools for use in visual effects production.
http://imageengine.github.io/gaffer/
BSD 3-Clause "New" or "Revised" License
59 stars 3 forks source link

Build gafferDependencies on Windows fails on boost: Unknown toolset: vc7 #1

Open BigRoy opened 5 years ago

BigRoy commented 5 years ago

Issue

I'm trying to build the msvc2017_for_merge branches to build gaffer for windows but I am getting stuck on gafferDependencies when building boost.

I'm using the gafferDependencies/winbuild/buildAll.bat and am running the commands from Visual Studio 2017 Developer Command Prompt v15.9.1 on Windows 10.

The Boost bootstrap.log states:

###
### "Unknown toolset: "
###
### You can specify the toolset as the argument, i.e.:
###     .\build.bat msvc
###
### Toolsets supported by this script are: borland, como, gcc, gcc-nocygwin,
###     intel-win32, metrowerks, mingw, msvc, vc7, vc8, vc9, vc10, vc11, vc12, vc14
###
###
### "Unknown toolset: vc7"
###
### You can specify the toolset as the argument, i.e.:
###     .\build.bat msvc
###
### Toolsets supported by this script are: borland, como, gcc, gcc-nocygwin,
###     intel-win32, metrowerks, mingw, msvc, vc7, vc8, vc9, vc10, vc11, vc12, vc14
###

And this is the output in the Developer command prompt:

===============================================================================
Building Boost...
===============================================================================
cmake -E tar xvf ../../Boost/archives\boost_1_61_0.tar.gz
bootstrap.bat --prefix=C:\dev_gaffer\gafferDependencies_build --with-python=C:\dev_gaffer\gafferDependencies_build --with-python-root=C:\dev_gaffer\gafferDependencies_build --without-libraries=log
Building Boost.Build engine

Failed to build Boost.Build engine.
Please consult bootstrap.log for further diagnostics.

You can try to obtain a prebuilt binary from

   http://sf.net/project/showfiles.php?group_id=7586&package_id=72941

Also, you can file an issue at http://svn.boost.org
Please attach bootstrap.log in that case.
echo "using python ^: 2.7 : C:\dev_gaffer\gafferDependencies_build ^: C:\dev_gaffer\gafferDependencies_build\include ^: C:\dev_gaffer\gafferDependencies_build\lib ^: ^<address-model^>64 ^; >> project-config.jam
"using python ^: 2.7 : C:\dev_gaffer\gafferDependencies_build ^: C:\dev_gaffer\gafferDependencies_build\include ^: C:\dev_gaffer\gafferDependencies_build\lib ^: ^<address-model^>64 ^; >> project-config.jam
b2 --prefix=C:\dev_gaffer\gafferDependencies_build --toolset=msvc-14.0 architecture=x86 address-model=64 --build-type=complete variant=release link=shared threading=multi -s ZLIB_SOURCE=%ROOT_DIR%\Zlib\working\zlib-1.2.11 -s ZLIB_INCLUDE=C:\dev_gaffer\gafferDependencies_build\include -s ZLIB_LIBPATH=C:\dev_gaffer\gafferDependencies_build\lib -s ZLIB_BINARY=zlib install
'b2' is not recognized as an internal or external command,
operable program or batch file.
Traceback (most recent call last):
  File "build/build.py", line 198, in <module>
  File "build/build.py", line 164, in __buildProject
  File "C:\Python27\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'b2 --prefix=C:\dev_gaffer\gafferDependencies_build --toolset=msvc-14.0 architecture=x86 address-model=64 --build-type=complete variant=release link=shared threading=multi -s ZLIB_SOURCE=%ROOT_DIR%\Zlib\working\zlib-1.2.11 -s ZLIB_INCLUDE=C:\dev_gaffer\gafferDependencies_build\include -s ZLIB_LIBPATH=C:\dev_gaffer\gafferDependencies_build\lib -s ZLIB_BINARY=zlib install' returned non-zero exit status 1
"Error(s) building Boost"

I noticed you already seem to be working around the bug in boost that cannot match msvc-14.1 and you seem to be forcing it to msvc-14.0. Yet it does not seem to be working for me.

Any ideas on how I could get around this?

I'm making the issue here since the gafferDependencies repo has no open issues area

ericmehl commented 5 years ago

Which command prompt are you running when you launch the developer command? Mine is called "x64 Native Tools Command Prompt for VS2017" in the start menu. When it's launched it says "[vcvarsall.bat] Environment initialized for: 'x64'" near the top, does yours show that?

Also check that when you launch the developer command prompt if you just type in cl does it give you something like `Microsoft (R) C/C++ Optimizing Compiler Version 19.13.26128 for x64 Copyright (C) Microsoft Corporation. All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]`

Boost uses a set of batch files to try and determine the toolset in the bootstrap phase so it looks like its autodetect is not working. I think that it should be getting the correct setting in the file Boost/working/Boost_1_61_0/tools/build/src/engine/build.bat starting on line 201.

That tests for the cl.exe command and I believe is where my setup is succeeding.

ericmehl commented 5 years ago

Also I just opened the issues section for the dependencies so we can use that for future issues that come up.

BigRoy commented 5 years ago

I tried both once, these are the ones I opened:

x64 Native Tools Command Prompt for VS 2017

**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.1
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community>

Developer Command Prompt for VS 2017

**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.1
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community>

Running cl gives me:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community>cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.16.27023.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

I'll try to play around some more to see what's up.

ericmehl commented 5 years ago

Hmmm, do you happen to have another version of MSVC installed? It looks like boost searches in order of earlier to later for some reason and it might be finding VC7 and running with it.

BigRoy commented 5 years ago

I double checked, it doesn't seem like it. I updated Visual Studio to the latest version - just to be sure, cleared other stuff that might have been related and tried again, yet still no luck.

I'm getting the exact same error.

BigRoy commented 5 years ago

If on this line here I add msvc after bootstrap.bat, so it becomes:

"bootstrap.bat msvc --prefix={buildDir} --with-python={buildDir} --with-python-root={buildDir} --without-libraries=log",

Then it actually is able to bootstrap Boost and I get this:

bootstrap.bat msvc --prefix=C:\dev_gaffer\gafferDependencies_build --with-python=C:\dev_gaffer\gafferDependencies_build --with-python-root=C:\dev_gaffer\gafferDependencies_build --without-libraries=log
Building Boost.Build engine
cl : Command line warning D9035 : option 'GZ' has been deprecated and will be removed in a future release
cl : Command line warning D9036 : use 'RTC1' instead of 'GZ'
cl : Command line warning D9002 : ignoring unknown option '/MLd'

Bootstrapping is done. To build, run:

    .\b2

To adjust configuration, edit 'project-config.jam'.
Further information:

    - Command line help:
    .\b2 --help

    - Getting started guide:
    http://boost.org/more/getting_started/windows.html

    - Boost.Build documentation:
    http://www.boost.org/build/doc/html/index.html

Then afterwards I get this:

echo "using python ^: 2.7 : C:\dev_gaffer\gafferDependencies_build ^: C:\dev_gaffer\gafferDependencies_build\include ^: C:\dev_gaffer\gafferDependencies_build\lib ^: ^<address-model^>64 ^; >> project-config.jam
"using python ^: 2.7 : C:\dev_gaffer\gafferDependencies_build ^: C:\dev_gaffer\gafferDependencies_build\include ^: C:\dev_gaffer\gafferDependencies_build\lib ^: ^<address-model^>64 ^; >> project-config.jam
b2 --prefix=C:\dev_gaffer\gafferDependencies_build --toolset=msvc-14.0 architecture=x86 address-model=64 --build-type=complete variant=release link=shared threading=multi -s ZLIB_SOURCE=%ROOT_DIR%\Zlib\working\zlib-1.2.11 -s ZLIB_INCLUDE=C:\dev_gaffer\gafferDependencies_build\include -s ZLIB_LIBPATH=C:\dev_gaffer\gafferDependencies_build\lib -s ZLIB_BINARY=zlib install
C:/dev_gaffer/gafferDependencies/Boost/working/boost_1_61_0/tools/build/src/tools\msvc.jam:834: in generate-setup-cmd
*** argument error
* rule maybe-rewrite-setup ( toolset : setup-script : setup-options : version : rewrite-setup ? )
* called with: ( msvc :  :  : default :  )
* missing argument setup-script
C:/dev_gaffer/gafferDependencies/Boost/working/boost_1_61_0/tools/build/src/tools\msvc.jam:746:see definition of rule 'maybe-rewrite-setup' being called
C:/dev_gaffer/gafferDependencies/Boost/working/boost_1_61_0/tools/build/src/tools\msvc.jam:1072: in configure-really
C:/dev_gaffer/gafferDependencies/Boost/working/boost_1_61_0/tools/build/src/tools\msvc.jam:201: in configure
C:/dev_gaffer/gafferDependencies/Boost/working/boost_1_61_0/tools/build/src/tools\msvc.jam:153: in msvc.init
C:/dev_gaffer/gafferDependencies/Boost/working/boost_1_61_0/tools/build/src/build\toolset.jam:43: in toolset.using
C:/dev_gaffer/gafferDependencies/Boost/working/boost_1_61_0/tools/build/src/build\project.jam:1052: in using
project-config.jam:3: in modules.load
C:/dev_gaffer/gafferDependencies/Boost/working/boost_1_61_0/tools/build/src\build-system.jam:249: in load-config
C:/dev_gaffer/gafferDependencies/Boost/working/boost_1_61_0/tools/build/src\build-system.jam:412: in load-configuration-files
C:/dev_gaffer/gafferDependencies/Boost/working/boost_1_61_0/tools/build/src\build-system.jam:524: in load
C:\dev_gaffer\gafferDependencies\Boost\working\boost_1_61_0\tools\build\src/kernel\modules.jam:295: in import
C:\dev_gaffer\gafferDependencies\Boost\working\boost_1_61_0\tools\build\src/kernel/bootstrap.jam:139: in boost-build
C:\dev_gaffer\gafferDependencies\Boost\working\boost_1_61_0\boost-build.jam:17: in module scope
Traceback (most recent call last):
  File "build/build.py", line 198, in <module>
  File "build/build.py", line 164, in __buildProject
  File "C:\Python27\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'b2 --prefix=C:\dev_gaffer\gafferDependencies_build --toolset=msvc-14.0 architecture=x86 address-model=64 --build-type=complete variant=release link=shared threading=multi -s ZLIB_SOURCE=%ROOT_DIR%\Zlib\working\zlib-1.2.11 -s ZLIB_INCLUDE=C:\dev_gaffer\gafferDependencies_build\include -s ZLIB_LIBPATH=C:\dev_gaffer\gafferDependencies_build\lib -s ZLIB_BINARY=zlib install' returned non-zero exit status 1
'1' is not recognized as an internal or external command,
operable program or batch file.
"Error(s) building ZLIB"

Which I think is related to Boost 1.61 not being easily compiled with Visual Studio 2017 if I read topics spread across the internet. I am still wondering why yours is passing, but you might be on a (much) older version of Visual Studio?

See here and here.

ericmehl commented 5 years ago

I updated to the latest MSVC the other day and still am getting successful builds, so it looks like it's probably something else. I'm getting the same error as your latest on AppVeyor though, so at least I'm able to somewhat reproduce your error. One weird thing is AppVeyor doesn't need they extra "msvc" you added to get boostrap working so there's still something extra going on.

I'm also considering pre-emptively upgrading to Boost 1.66 which should build out-of-the box with msvc 2017 (though I haven't tried it). It's part of the VFX Reference Platform for 2019 so it might be worth seeing if that smooths out the problems and seeing if the main Gaffer development team has objections to upgrading that part of the platform ahead of the other components.

For now though I'll look into getting 1.61 building on AppVeyor, hopefully this week as time allows.

ericmehl commented 5 years ago

Sorry for falling behind on this one - I think I found and fixed the offending Boost problem. It wasn't setting up the project-config.jam quite right in between configuring and building Boost: https://github.com/hypothetical-inc/gafferDependencies/blob/42e2640f878ce6e48a98fad93bd541d5589ef775/Boost/config.py#L51

I had Boost successfully building on AppVeyor a while back with that fix so it should build on a standard MSVC install. But I've removed the appveyor.yml for now since it won't build all the way in anything close to the 1 hour time limit of AppVeyor.

I do have a new dependencies release now that is up to date with John's latest on the main GafferDependencies repository. So you want to skip building this package altogether you can download the work-in-progress release here: https://github.com/hypothetical-inc/gafferDependencies/releases/tag/0.53.0.0.

Note that there is a version with and without Cortex - grab the one with Cortex (https://github.com/hypothetical-inc/gafferDependencies/releases/download/0.53.0.0/gafferDependencies-0.53.0.0-windows.zip) unless you know you want to build Cortex yourself. (The package without Cortex is mainly for building Cortex on AppVeyor)