coin-or / Clp

COIN-OR Linear Programming Solver
Other
412 stars 84 forks source link

Build failing in master with --enable-static under MSys with the Visual Studio compiler #134

Closed tkralphs closed 4 years ago

tkralphs commented 4 years ago

The static build is failing in master with the cl compiler when the ifort compiler is also configured. This is because of a tangle of things and I don't know how the solution yet. Here are the commands to replicate. First,

C:\Users\tkral>"C:\Program Files (x86)\Intel\Composer XE 2015\bin\ifortvars.bat" intel64
C:\Users\tkral>set PATH=C:\msys64\usr\bin;%PATH%
C:\Users\tkral>bash
~> coinbrew fetch Clp
~> coinbrew build Clp --no-prompt --enable-msvc --enable-static --disable-shared 

The eventual result of this is

make[1]: Entering directory '/home/tkral/Projects/build-static-msvc2017/Clp/master/src'
/bin/sh ../libtool  --tag=CXX   --mode=link /home/tkral/Projects/Clp/compile cl  -nologo -EHs -DNDEBUG -O2 -MD   -no-undefined -o libClp.la -rpath /home/tkral/Projects/dist-static/lib ClpCholeskyBase.lo ClpCholeskyDense.lo ClpConstraint.lo
ClpConstraintLinear.lo ClpConstraintQuadratic.lo Clp_C_Interface.lo ClpDualRowDantzig.lo ClpDualRowPivot.lo ClpDualRowSteepest.lo ClpDummyMatrix.lo ClpDynamicExampleMatrix.lo ClpDynamicMatrix.lo ClpEventHandler.lo ClpFactorization.lo ClpGubDynamicMatrix.lo ClpGubMatrix.lo ClpHelperFunctions.lo ClpInterior.lo ClpLinearObjective.lo ClpMatrixBase.lo ClpMessage.lo ClpModel.lo ClpNetworkBasis.lo ClpNetworkMatrix.lo ClpNonLinearCost.lo ClpNode.lo ClpObjective.lo ClpPackedMatrix.lo
ClpPlusMinusOneMatrix.lo ClpPredictorCorrector.lo ClpPdco.lo ClpPdcoBase.lo ClpLsqr.lo ClpPresolve.lo ClpPrimalColumnDantzig.lo ClpPrimalColumnPivot.lo ClpPrimalColumnSteepest.lo ClpQuadraticObjective.lo ClpSimplex.lo ClpSimplexDual.lo ClpSimplexNonlinear.lo ClpSimplexOther.lo ClpSimplexPrimal.lo ClpSolve.lo Idiot.lo IdiSolve.lo ClpCholeskyPardiso.lo ClpPESimplex.lo ClpPEPrimalColumnDantzig.lo ClpPEPrimalColumnSteepest.lo ClpPEDualRowDantzig.lo ClpPEDualRowSteepest.lo    ClpCholeskyMumps.lo  -L/home/tkral/Projects/dist-static/lib -lcoinmumps mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib -lCoinUtils mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib -lcoinasl -lcoinglpk

*** Warning: Linking the shared library libClp.la against the
*** static library mkl_intel_lp64.lib is not portable!

*** Warning: Linking the shared library libClp.la against the
*** static library mkl_sequential.lib is not portable!

*** Warning: Linking the shared library libClp.la against the
*** static library mkl_core.lib is not portable!
copying selected object files to avoid basename conflicts...
libtool: link: ln mkl_intel_lp64.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib || cp mkl_intel_lp64.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib
ln: failed to access 'mkl_intel_lp64.lib': No such file or directory
cp: cannot stat 'mkl_intel_lp64.lib': No such file or directory
libtool: link: ln mkl_sequential.lib .libs/libClp.lax/lt2-mkl_sequential.lib || cp mkl_sequential.lib .libs/libClp.lax/lt2-mkl_sequential.lib
ln: failed to access 'mkl_sequential.lib': No such file or directory
cp: cannot stat 'mkl_sequential.lib': No such file or directory
libtool: link: ln mkl_core.lib .libs/libClp.lax/lt3-mkl_core.lib || cp mkl_core.lib .libs/libClp.lax/lt3-mkl_core.lib
ln: failed to access 'mkl_core.lib': No such file or directory
cp: cannot stat 'mkl_core.lib': No such file or directory
libtool: link: lib -nologo -out:.libs/Clp.lib mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib .libs/libClp.lax/lt2-mkl_sequential.lib .libs/libClp.lax/lt3-mkl_core.lib ClpCholeskyBase.obj ClpCholeskyDense.obj ClpConstraint.obj ClpConstraintLinear.obj ClpConstraintQuadratic.obj Clp_C_Interface.obj ClpDualRowDantzig.obj ClpDualRowPivot.obj ClpDualRowSteepest.obj ClpDummyMatrix.obj ClpDynamicExampleMatrix.obj ClpDynamicMatrix.obj ClpEventHandler.obj ClpFactorization.obj ClpGubDynamicMatrix.obj ClpGubMatrix.obj ClpHelperFunctions.obj ClpInterior.obj ClpLinearObjective.obj ClpMatrixBase.obj ClpMessage.obj ClpModel.obj ClpNetworkBasis.obj ClpNetworkMatrix.obj ClpNonLinearCost.obj ClpNode.obj ClpObjective.obj ClpPackedMatrix.obj ClpPlusMinusOneMatrix.obj ClpPredictorCorrector.obj ClpPdco.obj ClpPdcoBase.obj ClpLsqr.obj ClpPresolve.obj ClpPrimalColumnDantzig.obj ClpPrimalColumnPivot.obj ClpPrimalColumnSteepest.obj ClpQuadraticObjective.obj ClpSimplex.obj ClpSimplexDual.obj ClpSimplexNonlinear.obj ClpSimplexOther.obj ClpSimplexPrimal.obj ClpSolve.obj Idiot.obj IdiSolve.obj ClpCholeskyPardiso.obj ClpPESimplex.obj ClpPEPrimalColumnDantzig.obj ClpPEPrimalColumnSteepest.obj ClpPEDualRowDantzig.obj ClpPEDualRowSteepest.obj ClpCholeskyMumps.obj
LINK : fatal error LNK1181: cannot open input file 'mkl_intel_lp64.lib'
make[1]: *** [Makefile:900: libClp.la] Error 157
make[1]: Leaving directory '/home/tkral/Projects/build-static-msvc2017/Clp/master/src'
make: *** [Makefile:757: all] Error 2

So it is trying to include the mkl libraries into the Clp library and cannot find them. Tracing the cause, it is because CoinUtils finds the mkl libraries during its configuration and adds them to the .pc file as follows.

Libs.private:   mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib

This happens only because ifort has also been set up and so the path to the mkl libraries is added to the environment variable LIB. Hence, the libraries are included in the link line for pkg-config --libs coinutils and we end up with

CLPLIB_LFLAGS='-L/home/tkral/Projects/dist-static/lib -lcoinmumps mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib -lCoinUtils mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib -lcoinasl -lcoinglpk   '

Meanwhile, in the src/Makefile.am for Clp (here), we have

libClp_la_LIBADD = $(CLPLIB_LFLAGS)

It appears that the LIB variable is not checked to find external libraries when linking a library, only when linking an executable, but this is not actually what's happening. The linker is finding the library, but libtool is itself trying to copy the library and this is the part that is failing (at least as far as I can tell right now).

This behavior does not arise with Osi, although it has the same setup and OSILIB_LFLAGS also includes the mkl libraries, because it seems Osi doesn't actually use any of the symbols in the library, so it is somehow ignored (or something like that, it's not very clear to me). During the Osi build, we get

*** Warning: Linking the shared library libOsiGlpk.la against the
*** static library mkl_intel_lp64.lib is not portable!

*** Warning: Linking the shared library libOsiGlpk.la against the
*** static library mkl_sequential.lib is not portable!

*** Warning: Linking the shared library libOsiGlpk.la against the
*** static library mkl_core.lib is not portable!
libtool: link: lib -nologo -out:.libs/OsiGlpk.lib mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib  OsiGlpkSolverInterface.obj
mkl_core.lib(psdftsrecm7as_mc3.obj) : warning LNK4221: This object file does not define any previously undefined public
symbols, so it will not be used by any link operation that consumes this library
mkl_core.lib(psdftsrecm7as_mc.obj) : warning LNK4221: This object file does not define any previously undefined public symbols, so it will not be used by any link operation that consumes this library
libtool: link: : .libs/OsiGlpk.lib
libtool: link: ( cd ".libs" && rm -f "libOsiGlpk.la" && cp -pR "../libOsiGlpk.la" "libOsiGlpk.la" )

and the library links just fine. It seems to be finding the library, but not trying to copy it.

There is another aspect to this, which is that the error did not happen when linking executables for the unit tests of CoinUtils and Osi. It's still not exactly clear to me what's going on there, but it is at least in part because we are using the .la files for linking those, e.g.,

unitTest_LDADD = ../src/libCoinUtils.la

and in libCoinUtils.la, we have

# Libraries that this one depends upon.
dependency_libs=' -L/home/tkral/Projects/dist-static/lib /home/tkral/Projects/dist-static/lib/libcoinasl.la /home/tkral/Projects/dist-static/lib/libcoinglpk.la'

so the mkl libraries are not linked. I suppose they are not needed for the CoinUtils unit test, but the autotools doesn't know this. Perhaps it is assuming that the mkl libraries are being rolled into the CoinUtils library, so those are no longer explicit dependencies. Anyway, this opens up a bunch of questions.

Anyway, this seems to be a thread that I tugged that's unraveling a lot of stuff. I'm still sorting through it, but thought I would get some folks involved (@svigerske @LouHafer )who know more about how the new build system hangs together before going much further.

svigerske commented 4 years ago

Yes, libtool copies the MKL into other libs when doing static builds. I think that also happened with BuildTools 0.8 sometimes. libtool tries to explain why it's doing this with copying selected object files to avoid basename conflicts.... So there is some check in libtool, which I'm not really understanding at the moment, but that might explain why it doesn't always do that.

The way how the MKL libs are named and presented in the linker flags may prevent libtool from adding them to the dependency_libs in the .la file.

I'm not sure that there is a good way to link the unittest of CoinUtils with the COINUTILSLIB_LFLAGS. So you suggest to do

unitTest_LDADD = -L../src -lCoinUtils $(COINUTILSLIB_LFLAGS)

But the CoinUtils library may sit in ../src/.libs, so that libtool will either not find it, or still picks up ../src/libCoinUtils.la. And since $(COINUTILSLIB_LFLAGS) has been recorded in the dependency_libs of libCoinUtils.la (except for MKL, it seems), dependencies would be duplicated.

Maybe one could extend the Lapack check to actually figure out the path to the MKL libs and add an appropriate -L flag or specify the libraries with full paths, so that libtool will find them for the Clp build.

svigerske commented 4 years ago

Well, with that change, linking libOsiClp now stops with an error from link.exe, saying it cannot link more than 65535 objects into one library :)

tkralphs commented 4 years ago

Adding full paths is a good thought, but it's not simply that the path to the libraries is completely unknown. They are only in play to begin with because Visual Studio does find the libraries from the path in the LIB variable and this is how they got detected in the first place. In config.log for CoinUtils, we have

configure:18536: checking for function dsyev_ in mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib 
configure:18552: /home/tkral/Projects/CoinUtils/compile cl -o conftest.exe -nologo -EHs -DNDEBUG -O2 -MD    conftest.cpp mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib  >&5
conftest.cpp
configure:18552: $? = 0
configure:18560: result: yes

So it works to just throw those libraries on the command line when linking an executable using compile. It just doesn't work to do it with libtool. The fundamental issue appears to me to be that the Visual Studio lib command doesn't seem to be checking the LIB variable for the path to libraries. However, according to this: https://docs.microsoft.com/en-us/cpp/build/reference/libpath-additional-libpath?view=vs-2019 it is supposed to use LIB to get its search path. It seems this shouldn't be an issue. Perhaps the LIB variable is being over-written somewhere. I guess I can just play with libtool a bit and try to get it to print out the value of the LIB variable. what is happening doesn't make sense to me.

tkralphs commented 4 years ago

Well, looking at the output above again, there are some errors even before libtool calls the lib command. I don't know what's going on... it seems to be a libtool issue, but we can't be the first to face it.

svigerske commented 4 years ago

I think lib is also fine with looking into $LIB. But the command that was failing was

libtool: link: ln mkl_intel_lp64.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib || cp mkl_intel_lp64.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib
ln: failed to access 'mkl_intel_lp64.lib': No such file or directory
cp: cannot stat 'mkl_intel_lp64.lib': No such file or directory

And I think that this is run by libtool and neither libtool nor ln look into $LIB.

So just having mkl_intel_lp64.lib as a flag that is passed to the compiler or linker is fine, but when libtool says "hey, this is a static library, so I can unpack it", then things go wrong.

tkralphs commented 4 years ago

Yeah, I see that the linking failed there, but then it appears to continue on and call lib later on with those libraries listed on the command line. The fatal error is actually from lib itself.

libtool: link: lib -nologo -out:.libs/Clp.lib mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib .libs/libClp.lax/lt2-mkl_sequential.lib .libs/libClp.lax/lt3-mkl_core.lib ClpCholeskyBase.obj ClpCholeskyDense.obj ClpConstraint.obj ClpConstraintLinear.obj ClpConstraintQuadratic.obj Clp_C_Interface.obj ClpDualRowDantzig.obj ClpDualRowPivot.obj ClpDualRowSteepest.obj ClpDummyMatrix.obj ClpDynamicExampleMatrix.obj ClpDynamicMatrix.obj ClpEventHandler.obj ClpFactorization.obj ClpGubDynamicMatrix.obj ClpGubMatrix.obj ClpHelperFunctions.obj ClpInterior.obj ClpLinearObjective.obj ClpMatrixBase.obj ClpMessage.obj ClpModel.obj ClpNetworkBasis.obj ClpNetworkMatrix.obj ClpNonLinearCost.obj ClpNode.obj ClpObjective.obj ClpPackedMatrix.obj ClpPlusMinusOneMatrix.obj ClpPredictorCorrector.obj ClpPdco.obj ClpPdcoBase.obj ClpLsqr.obj ClpPresolve.obj ClpPrimalColumnDantzig.obj ClpPrimalColumnPivot.obj ClpPrimalColumnSteepest.obj ClpQuadraticObjective.obj ClpSimplex.obj ClpSimplexDual.obj ClpSimplexNonlinear.obj ClpSimplexOther.obj ClpSimplexPrimal.obj ClpSolve.obj Idiot.obj IdiSolve.obj ClpCholeskyPardiso.obj ClpPESimplex.obj ClpPEPrimalColumnDantzig.obj ClpPEPrimalColumnSteepest.obj ClpPEDualRowDantzig.obj ClpPEDualRowSteepest.obj ClpCholeskyMumps.obj
LINK : fatal error LNK1181: cannot open input file 'mkl_intel_lp64.lib'

fatal error LINK 1181 is an error from lib, not libtool. Regardless of the earlier error, this seems like it should work. I'll try later to just call lib directly on the command line and see if I can figure out what's going on.

tkralphs commented 4 years ago

Weird when I re-run make by hanad, I now get

LINK : fatal error LNK1181: cannot open input file '.libs/libClp.lax/lt1-mkl_intel_lp64.lib'

If I run the lib command itself directly (removing reference to .libs/libClp.lax/lt1-mkl*, I get

$ lib -nologo -out:.libs/Clp.lib mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib ClpCholeskyBase.obj ClpCholeskyDense.o
bj ClpConstraint.obj ClpConstraintLinear.obj ClpConstraintQuadratic.obj Clp_C_Interface.obj ClpDualRowDantzig.obj ClpDualR
owPivot.obj ClpDualRowSteepest.obj ClpDummyMatrix.obj ClpDynamicExampleMatrix.obj ClpDynamicMatrix.obj ClpEventHandler.obj
 ClpFactorization.obj ClpGubDynamicMatrix.obj ClpGubMatrix.obj ClpHelperFunctions.obj ClpInterior.obj ClpLinearObjective.o
bj ClpMatrixBase.obj ClpMessage.obj ClpModel.obj ClpNetworkBasis.obj ClpNetworkMatrix.obj ClpNonLinearCost.obj ClpNode.obj
mkl_core.lib(psdftsrecm7as_mc3.obj) : warning LNK4221: This object file does not define any previously undefined public symbols, so it will not be used by any link operation that consumes this library
mkl_core.lib(psdftsrecm7as_mc.obj) : warning LNK4221: This object file does not define any previously undefined public symbols, so it will not be used by any link operation that consumes this library

but the library is created just fine! This is he same kind of output I saw in Osi, where things eventually went through fine, despite the warnings. So something is happening here that is a bit different than what is happening in Osi, but I don't see what yet.

tkralphs commented 4 years ago

I just updated my comment above, I had made a stupid mistake in what I posted earlier (running the lib command in the wrong directory.