Open diegoferigo opened 5 years ago
From what it seems, the entrypoint function of the mex is missing. More information at mexFunction.
TL;DR Matlab looks for this symbol when it tries to load a mex library. If it is present, it calls it and proceeds with the computation.
There are already few bug reports I found, specifically:
As the mexFunction
entry point is used also in SWIG bindings, I guess this problem affects also iDynTree and YARP matlab bindings. How do you suggest to proceed? Vendoring the latest FindMatlab.cmake
will work, but we can also manually add the line:
set_target_properties(${${prefix}_NAME}
PROPERTIES
LINK_FLAGS "${_previous_link_flags} /EXPORT:mexFunction")
outside the matlab_add_mex
function.
This happens only on Windows or also on Linux? It applies also to 2019a ?
Ok, 2019a is not out yet. : )
Yesterday on @aikolina setup (Ubuntu 16.04) we added:
set_target_properties(Mex PROPERTIES COMPILE_FLAGS "-fvisibility=default")
For sure this works, I am not yet sure if this is too drastic (though it's not a big issue since we've never hidden any symbol in this project). I am waiting @aikolina pc to investigate deeper.
Further infos on the reason why the symbols are usually hidden:
Symbol clash in a MEX target
By default, every symbols inside a MEX file defined with the command
matlab_add_mex()
have hidden visibility, except for the entry point. This is the default behaviour of the MEX compiler, which lowers the risk of symbol collision between the libraries shipped with Matlab, and the libraries to which the MEX file is linking to. This is also the default on Windows platforms.However, this is not sufficient in certain case, where for instance your MEX file is linking against libraries that are already loaded by Matlab, even if those libraries have different SONAMES. A possible solution is to hide the symbols of the libraries to which the MEX target is linking to. This can be achieved in GNU GCC compilers with the linker option
-Wl,--exclude-libs,ALL
.
Does this mean that the upstream FindMatlab.cmake
is broken for Matlab 2018b + Ubuntu 16.04/18.04 ?
If there was an equivalent to MSVC's /EXPORT
function in GCC/Clang/ld that would be the ideal solution, these lines in the old FindMatlab in iDynTree see to implement something similar:
https://github.com/robotology/idyntree/blob/4a231725512978216e316d314ff4fe5d82aee5ea/cmake/FindMatlab.cmake#L927 .
However, it seems that the visibility of the mexFunction
symbols is handled by the DLL_EXPORT_SYM
macro: what happened in Matlab 2018b? The DLL_EXPORT_SYM
is not used anymore in the headers that define the function mexFunction
?
If you have time, can you check inside the mex.h
and related headers how the mexFunction
declaration changed between 2018a (or earlier) and 2018b ?
Does this mean that the upstream
FindMatlab.cmake
is broken for Matlab 2018b + Ubuntu 16.04/18.04?
No my last quote is the reason why CMake hides by default all symbols but mexFunction
. The fact that also mexFunction
is hidden seems a problem of Matlab headers. Here below the relevant sections:
// R2018a
#ifdef _WIN32
# define DLL_EXPORT_SYM __declspec(dllexport)
# define SUPPORTS_PRAGMA_ONCE
#elif __GNUC__ >= 4
# define DLL_EXPORT_SYM __attribute__ ((visibility("default")))
# define SUPPORTS_PRAGMA_ONCE
#else
# define DLL_EXPORT_SYM
#endif
#ifdef DLL_EXPORT_SYM
# define MEXFUNCTION_LINKAGE EXTERN_C DLL_EXPORT_SYM
#else
# ifdef MW_NEEDS_VERSION_H
# include "version.h"
# define MEXFUNCTION_LINKAGE EXTERN_C DLL_EXPORT_SYM
# else
# define MEXFUNCTION_LINKAGE EXTERN_C
# endif
#endif
// R2018b
#ifdef _MSC_VER
# define MWMEX_EXPORT_SYM __declspec(dllexport)
#elif __GNUC__ >= 4
# define MWMEX_EXPORT_SYM __attribute__ ((visibility("default")))
#else
# define MWMEX_EXPORT_SYM
#endif
#ifdef MW_NEEDS_VERSION_H
# define MEXFUNCTION_LINKAGE LIBMWMEX_API_EXTERN_C MWMEX_EXPORT_SYM
#else
# define MEXFUNCTION_LINKAGE LIBMWMEX_API_EXTERN_C
#endif
Refer to the upstream CMake fix, particularly:
if(NOT ${Matlab_VERSION_STRING} VERSION_LESS "9.5") # For 9.5 (R2018b) (and newer?)
target_compile_options(${${prefix}_NAME} PRIVATE "-fvisibility=default")
# This one is weird, it might be a bug in <mex.h> for R2018b. When compiling with
# -fvisibility=hidden, the symbol `mexFunction` cannot be exported. Reading the
# source code for <mex.h>, it seems that the preprocessor macro `MW_NEEDS_VERSION_H`
# needs to be defined for `__attribute__ ((visibility("default")))` to be added
# in front of the declaration of `mexFunction`. In previous versions of MATLAB this
# was not the case, there `DLL_EXPORT_SYM` needed to be defined.
# Adding `-fvisibility=hidden` to the `mex` command causes the build to fail.
# TODO: Check that this is still necessary in R2019a when it comes out.
endif()
Why we don't simply define MW_NEEDS_VERSION_H
?
It would be cool to see which compiler commands are actually called by the mex
command when compiling (see https://it.mathworks.com/help/matlab/ref/mex.html), for example by launching matlab under strace.
Reading comments from the issues I linked, it does not seem the right way to fix this. If CMake went to this direction I would apply the same fix in order to match the behavior of recent CMake version (I guess, >= 3.13.0, not sure if it will be backported).
Reading comments from the issues I linked, it does not seem the right way to fix this.
Why?
If CMake went to this direction I would apply the same fix in order to match the behavior of recent CMake version (I guess, >= 3.13.0, not sure if it will be backported).
Ok, this make sense even if I suspect the fix is sub-optimal.
Why?
I am not sure if it is used in other places nor what can happen now or in the future by enabling it. I think that the solution of exposing more symbols knowing for sure that there is no symbols collision (knowing our code) is safer than enabling an option that can insert code that we do not control (and are not aware since we cannot test many Matlab versions).
. I think that the solution of exposing more symbols knowing for sure that there is no symbols collision (knowing our code) is safer than enabling an option that can insert code that we do not control (and are not aware since we cannot test many Matlab versions).
I see. I think the definitive fix is to check which kind of preprocessor defines the mex
command is passing to the compiler (either using strace
or passing the -v
option as defined in https://it.mathworks.com/matlabcentral/answers/84440-how-can-i-see-the-command-line-options-that-the-mex-command-passes-to-the-c-compiler): the mex
command by definition (and because it is actually tested by Mathworks) is passing the intended and working flags, and in CMake we should simply try to replicate that. I did not see the full Matlab 2018b mex.h
, but I suspect that the mex
command is actually defining MW_NEEDS_VERSION_H
(or something that results in MW_NEEDS_VERSION_H
being defined).
It looks that mex
is a matlab command. In order to use it, we should figure out how to compile our S-Function from matlab command line. If this verbose flag as I think prints the complete set of options at the very beginning (as VERBOSE=1
of make), we don't care to specify the right includes and linked libraries.
The only user that as of today has the R2018b version of Matlab is @aikolina as far as I know.
I think we just need to run mex on any mex library (for example a one of the matlab examples, see the examples in https://it.mathworks.com/help/matlab/ref/mex.html), and check which flags are passed: we do not need to compile one of our specific projects. From my understanding, this problem is shared by all the projects that compile mex libraries (such as our YARP and iDynTree SWIG-based bindings) and so I am concerned in finding a solution that works fine for all our use cases (and I suspect at that point also CMake fix will need to be changed, if it turns out that mex
defines something that results in MW_NEEDS_VERSION_H
being defined).
The only user that as of today has the R2018b version of Matlab is @aikolina as far as I know.
If @aikolina can do run the mex
example in https://it.mathworks.com/help/matlab/ref/mex.html with the -V
flag and could copy paste the output here, it would be great! Furthermore, could you check if the iDynTree/YARP matlab bindings are working correctly?
Anyhow, I just realized that I had run an old copy of https://github.com/robotology/mex-wholebodymodel on a Matlab 2018b recently, and everything was working fine. Note that mex-wholebodymodel and also iDynTree YARP bindings actually use a vendored version of FindMatlab.cmake, that is probably different from the one that blockfactory
is using on @aikolina machine.
@traversaro Probably the vendored FindMatlab.cmake
does not hide by default the symbols, and everything runs fine for this reason.
As soon as we will be back we should do the check on the definitions you were mentioning.
@traversaro Probably the vendored
FindMatlab.cmake
does not hide by default the symbols, and everything runs fine for this reason.
Apparently this is not the case: https://github.com/robotology/mex-wholebodymodel/blob/master/cmake/FindMatlab.cmake#L909 , but it is possible that that code is not working as intended.
@traversaro Sorry for the late reply. I run the mex
example as you suggested in a previous comment with the -V
option. I copy and paste the ouput here:
>> openExample('matlab/MexBuildCMEXFileExample')
>> MexBuildCMEXFileExample
Warning: Directory already exists.
> In MexBuildCMEXFileExample (line 8)
Verbose mode is on.
No MEX options file identified; looking for an implicit selection.
... Looking for compiler 'gcc' ...
... Executing command 'which gcc' ...Yes ('/usr/bin/gcc').
... Executing command 'gcc -print-file-name=libstdc++.so' ...Yes ('/usr/lib/gcc/x86_64-linux-gnu/5/libstdc++.so').
Found installed compiler 'gcc'.
Options file details
-------------------------------------------------------------------
Compiler location: /usr/bin/gcc
Options file: /usr/local/MATLAB/R2018b/bin/glnxa64/mexopts/gcc_glnxa64.xml
CMDLINE2 : /usr/bin/gcc -pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64 -shared -O -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map" /tmp/mex_9403557286066_21341/yprime.o /tmp/mex_9403557286066_21341/c_mexapi_version.o -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o yprime.mexa64
CC : /usr/bin/gcc
DEFINES : -DMATLAB_DEFAULT_RELEASE=R2017b -DUSE_MEX_CMD -D_GNU_SOURCE -DMATLAB_MEX_FILE
MATLABMEX : -DMATLAB_MEX_FILE
CFLAGS : -fexceptions -fPIC -fno-omit-frame-pointer -pthread
INCLUDE : -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include"
COPTIMFLAGS : -O2 -fwrapv -DNDEBUG
CDEBUGFLAGS : -g
LD : /usr/bin/gcc
LDFLAGS : -pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64
LDTYPE : -shared
FUNCTIONMAP : "/usr/local/MATLAB/R2018b/extern/lib/glnxa64/mexFunction.map"
VERSIONMAP : "/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map"
LINKEXPORT : -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/mexFunction.map"
LINKEXPORTVER : -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map"
LINKLIBS : -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++
LDOPTIMFLAGS : -O
LDDEBUGFLAGS : -g
MWCPPLIB : "/usr/local/MATLAB/R2018b/sys/os/glnxa64/libstdc++.so.6"
OBJEXT : .o
LDEXT : .mexa64
SETENV : CC="/usr/bin/gcc"
CXX="g++"
CFLAGS="-fexceptions -fPIC -fno-omit-frame-pointer -pthread -DMATLAB_DEFAULT_RELEASE=R2017b -DUSE_MEX_CMD -D_GNU_SOURCE -DMATLAB_MEX_FILE "
CXXFLAGS="-fexceptions -fPIC -fno-omit-frame-pointer -pthread -std=c++11 -DMATLAB_DEFAULT_RELEASE=R2017b -DUSE_MEX_CMD -D_GNU_SOURCE -DMATLAB_MEX_FILE "
COPTIMFLAGS="-O2 -fwrapv -DNDEBUG"
CXXOPTIMFLAGS="-O2 -fwrapv -DNDEBUG"
CDEBUGFLAGS="-g"
CXXDEBUGFLAGS="-g"
LD="/usr/bin/gcc"
LDXX="g++"
LDFLAGS="-pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64 -shared -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/mexFunction.map""
LDDEBUGFLAGS="-g"
GCC : /usr/bin/gcc
CPPLIBS : /usr/lib/gcc/x86_64-linux-gnu/5/libstdc++.so
MATLABROOT : /usr/local/MATLAB/R2018b
ARCH : glnxa64
SRC : "/home/adinale/Documents/MATLAB/Examples/matlab/MexBuildCMEXFileExample/c:\work/yprime.c";"/usr/local/MATLAB/R2018b/extern/version/c_mexapi_version.c"
OBJ : /tmp/mex_9403557286066_21341/yprime.o;/tmp/mex_9403557286066_21341/c_mexapi_version.o
OBJS : /tmp/mex_9403557286066_21341/yprime.o /tmp/mex_9403557286066_21341/c_mexapi_version.o
SRCROOT : /home/adinale/Documents/MATLAB/Examples/matlab/MexBuildCMEXFileExample/c:\work/yprime
DEF : /tmp/mex_9403557286066_21341/yprime.def
EXP : "yprime.exp"
LIB : "yprime.lib"
EXE : yprime.mexa64
ILK : "yprime.ilk"
MANIFEST : "yprime.mexa64.manifest"
TEMPNAME : yprime
EXEDIR :
EXENAME : yprime
OPTIM : -O2 -fwrapv -DNDEBUG
LINKOPTIM : -O
CMDLINE1_0 : /usr/bin/gcc -c -DMATLAB_DEFAULT_RELEASE=R2017b -DUSE_MEX_CMD -D_GNU_SOURCE -DMATLAB_MEX_FILE -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/home/adinale/Documents/MATLAB/Examples/matlab/MexBuildCMEXFileExample/c:\work/yprime.c" -o /tmp/mex_9403557286066_21341/yprime.o
CMDLINE1_1 : /usr/bin/gcc -c -DMATLAB_DEFAULT_RELEASE=R2017b -DUSE_MEX_CMD -D_GNU_SOURCE -DMATLAB_MEX_FILE -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/usr/local/MATLAB/R2018b/extern/version/c_mexapi_version.c" -o /tmp/mex_9403557286066_21341/c_mexapi_version.o
-------------------------------------------------------------------
Building with 'gcc'.
Warning: You are using gcc version '5.4.0'. The version of gcc is not supported. The version currently supported with MEX
is '6.3.x'. For a list of currently supported compilers see: https://www.mathworks.com/support/compilers/current_release.
> In MexBuildCMEXFileExample (line 17)
/usr/bin/gcc -c -DMATLAB_DEFAULT_RELEASE=R2017b -DUSE_MEX_CMD -D_GNU_SOURCE -DMATLAB_MEX_FILE -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/home/adinale/Documents/MATLAB/Examples/matlab/MexBuildCMEXFileExample/c:\work/yprime.c" -o /tmp/mex_9403557286066_21341/yprime.o
/usr/bin/gcc -c -DMATLAB_DEFAULT_RELEASE=R2017b -DUSE_MEX_CMD -D_GNU_SOURCE -DMATLAB_MEX_FILE -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/usr/local/MATLAB/R2018b/extern/version/c_mexapi_version.c" -o /tmp/mex_9403557286066_21341/c_mexapi_version.o
/usr/bin/gcc -pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64 -shared -O -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map" /tmp/mex_9403557286066_21341/yprime.o /tmp/mex_9403557286066_21341/c_mexapi_version.o -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o yprime.mexa64
Recompile embedded version with '-DMATLAB_MEXCMD_RELEASE=R2017b'
/usr/bin/gcc -c -DMATLAB_DEFAULT_RELEASE=R2017b -DUSE_MEX_CMD -D_GNU_SOURCE -DMATLAB_MEX_FILE -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/usr/local/MATLAB/R2018b/extern/version/c_mexapi_version.c" -o /tmp/mex_9403557286066_21341/c_mexapi_version.o -DMATLAB_MEXCMD_RELEASE=R2017b
/usr/bin/gcc -pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64 -shared -O -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map" /tmp/mex_9403557286066_21341/yprime.o /tmp/mex_9403557286066_21341/c_mexapi_version.o -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o yprime.mexa64
MEX completed successfully.
ans =
2.0000 8.9685 4.0000 -1.0947
Mmh I don't see any interesting define. Out from curiosity, are you on R2018b
? I am puzzled by the R2017b
occurrences.
Yes, I am on MATLAB 2018b (Version: 9.5.0.944444) according to the ver
command. :sweat_smile:
I think the definitive resource on that is https://mathworks.com/matlabcentral/answers/377799-compiling-mex-files-without-the-mex-command . Apparently MathWorks them-self suggest to use the option -Wl,--version-script,"$(MWROOT)/extern/lib/glnxa64/c_exportsmexfileversion.map"
to pass to the compiler to specify the version-script
ld
argument (see https://www.gnu.org/software/gnulib/manual/html_node/LD-Version-Scripts.html for more). I guess is that option that correctly exports the mexFunction
symbol, but I am not 100% sure.
That documentation seems to be coherent with @aikolina findings.
That link is great! I already saw this method in another thread somewhere, but the usage of an external map file always seemed to me overly complex, and it needs to be tested in all the supported OSs. At this point I would consider to keep a vendored version of FindMatlab.cmake
inside blockfactory and perform some experiments.
Here is the map file provided in Matlab 2018a:
# This is the symbol export map file for glibc Linux. The file specifies
# that only the mexFunction symbol should have global scope,
# and that all others should be local.
# Copyright 2004-2017 The MathWorks, Inc.
MEX {
global:
mexFunction;
mexCreateMexFunction;
mexDestroyMexFunction;
mexFunctionAdapter;
mexfilerequiredapiversion;
local:
*;
};
At least this is provided directly from Mathworks. From the MATLAB Central thread it seems that this is not the case for Windows.
Indeed, we should check also what they are doing on macOS, as that is not reported in the thread.
From the MATLAB Central thread it seems that this is not the case for Windows.
The equivalent functionality in Windows is provided by the .def
file, but the command line /EXPORT
option should provide the same functionality in a more compact way.
It is interesting that for some reason they document to export mexFunction
and mexfilerequiredapiversion
in both Linux and Windows, but mexCreateMexFunction
, mexDestroyMexFunction
and mexDestroyMexFunction
only on Linux. I wonder if that is just something that was accidentally omitted in the MATLAB Answers docs.
Next week I plan to release the first release. If you don't have anything against it, I would temporary go for the visibility fix that is quick and effective, and leave more in-depth investigations and tests to the release after. @traversaro
Ok for me.
PR #36 should fix this issue but let's keep it open to check if the situation can be improved further.
I had a similar issue when compiling BlockFactory
in MacOS 10.13.6
with MATLAB_R2018a
.
The error I was getting is the following:
[ 22%] Performing build step for 'BlockFactory'
[ 9%] Built target mxpp
[ 28%] Built target shlibpp
[ 71%] Built target Core
[ 80%] Built target SimulinkCoder
[ 85%] Linking CXX shared library ../../lib/BlockFactory.mexmaci64
Undefined symbols for architecture x86_64:
"_mexfilerequiredapiversion", referenced from:
-exported_symbol[s_list] command line option
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
@diegoferigo helped me to debug the problem, and it is currently fixed by commenting out _mexfilerequiredapiversion
from MATLAB_R2018a.app/extern/lib/mac/c_exportsmexfileversion.map
Here some more details. @lrapetti configuration arrives at this line of the FindMatlab.cmake
file:
since his matlab is recognized as R2018a 9.3
. The difference between mexFunction.map
and c_exportsmexfileversion.map
is only the problematic symbol. It might be a upstream CMake bug, even if it is strange because that symbol should be exported starting from R2016b
accordingly to the comments in FindMatlab.cmake
, quite an old release.
@diegoferigo @traversaro @lrapetti , I just got the same problem on MacOS Sierra with Matlab 2017b. commenting the mentioned line also worked. Any observed side effects?
Any observed side effects?
Not that I know. Since blockfactory has a vendored version of FindMatlab.cmake
, we can fix it locally by editing the following section:
and changing the check on the Matlab version. However, it will be just a workaround since this problem might appear also on other Matlab versions and we cannot test more than the ones that we own. What do you think @traversaro?
I would fix the blockfactory vendored FindMatlab.cmake
's file, and open an issue in CMake reporting the problem.
If I understood correctly, @Giulero has Ubuntu 18.04 and Matlab 2019a and he is able to run the WB-Toolbox without problems. Probably the workaround in https://github.com/robotology/blockfactory/issues/4#issuecomment-460187364 is necessary just for macOS (or at least it is not necessary on Linux)?
MathWorks never replied to my comment in https://github.com/robotology/blockfactory/issues/4#issuecomment-454575694 . Given how this answer system works, it may be worth to try to ask the same question in a new question, instead of just commenting an old answered question.
Maybe they fixed it in 2019a? I still didn't update, they might have changed piece of code reported above in https://github.com/robotology/blockfactory/issues/4#issuecomment-449350853.
@Giulero (or anyone that has 2019a) can you check what the mex.h
file contains in the portion mentioned in https://github.com/robotology/blockfactory/issues/4#issuecomment-449350853 ?
When compiling the
Mex
target, the compilation fails with the following error:This happens only on Matlab R2018b. The same commit builds fine on (at least) R2018a.