search

OpenFAST / openfast

Main repository for the NREL-supported OpenFAST whole-turbine and FAST.Farm wind farm simulation codes.
http://openfast.readthedocs.io
Apache License 2.0
675 stars 455 forks source link

Undefined references when creating S-Function #482

Open brendansndrs opened 4 years ago

brendansndrs commented 4 years ago

I am trying to use OpenFAST with Simulink, however, I get errors when running create_FAST_SFunc.m. Some of the error message is below (the rest is similar, but for different functions).

>> create_FAST_SFunc

----------------------------
Creating ./FAST_SFunc.mexa64

Building with 'gcc'.
Error using mex
/pathToOpenFAST/openfast/install/lib/libopenfastlib.a(FAST_Library.f90.o): In function `FAST_AllocateTurbines':
FAST_Library.f90:(.text+0x48): undefined reference to `nwtc_io_mp_wrscr1_'
FAST_Library.f90:(.text+0x59): undefined reference to `nwtc_io_mp_wrscr1_'
FAST_Library.f90:(.text+0xf2): undefined reference to `for_check_mult_overflow64'
FAST_Library.f90:(.text+0x166): undefined reference to `for_alloc_allocatable'
FAST_Library.f90:(.text+0x197): undefined reference to `_intel_fast_memcpy'
FAST_Library.f90:(.text+0x1d3): undefined reference to `_intel_fast_memset'
/pathToOpenFAST/openfast/install/lib/libopenfastlib.a(FAST_Library.f90.o): In function `FAST_DeallocateTurbines':
FAST_Library.f90:(.text+0xf172): undefined reference to `for_deallocate_all'
FAST_Library.f90:(.text+0xf19d): undefined reference to `_intel_fast_memset'
/pathToOpenFAST/openfast/install/lib/libopenfastlib.a(FAST_Library.f90.o): In function `FAST_Sizes':
FAST_Library.f90:(.text+0xf1ef): undefined reference to `_intel_fast_memcpy'
FAST_Library.f90:(.text+0xf21f): undefined reference to `_intel_fast_memmove'
FAST_Library.f90:(.text+0xf233): undefined reference to `_intel_fast_memmove'
FAST_Library.f90:(.text+0xf25d): undefined reference to `for_f90_index'
FAST_Library.f90:(.text+0xf286): undefined reference to `for_cpystr'
FAST_Library.f90:(.text+0xf2a1): undefined reference to `__dtoq'
FAST_Library.f90:(.text+0xf3b5): undefined reference to `fast_subs_mp_fast_initializeall_t_'
FAST_Library.f90:(.text+0xf3c0): undefined reference to `nwtc_base_mp_aborterrlev_'
FAST_Library.f90:(.text+0xf483): undefined reference to `__qtod'
FAST_Library.f90:(.text+0xf4b2): undefined reference to `for_trim'
FAST_Library.f90:(.text+0xf4e5): undefined reference to `for_concat'
FAST_Library.f90:(.text+0xf532): undefined reference to `for_concat'
FAST_Library.f90:(.text+0xf557): undefined reference to `_intel_fast_memcpy'
FAST_Library.f90:(.text+0xf585): undefined reference to `for_cpystr'
FAST_Library.f90:(.text+0xf5aa): undefined reference to `_intel_fast_memcpy'
FAST_Library.f90:(.text+0xf5bb): undefined reference to `_intel_fast_memcpy'
FAST_Library.f90:(.text+0xf6aa): undefined reference to `_intel_fast_memset'

System Information OS: CentOS 7 Compiler: Intel 2018.3 Matlab Version: 2017a Matlab Compiler: gcc 4.9.4

I can successfully build the software using:

FC=ifort CC=icc CXX=icc  cmake .. -DBUILD_OPENFAST_SIMULINK_API=ON
make

upon calling make install there is an error:

Error copying directory from "/pathToOpenFAST/openfast/build/ftnmods" to "/usr/local/include/openfast/".

but otherwise installs successfully (note that /usr/local/... is not available on this machine). I did try moving the folder ftnmods to openfast/install/include/openfast but it had no effect. I'm not sure what libraries the undefined references are in, so I haven't been able to look into it too much. Issue #387 had similar problems, but I have not been able to use any of the suggestions there. It seems that some of the undefined references are to functions with intel in the name, could these errors be caused by compiler differences when compiling the library vs compiling the s-function? In #387 there was an issue with using a compiler not supported by Matlab, but it isn't clear to me whether the difference in compilers causes any issues. Finally, I have edited the create_FAST_SFunc.m file to point to the correct install location.

Thanks

andrew-platt commented 4 years ago

This appears to be an issue with the cmake scripts assuming /usr/local/ exists and is user writeable.

@rafmudaf, do you have any advice here?

rafmudaf commented 4 years ago

I think you'll need to compile with shared libraries (cmake .. -DBUILD_SHARED_LIBS=ON). The linker line from Matlab is trying to use a .a file, but thats a statically linked library and I think it needs to be a shared library.

brendansndrs commented 4 years ago

Thank you, I have since been able to compile the S-Function by changing to a shared library. I also had to manually copy the various files into the install directory to preserve the rpath variables as cmake deleted them. I also had to specify the rpath to libopenfastlib.so when compiling the S-Function.

However, when I try to test the install by running the 5MW_OC4Semi_WSt_WavesWN model with the supplied OpenLoop Simulink model I get the following error:

Intel MKL ERROR: Parameter 4 was incorrect on entry to .

The stack trace is as follows:

[  0] /matlab/2018b/bin/glnxa64/mkl.so+60692594 mkl_blas_avx2_dgemm_dcopy_down4_ea+00000370
[  1] /matlab/2018b/bin/glnxa64/mkl.so+21827573 mkl_blas_avx2_xdgemm+00001301
[  2] <unknown-module>+00000000

This error comes directly after the lines:

Fairlead tensions converged to 1 0x0.02b3463fd2dbp-1022fter 18 seconds.
Time: 0 of 20 seconds.

are printed to the screen, the FAST output file only has the output headers and units, but no time series data. The only changes I made were to the .fst file to specify the control inputs are from Simulink. I also tried compiling and running v2.4.0, but in that case a different failure occurred before Time: 0 of 20 seconds. was printed to the screen.

bjonkman commented 4 years ago

I am a little concerned that Matlab's LAPACK interface is a different than the LAPACK interface in OpenFAST. I would suggest building libopenfastlib.so with static libraries if possible and seeing if that helps. Based on @rafmudaf 's comment about building with shared libraries, though, you may have to modify the create_FAST_SFunc.m routine to link with the appropriate static library instead.

Or check the Matlab Answers page for some advice: https://www.mathworks.com/matlabcentral/answers/394673-mex-error-intel-mkl-error-parameter-6-was-incorrect-on-entry-to-dtrsv

brendansndrs commented 3 years ago

Thank you. So far I have gotten OpenFAST to work with the solution from matlabcentral, by overriding the built in libraries it does run. The downside is, as mentioned in the forum post, parts of Matlab break, for example, using a scope in Simulink causes crashes/errors.

I also spent some more time trying to compile with static libraries, however, during compilation/linking of the mex file none of the libraries are found and they must be manually specified (for example '-lservodynlib'). This happens for what seems like every library used during the compile process of OpenFAST, and eventually I wasn't able to find which libraries the undefined references belonged to and had to stop.

bjonkman commented 3 years ago

So, I am wondering if you can build OpenFAST with static libraries 

cmake .. -DBUILD_OPENFAST_SIMULINK_API=ON -DBUILD_SHARED_LIBS=OFF

That should hopefully just create one openfast library that Matlab would have to link with (instead of one library per module). Then tell Matlab to link with static libraries, too:

In the create_FAST_SFunc.m routine, replace COMPFLAGS with CFLAGS in the mex command.

brendansndrs commented 3 years ago

When I compile with the following command:

cmake .. -DBLAS_LIBRARIES="/usr/local/lib/libopenblas.a" -DLAPACK_LIBRARIES="/usr/local/lib/libopenblas.a" -DCMAKE_INSTALL_PREFIX="/usr/local/" -DBUILD_OPENFAST_SIMULINK_API=ON -DBUILD_SHARED_LIBS=OFF

I get the error:

[ 28%] Linking Fortran executable aerodyn_driver
/usr/bin/ld: /usr/local/lib/libopenblas.a(blas_server.o): undefined reference to symbol 'pthread_setaffinity_np@@GLIBC_2.3.4'
/usr/bin/ld: /lib/x86_64-linux-gnu/libpthread.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [modules/aerodyn/CMakeFiles/aerodyn_driver.dir/build.make:119: modules/aerodyn/aerodyn_driver] Error 1
make[1]: *** [CMakeFiles/Makefile2:698: modules/aerodyn/CMakeFiles/aerodyn_driver.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

To fix this I added the line SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -pthread") to the OpenFAST cmakelists.txt file which then results in OpenFAST compiling.

In Matlab if I then change the mex command to:

mex('-v', '-largeArrayDims', ...
    ['-L' libDir], ...
    ['-l' libName], ...
    ['-I' includeDir], ...
    '-I../../../modules/supercontroller/src', ... % needed for visual studio builds to find "SuperController_Types.h"
    '-I../../../modules/openfoam/src',        ... % needed for visual studio builds to find "OpenFOAM_Types.h"
    '-outdir', outDir, ...
    'CFLAGS=$CFLAGS -MT -D', ...
    ['S_FUNCTION_NAME=' mexname], ...
    '-output', mexname, ...
    'FAST_SFunc.c');

I get the following error:

Error using mex
cc1: error: to generate dependencies you must specify either -M or -MM

I then add the -M or -MM flags as requested:

mex('-v', '-largeArrayDims', ...
    ['-L' libDir], ...
    ['-l' libName], ...
    ['-I' includeDir], ...
    '-I../../../modules/supercontroller/src', ... % needed for visual studio builds to find "SuperController_Types.h"
    '-I../../../modules/openfoam/src',        ... % needed for visual studio builds to find "OpenFOAM_Types.h"
    '-outdir', outDir, ...
    'CFLAGS=$CFLAGS -M -MT -D', ...
    ['S_FUNCTION_NAME=' mexname], ...
    '-output', mexname, ...
    'FAST_SFunc.c');

and get this error:

Error using mex
/usr/bin/ld:/tmp/mex_57030495234624_12258/FAST_SFunc.o: file format not recognized; treating as linker script
/usr/bin/ld:/tmp/mex_57030495234624_12258/FAST_SFunc.o:1: syntax error
collect2: error: ld returned 1 exit status

It may also be worth noting that when I compile with -DBUILD_SHARED_LIBS=OFF I get a .a file for each library (e.g. libaerodynlib.a, libbeamdynlib.a, etc.) whereas I would have expected it to produce just the libopenfastlib.a.

Seager1989 commented 3 years ago

overriding the built in libraries it does run

Hi,

I encountered similar problem with you. Can you introduced some details regarding the: "So far I have gotten OpenFAST to work with the solution from matlabcentral, by overriding the built in libraries it does run?" Thank you

In addition, it is curious that I can compile do the Simulink API on my Ubuntu Linux system but got the Intel MKL ERROR on the Centos Linux system. The matlab on these two systems shipped the totally same mkl.so and blas linraries.

brendansndrs commented 3 years ago

@Seager1989

To load Matlab, from the terminal I use:

export BLAS_VERSION=libopenblas.so
export LAPACK_VERSION=libopenblas.so
matlab

When running with MKL rather than OpenBLAS I use:

export BLAS_VERSION=libmkl_intel_lp64.so
export LAPACK_VERSION=libmkl_intel_lp64.so
matlab

I have compiled on Ubuntu 20.04 (Matlab 2020, OpenBLAS, gcc 6.3.0) and CentOS 7 (Matlab 2017, MKL, gcc 4.9.4), both required me to apply this change in order to run, otherwise I got the errors as mentioned in my previous comments. I ended up using gcc instead of Intel compilers due to compatibility issues with Matlab, the Intel compilers available to me did not provide the gcc versions that Matlab required.

Seager1989 commented 3 years ago

@Seager1989

To load Matlab, from the terminal I use:

export BLAS_VERSION=libopenblas.so
export LAPACK_VERSION=libopenblas.so
matlab

When running with MKL rather than OpenBLAS I use:

export BLAS_VERSION=libmkl_intel_lp64.so
export LAPACK_VERSION=libmkl_intel_lp64.so
matlab

I have compiled on Ubuntu 20.04 (Matlab 2020, OpenBLAS, gcc 6.3.0) and CentOS 7 (Matlab 2017, MKL, gcc 4.9.4), both required me to apply this change in order to run, otherwise I got the errors as mentioned in my previous comments. I ended up using gcc instead of Intel compilers due to compatibility issues with Matlab, the Intel compilers available to me did not provide the gcc versions that Matlab required.

Thank you for your detailed explanation. I will try them later on my system.

Seager1989 commented 3 years ago

@Seager1989

To load Matlab, from the terminal I use:

export BLAS_VERSION=libopenblas.so
export LAPACK_VERSION=libopenblas.so
matlab

When running with MKL rather than OpenBLAS I use:

export BLAS_VERSION=libmkl_intel_lp64.so
export LAPACK_VERSION=libmkl_intel_lp64.so
matlab

I have compiled on Ubuntu 20.04 (Matlab 2020, OpenBLAS, gcc 6.3.0) and CentOS 7 (Matlab 2017, MKL, gcc 4.9.4), both required me to apply this change in order to run, otherwise I got the errors as mentioned in my previous comments. I ended up using gcc instead of Intel compilers due to compatibility issues with Matlab, the Intel compilers available to me did not provide the gcc versions that Matlab required.

Thank you, this solved my problem in #556 . The OpenFAST can run with S-function now.

isablanca commented 2 years ago

When I compile with the following command:

cmake .. -DBLAS_LIBRARIES="/usr/local/lib/libopenblas.a" -DLAPACK_LIBRARIES="/usr/local/lib/libopenblas.a" -DCMAKE_INSTALL_PREFIX="/usr/local/" -DBUILD_OPENFAST_SIMULINK_API=ON -DBUILD_SHARED_LIBS=OFF

I get the error:

[ 28%] Linking Fortran executable aerodyn_driver
/usr/bin/ld: /usr/local/lib/libopenblas.a(blas_server.o): undefined reference to symbol 'pthread_setaffinity_np@@GLIBC_2.3.4'
/usr/bin/ld: /lib/x86_64-linux-gnu/libpthread.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [modules/aerodyn/CMakeFiles/aerodyn_driver.dir/build.make:119: modules/aerodyn/aerodyn_driver] Error 1
make[1]: *** [CMakeFiles/Makefile2:698: modules/aerodyn/CMakeFiles/aerodyn_driver.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

To fix this I added the line SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -pthread") to the OpenFAST cmakelists.txt file which then results in OpenFAST compiling.

In Matlab if I then change the mex command to:

mex('-v', '-largeArrayDims', ...
    ['-L' libDir], ...
    ['-l' libName], ...
    ['-I' includeDir], ...
    '-I../../../modules/supercontroller/src', ... % needed for visual studio builds to find "SuperController_Types.h"
    '-I../../../modules/openfoam/src',        ... % needed for visual studio builds to find "OpenFOAM_Types.h"
    '-outdir', outDir, ...
    'CFLAGS=$CFLAGS -MT -D', ...
    ['S_FUNCTION_NAME=' mexname], ...
    '-output', mexname, ...
    'FAST_SFunc.c');

I get the following error:

Error using mex
cc1: error: to generate dependencies you must specify either -M or -MM

I then add the -M or -MM flags as requested:

mex('-v', '-largeArrayDims', ...
    ['-L' libDir], ...
    ['-l' libName], ...
    ['-I' includeDir], ...
    '-I../../../modules/supercontroller/src', ... % needed for visual studio builds to find "SuperController_Types.h"
    '-I../../../modules/openfoam/src',        ... % needed for visual studio builds to find "OpenFOAM_Types.h"
    '-outdir', outDir, ...
    'CFLAGS=$CFLAGS -M -MT -D', ...
    ['S_FUNCTION_NAME=' mexname], ...
    '-output', mexname, ...
    'FAST_SFunc.c');

and get this error:

Error using mex
/usr/bin/ld:/tmp/mex_57030495234624_12258/FAST_SFunc.o: file format not recognized; treating as linker script
/usr/bin/ld:/tmp/mex_57030495234624_12258/FAST_SFunc.o:1: syntax error
collect2: error: ld returned 1 exit status

It may also be worth noting that when I compile with -DBUILD_SHARED_LIBS=OFF I get a .a file for each library (e.g. libaerodynlib.a, libbeamdynlib.a, etc.) whereas I would have expected it to produce just the libopenfastlib.a.

Hi all, do you have any solution of linking the static library now> Because the recent Maltab can only work its own blas/lapack package, if we use the method of dynamic library linking and "export BLAS_VERSION and LAPACK_VERSION", many matlab builtin function even like inv() cannot run.