Snopt library cannot be loaded.

[aespielberg]

I am simply trying to run the example as written. However, when I try to run evolve, I get the following error:

function: evolve
where: /pagmo_plugins_nonfree/include/pagmo_plugins_nonfree/snopt7.hpp, 506
what: 
An error occurred while loading the snopt7_c library at run-time. This is typically caused by one of the following
reasons:

- The file declared to be the snopt7_c library, i.e. /home/aespielberg/snopt/lib/libsnopt7_c.so, is not a shared library containing the necessary C interface symbols (is the file path really pointing to
a valid shared library?)
 - The library is found and it does contain the C interface symbols, but it needs linking to some additional libraries that are not found
at run-time.

We report the exact text of the original exception thrown:

 boost::dll::shared_library::load() failed (dlerror system message: /home/aespielberg/snopt/lib/libsnopt7_c.so: undefined symbol: snstop_): Bad file descriptor

What is the cause of this? libsnopt7_c.so is exactly where it is pointed.

[darioizzo]

This is not a crash. The program throws as you library is either not found (and this seems not to be the case) or it does not contain an API compatible with our plugin. Where did you take the library? Have you compiled it yourself?

Can you post the result of ldd on your snopt7_c library?

ldd libsnopt7_c.so

[aespielberg]

ldd libsnopt7_c.so linux-vdso.so.1 => (0x00007ffe5a5ab000) libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007fe1649a9000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe1645df000) libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007fe1643a0000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe164097000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe163e81000) /lib64/ld-linux-x86-64.so.2 (0x00007fe164ee2000)

On Fri, Sep 7, 2018 at 5:42 AM Dario Izzo notifications@github.com wrote:

THis is not a crash. The program throws as you library is either not found (and this seems not to be the case) or it does not contain an API compatible with our plugin. Where did you take the library? Have you compiled it yourself?

Can you post the result of ldd on your snopt7_c library?

ldd libsnopt7_c.so

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/esa/pagmo_plugins_nonfree/issues/2#issuecomment-419384732, or mute the thread https://github.com/notifications/unsubscribe-auth/ADbqdFD6u0QIQYTzxRWHo7-AAbJLUXDZks5uYj-QgaJpZM4WeDfV .

[darioizzo]

Where is the library taken from? Do you compile it yourself? What version of SNOPT is this?

[aespielberg]

This was downloaded directly from the website. I did not compile it myself.

v2.1.0 for SNOPT 7.7

On Fri, Sep 14, 2018 at 3:14 AM Dario Izzo notifications@github.com wrote:

Where is the library taken from? Do you compile it yourself? What version of SNOPT is this?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/esa/pagmo_plugins_nonfree/issues/2#issuecomment-421254266, or mute the thread https://github.com/notifications/unsubscribe-auth/ADbqdJZTH3P5rr6T0gaKL0kZIa3r6Uwnks5ua1dRgaJpZM4WeDfV .

[darioizzo]

What is needed is for pagmo_plugins to work is the snopt7_c library to containing the following symbols:

            void snInit (snProblem *, char *, char *, int);
            int setIntParameter(snProblem *, char[], int);
            int setRealParameter(snProblem *, char[], double);
            void deleteSNOPT(snProblem *);
            int solveA(snProblem *, int, int, int, double, int, snFunA, int, int *, int *,
                                            double *, int, int *, int *, double *, double *, double *, double *,
                                            double *, int *, double *, double *, int *, double *, int *, int *,
                                            double *);

which I think it does (you can check easily).

Looking at your error message "undefined symbol: snstop_" it seems your library is not finding a different symbol when loaded in memory (dlopen) and it looks as a symbol from the snopt fortran library which you are not linked to. This does not look as a problem with pagmo_plugins_nonfree, rather with the snopt7 libraries you have downloaded. If you have the symbols above (which I think its likely at this point), I would try to contact them and ask why their library cannot be find that snstop symbol missing.

It is also weird for me that your library does not link to the fortran library. The way I usually install snopt7 is to get the fortran library and then compile snopt7_c from https://github.com/snopt/snopt-interface. If I were you I would try.

On my system the libsnopt7_c.so library links also to the fortran library libsnopt7.so

I am curious to try and fix this issue, so if you need help ask here.

[aespielberg]

How can I check that I have those symbols?

On Fri, Sep 14, 2018 at 1:48 PM Dario Izzo notifications@github.com wrote:

Looking better at your error message "undefined symbol: snstop_" it seems your library is not finding a symbol when loaded in memory. Does not look as a problem with pagmo_plugins_non_free, rather with the snopt7 libraries you have downloaded. If you have the symbols above (which I think its likely at this point), I would try to contact them and ask why their library cannot be loaded having that symbol missing.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/esa/pagmo_plugins_nonfree/issues/2#issuecomment-421434637, or mute the thread https://github.com/notifications/unsubscribe-auth/ADbqdNNazw8A3HM9Vo8u7bguYuOr_LPfks5ua-vqgaJpZM4WeDfV .

[darioizzo]

I am not currently in a linux system, but the following command should give you the list:

nm -D libsnopt7_c.so

[darioizzo]

On my OSX this is the result:

(pagmo) Darios MacBook Pro:lib darioizzo$ nm -gU libsnopt7_c.dylib 
00000000000091a0 S ___snopt_wrapper_MOD_cw
000000000000a140 S ___sqopt_wrapper_MOD_cw
000000000000454b T _allocI
0000000000006632 T _allocIQ
0000000000004582 T _allocR
0000000000006669 T _allocRQ
000000000000645c T _deleteSNOPT
0000000000007422 T _deleteSQOPT
0000000000001060 T _f_npkern
0000000000001250 T _f_npopt
0000000000000ff0 T _f_snend
0000000000000bd0 T _f_sngetc
0000000000000af0 T _f_sngeti
0000000000000a10 T _f_sngetr
0000000000002f70 T _f_sninit
0000000000002a00 T _f_snjac
00000000000022b0 T _f_snkera
0000000000001ac0 T _f_snkerb
00000000000013d0 T _f_snkerc
00000000000037f0 T _f_snkerq
00000000000021d0 T _f_snmem
0000000000002ba0 T _f_snmema
00000000000026b0 T _f_snopta
0000000000001ea0 T _f_snoptb
00000000000017a0 T _f_snoptc
0000000000000f00 T _f_snset
0000000000000e10 T _f_snseti
0000000000002e10 T _f_snsetprint
0000000000000d20 T _f_snsetr
0000000000002c70 T _f_snspec
0000000000003780 T _f_sqend
0000000000003360 T _f_sqgetc
0000000000003280 T _f_sqgeti
00000000000031a0 T _f_sqgetr
0000000000004220 T _f_sqinit
0000000000003e60 T _f_sqmem
0000000000003b30 T _f_sqopt
0000000000003690 T _f_sqset
00000000000035a0 T _f_sqseti
00000000000040c0 T _f_sqsetprint
00000000000034b0 T _f_sqsetr
0000000000003f20 T _f_sqspec
0000000000004972 T _getIntParameter
000000000000481e T _getParameter
0000000000004ab1 T _getRealParameter
0000000000004494 T _init2zero
000000000000659f T _init2zeroQ
00000000000045b9 T _reallocI
00000000000066a0 T _reallocIQ
000000000000461a T _reallocR
0000000000006701 T _reallocRQ
00000000000048d5 T _setIntParameter
0000000000004e4a T _setLog
0000000000004791 T _setParameter
000000000000467b T _setPrintfile
0000000000004a10 T _setRealParameter
0000000000004e84 T _setSTOP
0000000000004704 T _setSpecsfile
0000000000004b51 T _setUserI
0000000000004b78 T _setUserR
0000000000004b9f T _setUserspace
0000000000004be4 T _setWorkspace
0000000000004d27 T _setWorkspaceA
00000000000043b1 T _snInit
0000000000004e9e T _snoptA
0000000000005692 T _snoptB
0000000000005d86 T _snoptC
0000000000005298 T _solveA
0000000000005a0c T _solveB
00000000000060f1 T _solveC
0000000000006dee T _solveQ
0000000000006a59 T _sqGetIntParameter
0000000000006905 T _sqGetParameter
0000000000006b98 T _sqGetRealParameter
00000000000064bc T _sqInit
0000000000006762 T _sqPrintfile
00000000000069bc T _sqSetIntParameter
0000000000006dd4 T _sqSetLog
0000000000006878 T _sqSetParameter
0000000000006af7 T _sqSetRealParameter
00000000000067eb T _sqSpecsfile
0000000000006c38 T _sqUserI
0000000000006c5f T _sqUserR
0000000000006c86 T _sqUserspace
0000000000006ccb T _sqWorkspace
0000000000007108 T _sqopt

[aespielberg]

Here is the result:

nm -D libsnopt7_c.so
...
000000000020b360 B __snopt_wrapper_MOD_cw
                 U snset_
                 U snseti_
                 U snsetr_
                 U snspec_
                 U snspecf_
                 U snstop_
...

[darioizzo]

You see that the required symbols are there defining the correct API version.

[darioizzo]

On my fortran library instead I have:

(pagmo) Darios MacBook Pro:lib darioizzo$ nm -gU libsnopt7.dylib 
```bash
....
0000000000002c10 T _snseti_
0000000000008dd0 T _snsetr_
0000000000005100 T _snsolf_
0000000000004140 T _snspec_
0000000000008d40 T _snstop_
0000000000002e50 T _sntitl_
0000000000003110 T _snwrap_
...

as you see there I have the symbol snstop_ you are missing and that is probably used by the snopt7_c library.

[darioizzo]

Can you format your symbol list with markdown please?

Anyhow, as you see from your list the line:

 U snstop_

says that the library needs the symbol snstop_ but its undefined. As a consequence that library will only work if when loaded in memory, the fortran library that defines that symbol has already been loaded.

[darioizzo]

It seems to me your snopt7_c library should be linked to the fortran library libsnopt7.so, but its not. To me this seems like a mistake (or may be a feature, who knows) of the libsnopt7c.so you downloaded.

[aespielberg]

Is there a way to manually link them?

[darioizzo]

If you do have the snopt library (should be called libsnopt7.so under linux), then download for github and compile the project https://github.com/snopt/snopt-interface

That project will create the libsnopt7_c.so library and link it to the libsnopt7.so library.

[aespielberg]

I'm sorry for so many questions. I'm a little confused. There does not appear to be a way to build snopt-interface as a library (or at least no insructions to do so) there, all I see is a way to build binary optimization programs that link to existing .so files.

Am I missing something?

[darioizzo]

Download the code in a folder using

git clone  https://github.com/snopt/snopt-interface
cd snopt-interface
./autogen
./configure
make snopt_c
make install snopt_c

the library will be created in the folder snopt-interface/lib and it will link to your libsnopt7.so library (which must be found by the system otherwise will appear as missing)

[aespielberg]

Sorry for the extreme ignroance here. Still cannot build that library. Why would I get this?

$ ./configure 
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking how to print strings... printf
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking for gawk... no
checking for mawk... mawk
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... mt
checking if mt is a manifest tool... no
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking for gfortran... no
checking for g95... no
checking for xlf95... no
checking for f95... no
checking for fort... no
checking for ifort... no
checking for ifc... no
checking for efc... no
checking for pgfortran... no
checking for pgf95... no
checking for lf95... no
checking for ftn... no
checking for nagfor... no
checking for xlf90... no
checking for f90... no
checking for pgf90... no
checking for pghpf... no
checking for epcf90... no
checking for g77... no
checking for xlf... no
checking for f77... no
checking for frt... no
checking for pgf77... no
checking for cf77... no
checking for fort77... no
checking for fl32... no
checking for af77... no
checking whether we are using the GNU Fortran compiler... no
checking whether  accepts -g... no
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking how to get verbose linking output from ... configure: WARNING: compilation failed

checking for Fortran libraries of ... 
checking Fortran 90 module output flag... unknown
checking Fortran 90 module inclusion flag... unknown
checking for C interface... yes
checking for C++ interface... yes
sed: can't read confdefs.h: No such file or directory
configure: creating ./config.status
config.status: error: cannot find input file: `Makefile.in'

[darioizzo]

This seems to be a problem with your linux system. Have you ever built anything successfully in your linux? If not you might be missing dev. packages to allow and build code.

You need to have a system that is able to build succesfully code using the standard toolchain of autoconf. If you have a linux system with apt-get, try:

sudo apt-get install autotools-dev
sudo apt-get install autoconf

[shreya-dupa]

I am trying to build SNOPT with the PaGMO interface in Windows. Do you know if pagmo requires a the snopt7 library to be a .so file? Currently when I build SNOPT from source using MinGW with shared libraries enabled, it gives me a .dll file.

[darioizzo]

.dll is fine. As far as all its dependenies are also found. When building with mingw you need to make all the mingw related libraries also be found. (you can use dependency walker to know what the dll needs)

[darioizzo]

I would be interested to know if you succeed. Can you post here any progress?

[shreya-dupa]

Sure, I'm curious if you have SNOPT currently working with this interface.

Additionally, I'm trying to determine whether to use the 64-bit MinGW toolkit or the 32-bit one. Do you have any insight on this?

[darioizzo]

Yes the code is tested and work beautifully with my version of SNOPT7 (I have the source files). Here is how I do it: 1) I compile the fortran library 2) I use the github snopt-interface code to generate the snopt_c library. But I need to use an old version as they have changed the API and the headers files (breaking compatibility w.r.t. previous APIs). So I need to use the commit (https://github.com/snopt/snopt-interface/commit/76b166ecdf5c55a3289ce0f849d8d3d101954a22)

I used both 32 and 64 to do the above "chain" and produce the dlls.

[darioizzo]

I am working to a new interface compatible with snopt 77, see #6, in case it affects you

[darioizzo]

@shreya-dupa

The PR #7 introduces the possibility of specifying the snopt api to be used and has been tested on 7.7 (evaluation copy from the snopt web site). Let me know if it works for you too.

[darioizzo]

I close the issue, if a problem is found with the new code open a new issue.

[shreya-dupa]

@darioizzo Sorry for the late response. Had been working on other projects.

I have built the fortran libraries for SNOPT in 64bit using MSYS2 toolkit. The C interface has also been compiled and linked. I don't know how to check a dll to see what symbol it contains. I have also installed pyGMO and pygmo_plugins_nonfree using pip for Python 3.

Just have to test and see if I can call SNOPT correctly.

[shreya-dupa]

I will open a new issue if I run into problems

[darioizzo]

You can check the dll needed by the dll you compiled using "dependency walker", a small software useful in this case. When compiling with minGW the dll of snopt will require a number of mingw dll that, if not found, will give you problems.

[orsoromeo]

I am also experiencing the same configure error that @aespielberg reported while trying to install the snopt-interface: I would be curious to know if and how he solved it.

[darioizzo]

seems a problem of interaction between your environment and snopt-interface, maybe raise the issue there? (https://github.com/snopt/snopt-interface)

[klevis-a]

I experienced the same error as @aespielberg. The key to fixing the issue was to use the specific commit that @darioizzo references. I tried several of the releases from snopt-interface with no luck.