Open hjmshi opened 6 years ago
Hi,
I do believe that your AMPL is not linked properly with your Python which raise an issue when you are importing Cutestmodel.py
Don't forget to copy the file site.template.cfg in cutest/tools as site.cfg to compile correctly your CUTEst problem
Thanks, I have fixed the AMPL linking problem - had to do with not having installed ASL (or AMPL-MP). (FYI, since homebrew/science was deprecated, one now obtains ASL from homebrew/core under ampl-mp.)
I also had to manually define the environment variables according to the Travis file. There was a typo there - it should be SIFDECODE, not SIFEDECODE, and the paths needs to be adjusted accordingly.
I did copy the site.template.cfg as site.cfg, but am now running into the issue when running python setup.py test
:
running egg_info
writing CUTEst.py.egg-info/PKG-INFO
writing top-level names to CUTEst.py.egg-info/top_level.txt
writing dependency_links to CUTEst.py.egg-info/dependency_links.txt
reading manifest file 'CUTEst.py.egg-info/SOURCES.txt'
writing manifest file 'CUTEst.py.egg-info/SOURCES.txt'
running build_ext
Reading problem HS13...
dyld: Library not loaded: /usr/local/opt/gcc/lib/gcc/7/libgfortran.4.dylib
Referenced from: /usr/local/Cellar/sifdecode/0.4/libexec/objects/mac64.osx.gfo/double/run_sifdecode
Reason: image not found
/usr/local/bin/sifdecoder: line 451: 85792 Abort trap: 6 $DECODER
ERROR: Error exit from decoding stage. terminating execution.
Exception AttributeError: "'CUTEstModel' object has no attribute '_name'" in <object repr() failed> ignored
Do you know why this might be? Thanks again.
EDIT: I noticed that my gcc version is 8.2, so the library that it probably should be looking for is under /usr/local/opt/gcc/lib/gcc/8/libgfortran.5.dylib
. Perhaps sifdecode wasn't installed correctly?
It seems that the library want to include gfortran which was only the case for Linux during the problem importation.
I suggest you to look at the function compile_SIF (in the file cutest/tools/compile.py), called during initialization of the CUTEstModel. Basically, this function call the problem from CUTEst and link it to Python with Cython.
If everything is installed well, I suggest you to follow manually this method and check where the issue is raised.
I've tried to do this now when trying model = CUTEstModel('ROSENBR')
. I encounter *** OSError: [Errno 2] No such file or directory
at line 75: subprocess.call(['sifdecoder', problem_name])
in compile.py. How does the SIFDecoder know where to find the problem?
Checking the temporary folder, I see three files: ROSENBR.c, ROSENBR.pxd, and ROSENBR.pyx.
Hi Michael,
Sorry about all those difficulties.
Did you set the MASTSIF environment variable?
Dominique
On Aug 22, 2018, at 17:34, Hao-Jun Michael Shi notifications@github.com wrote:
I've tried to do this now when trying model = CUTEstModel('ROSENBR'). I encounter *** OSError: [Errno 2] No such file or directory at line 75: subprocess.call(['sifdecoder', problem_name]) in compile.py. How does the SIFDecoder know where to find the problem?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.
Hi Dominique,
I did I believe. I used export MASTSIF="$(brew --prefix)/Cellar/mastsif/0.4/share/mastsif"
.
I realize now that I have two versions of sifdecode, cutest, archdefs, and mastsif installed. For example, I have sifdecode 0.4 and 2.0.0. Should I be using 2.0.0? I'm not sure why I have two versions - perhaps I tried to install before and wound up with an older version and the version provided by Homebrew.
Michael
EDIT: I've modified all of the environment variables to use 2.0.0 (except MASTSIF) since all the tests from Homebrew seem to use 2.0.0. I've also modified mastsif to use 0.3 instead of 0.4.
The problems are now successfully found and decoded. I'm now encountering a different issue:
running test
running egg_info
writing CUTEst.py.egg-info/PKG-INFO
writing top-level names to CUTEst.py.egg-info/top_level.txt
writing dependency_links to CUTEst.py.egg-info/dependency_links.txt
reading manifest file 'CUTEst.py.egg-info/SOURCES.txt'
writing manifest file 'CUTEst.py.egg-info/SOURCES.txt'
running build_ext
Reading problem HS13...
Problem name: HS13
Double precision version will be formed
The objective function uses 1 nonlinear group
There is 1 nonlinear inequality constraint
There are 2 variables bounded only from below
File successfully decoded
ld: warning: directory not found for option '-L$(brew --prefix)/lib'
ld: warning: could not create compact unwind for ___cutest_MOD_cutest_assemble_hessian_pattern: stack subq instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for ___cutest_MOD_cutest_assemble_hessian: stack subq instruction is too different from dwarf stack size
Exception AttributeError: "'CUTEstModel' object has no attribute '_name'" in <object repr() failed> ignored
Traceback (most recent call last):
File "setup.py", line 15, in <module>
test_suite = "cutest.test.test_cutest"
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/setuptools/__init__.py", line 129, in setup
return distutils.core.setup(**attrs)
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/setuptools/command/test.py", line 226, in run
self.run_tests()
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/setuptools/command/test.py", line 248, in run_tests
exit=False,
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 94, in __init__
self.parseArgs(argv)
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 149, in parseArgs
self.createTests()
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 158, in createTests
self.module)
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_cutest'
Sorry again for bothering the both of you - I think this is very close to being resolved.
Indeed, I think that we are close to solve your issue. It seems that the issue is raised at the final compilation (l.113).
Can you check if your library cutest is present at the path $(brew --prefix)/lib?
There is no cutest directory, but it does contain files libcutest.a, libcutest.dylib, libcutest_double.a, libcutest_double.dylib, libcutest_single.a, and libcutest_single.dylib. Is this what is needed?
Yes, I think that should be good for the last compilation
But actually, I think that your issue raise at line 100. What are your CUTEST and MYARCH environment variable?
I'm defining my CUTEST and MYARCH environment variables as the following:
export ARCHDEFS="$(brew --prefix)/Cellar/archdefs/2.0.0/libexec"
export CUTEST="$(brew --prefix)/Cellar/cutest/2.0.0/libexec"
There also an environment variable MYARCH to define. For example, in the travis MYARCH is defined as follow
export MYARCH=mac64.osx.gfo
It allows to access to the library located in the path $CUTEST_DIR/objects/$MYARCH/double
Oh sorry! I thought you said ARCHDEFS. I did define MYARCH as:
export MYARCH=mac64.osx.gfo
Can you confirm that the path $CUTEST_DIR/objects/$MYARCH/double exist on your computer?
It does. It contains the following files:
algencan_main.o dfo_main.o gen90_main.o lbfgs_main.o newuoa_main.o praxis_main.o tenmin_main.o
bobyqa_main.o directsearch_main.o genc.o lbfgsb_main.o nitsol_main.o ql_main.o test_main.o
ccutest.c filtersd_main.o genc_main.o libcutest.a nlpqlp_main.o qplib_main.o tron_main.o
cg_descent_main.o filtersqp_main.o hrb_main.o libcutest_double.dylib npsol_main.o snopt_main.o uncmin_main.o
cgplus_main.o gen77.o ipopt_main.o lincoa_main.o osqp_main.o spg_main.o vf13_main.o
cobyla_main.o gen77_main.o knitro_main.o loqo_main.o pds_main.o stats_main.o worhp_main.o
derchk_main.o gen90.o la04_main.o minos_main.o pennlp_main.o stenmin_main.o
Something else that may help with debugging: when I try to load the Rosenbrock problem
from cutest.model.cutestmodel import CUTEstModel
model = CUTEstModel('ROSENBR')
I obtain the following error
cp: cutest/model/src/cutest.pyx: No such file or directory
cp: cutest/model/src/cutest.pxd: No such file or directory
[Errno 2] No such file or directory: '/private/var/folders/m6/qrrd98hx2gsfrv1cbknm292h0000gn/T/tmpfPvpGJ/ROSENBR.pyx'
Problem name: ROSENBR
Double precision version will be formed
The objective function uses 1 linear group
The objective function uses 1 nonlinear group
There are 2 free variables
File successfully decoded
clang: error: no such file or directory: 'ROSENBR.c'
clang: error: no input files
clang: error: no such file or directory: 'ROSENBR.o'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "cutest/model/cutestmodel.py", line 35, in __init__
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named ROSENBR
The fact that you have this message seems to mean that the source file has not been copied in your Python during the installation.
I suggest you to execute once again 'python setup.py install' and check if you have the script cutest.pyx, cutest.pxd in your repository CUTEst.py-0.0.1-py2.7.egg/cutest from your python site-package,
I'm sorry that you have all these difficulties. As I told you, I suggest you to complete manually the procedure defined in the function compile_SIF. At the end of the process, you should have a dynamic library containing the CUTEst problem loadable in Python.
The weird thing is that this didn't happen before - let me play with this and the environment variables and get back to you.
OK, I believe I have fixed compile_SIF - I need to add ".SIF" to the end of the problem name when calling the sifdecoder, otherwise it does not work. The error I'm now encountering has to do with importing the Cython library, I believe.
File "<ipython-input-5-02baaee68558>", line 1, in <module>
runfile('/Users/hjmshi/Downloads/temp', wdir='/Users/hjmshi/Downloads')
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 705, in runfile
execfile(filename, namespace)
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 94, in execfile
builtins.execfile(filename, *where)
File "/Users/hjmshi/Downloads/temp", line 11, in <module>
model = CUTEstModel('ROSENBR.SIF')
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/CUTEst.py-0.0.1-py2.7.egg/cutest/model/cutestmodel.py", line 35, in __init__
cc = importlib.import_module(cython_lib_name)
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/pyximport/pyximport.py", line 458, in load_module
language_level=self.language_level)
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/pyximport/pyximport.py", line 233, in load_module
exec("raise exc, None, tb", {'exc': exc, 'tb': tb})
File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/pyximport/pyximport.py", line 216, in load_module
mod = imp.load_dynamic(name, so_path)
ImportError: Building module ROSENBR failed: ['ImportError: dlopen(/Users/hjmshi/.pyxbld/lib.macosx-10.6-x86_64-2.7/ROSENBR.so.reload3, 2): Symbol not found: _cutest_cdh_\n Referenced from: /Users/hjmshi/.pyxbld/lib.macosx-10.6-x86_64-2.7/ROSENBR.so.reload3\n Expected in: flat namespace\n in /Users/hjmshi/.pyxbld/lib.macosx-10.6-x86_64-2.7/ROSENBR.so.reload3\n']
Any ideas on how to resolve this?
I suggest you to compile manually the Cython library.
If you look at the code, once you have decode the problem with Sifdecoder, you should have cythonize the Cython file which create a C file.
So the next step is to compile the C file with cutest, Python and numpy (l. 93) Then, the code compile some Fortran codes at the line 96 and link them to the cutest problem and cutest library (line 100). And finally, the dynamic library which is callable from python is create at line 113.
Could tell me which one of these step is not executed well? I'm pretty sure all that those difficulties are due to one of those manipulation.
At line 113, I receive the following warnings:
ld: warning: could not create compact unwind for ___cutest_MOD_cutest_assemble_hessian_pattern: stack subq instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for ___cutest_MOD_cutest_assemble_hessian: stack subq instruction is too different from dwarf stack size
Everything else appears to work fine when I manually compile the Cython library (going line-by-line).
I'm currently trying to use pycutest on Google Colab but I'm having an issue setting the environment variables CUTEST and MYARCH correctly, based on the above I can't tell what they should be set to. I get the error:
RuntimeError: Could not find CUTEST installation - have CUTEST and MYARCH environment variables been set correctly?
@Vonny0123 Step 2 in this README explains what the environment variables should be.
Thanks for providing this Python interface! I'm running into some problems with the installation.
When I run
python setup.py test
, I receive the following error:Could I get your help with addressing this issue?
Thanks in advance!