Issue with CUTEST.py test

[hjmshi]

Thanks for providing this Python interface! I'm running into some problems with the installation.

When I run python setup.py test, I receive the following error:

running egg_info
writing CUTEst.py.egg-info/PKG-INFO
writing top-level names to CUTEst.py.egg-info/top_level.txt
writing dependency_links to CUTEst.py.egg-info/dependency_links.txt
reading manifest file 'CUTEst.py.egg-info/SOURCES.txt'
writing manifest file 'CUTEst.py.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "setup.py", line 15, in <module>
    test_suite = "cutest.test.test_cutest"
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/setuptools/command/test.py", line 226, in run
    self.run_tests()
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/setuptools/command/test.py", line 248, in run_tests
    exit=False,
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_cutest'

Could I get your help with addressing this issue?

Thanks in advance!

[kenjydem]

Hi,

I do believe that your AMPL is not linked properly with your Python which raise an issue when you are importing Cutestmodel.py

Don't forget to copy the file site.template.cfg in cutest/tools as site.cfg to compile correctly your CUTEst problem

[hjmshi]

Thanks, I have fixed the AMPL linking problem - had to do with not having installed ASL (or AMPL-MP). (FYI, since homebrew/science was deprecated, one now obtains ASL from homebrew/core under ampl-mp.)

I also had to manually define the environment variables according to the Travis file. There was a typo there - it should be SIFDECODE, not SIFEDECODE, and the paths needs to be adjusted accordingly.

I did copy the site.template.cfg as site.cfg, but am now running into the issue when running python setup.py test:

running egg_info
writing CUTEst.py.egg-info/PKG-INFO
writing top-level names to CUTEst.py.egg-info/top_level.txt
writing dependency_links to CUTEst.py.egg-info/dependency_links.txt
reading manifest file 'CUTEst.py.egg-info/SOURCES.txt'
writing manifest file 'CUTEst.py.egg-info/SOURCES.txt'
running build_ext
Reading problem HS13...
dyld: Library not loaded: /usr/local/opt/gcc/lib/gcc/7/libgfortran.4.dylib
  Referenced from: /usr/local/Cellar/sifdecode/0.4/libexec/objects/mac64.osx.gfo/double/run_sifdecode
  Reason: image not found
/usr/local/bin/sifdecoder: line 451: 85792 Abort trap: 6           $DECODER
 ERROR: Error exit from decoding stage. terminating execution.
Exception AttributeError: "'CUTEstModel' object has no attribute '_name'" in <object repr() failed> ignored

Do you know why this might be? Thanks again.

EDIT: I noticed that my gcc version is 8.2, so the library that it probably should be looking for is under /usr/local/opt/gcc/lib/gcc/8/libgfortran.5.dylib. Perhaps sifdecode wasn't installed correctly?

[kenjydem]

It seems that the library want to include gfortran which was only the case for Linux during the problem importation.

I suggest you to look at the function compile_SIF (in the file cutest/tools/compile.py), called during initialization of the CUTEstModel. Basically, this function call the problem from CUTEst and link it to Python with Cython.

If everything is installed well, I suggest you to follow manually this method and check where the issue is raised.

[hjmshi]

I've tried to do this now when trying model = CUTEstModel('ROSENBR'). I encounter *** OSError: [Errno 2] No such file or directory at line 75: subprocess.call(['sifdecoder', problem_name]) in compile.py. How does the SIFDecoder know where to find the problem?

Checking the temporary folder, I see three files: ROSENBR.c, ROSENBR.pxd, and ROSENBR.pyx.

[dpo]

Hi Michael,

Sorry about all those difficulties.

Did you set the MASTSIF environment variable?

Dominique

On Aug 22, 2018, at 17:34, Hao-Jun Michael Shi notifications@github.com wrote:

I've tried to do this now when trying model = CUTEstModel('ROSENBR'). I encounter *** OSError: [Errno 2] No such file or directory at line 75: subprocess.call(['sifdecoder', problem_name]) in compile.py. How does the SIFDecoder know where to find the problem?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[hjmshi]

Hi Dominique,

I did I believe. I used export MASTSIF="$(brew --prefix)/Cellar/mastsif/0.4/share/mastsif".

I realize now that I have two versions of sifdecode, cutest, archdefs, and mastsif installed. For example, I have sifdecode 0.4 and 2.0.0. Should I be using 2.0.0? I'm not sure why I have two versions - perhaps I tried to install before and wound up with an older version and the version provided by Homebrew.

Michael

EDIT: I've modified all of the environment variables to use 2.0.0 (except MASTSIF) since all the tests from Homebrew seem to use 2.0.0. I've also modified mastsif to use 0.3 instead of 0.4.

[hjmshi]

The problems are now successfully found and decoded. I'm now encountering a different issue:

running test
running egg_info
writing CUTEst.py.egg-info/PKG-INFO
writing top-level names to CUTEst.py.egg-info/top_level.txt
writing dependency_links to CUTEst.py.egg-info/dependency_links.txt
reading manifest file 'CUTEst.py.egg-info/SOURCES.txt'
writing manifest file 'CUTEst.py.egg-info/SOURCES.txt'
running build_ext
Reading problem HS13...

 Problem name: HS13

 Double precision version will be formed

 The objective function uses 1 nonlinear group

 There is 1 nonlinear inequality constraint

 There are 2 variables bounded only from below

 File successfully decoded
ld: warning: directory not found for option '-L$(brew --prefix)/lib'
ld: warning: could not create compact unwind for ___cutest_MOD_cutest_assemble_hessian_pattern: stack subq instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for ___cutest_MOD_cutest_assemble_hessian: stack subq instruction is too different from dwarf stack size
Exception AttributeError: "'CUTEstModel' object has no attribute '_name'" in <object repr() failed> ignored
Traceback (most recent call last):
  File "setup.py", line 15, in <module>
    test_suite = "cutest.test.test_cutest"
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/setuptools/command/test.py", line 226, in run
    self.run_tests()
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/setuptools/command/test.py", line 248, in run_tests
    exit=False,
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_cutest'

Sorry again for bothering the both of you - I think this is very close to being resolved.

[kenjydem]

Indeed, I think that we are close to solve your issue. It seems that the issue is raised at the final compilation (l.113).

Can you check if your library cutest is present at the path $(brew --prefix)/lib?

[hjmshi]

There is no cutest directory, but it does contain files libcutest.a, libcutest.dylib, libcutest_double.a, libcutest_double.dylib, libcutest_single.a, and libcutest_single.dylib. Is this what is needed?

[kenjydem]

Yes, I think that should be good for the last compilation

But actually, I think that your issue raise at line 100. What are your CUTEST and MYARCH environment variable?

[hjmshi]

I'm defining my CUTEST and MYARCH environment variables as the following:

export ARCHDEFS="$(brew --prefix)/Cellar/archdefs/2.0.0/libexec"
export CUTEST="$(brew --prefix)/Cellar/cutest/2.0.0/libexec"

[kenjydem]

There also an environment variable MYARCH to define. For example, in the travis MYARCH is defined as follow

export MYARCH=mac64.osx.gfo

It allows to access to the library located in the path $CUTEST_DIR/objects/$MYARCH/double

[hjmshi]

Oh sorry! I thought you said ARCHDEFS. I did define MYARCH as:

export MYARCH=mac64.osx.gfo

[kenjydem]

Can you confirm that the path $CUTEST_DIR/objects/$MYARCH/double exist on your computer?

[hjmshi]

It does. It contains the following files:

algencan_main.o     dfo_main.o      gen90_main.o        lbfgs_main.o        newuoa_main.o       praxis_main.o       tenmin_main.o
bobyqa_main.o       directsearch_main.o genc.o          lbfgsb_main.o       nitsol_main.o       ql_main.o       test_main.o
ccutest.c       filtersd_main.o     genc_main.o     libcutest.a     nlpqlp_main.o       qplib_main.o        tron_main.o
cg_descent_main.o   filtersqp_main.o    hrb_main.o      libcutest_double.dylib  npsol_main.o        snopt_main.o        uncmin_main.o
cgplus_main.o       gen77.o         ipopt_main.o        lincoa_main.o       osqp_main.o     spg_main.o      vf13_main.o
cobyla_main.o       gen77_main.o        knitro_main.o       loqo_main.o     pds_main.o      stats_main.o        worhp_main.o
derchk_main.o       gen90.o         la04_main.o     minos_main.o        pennlp_main.o       stenmin_main.o

[hjmshi]

Something else that may help with debugging: when I try to load the Rosenbrock problem

from cutest.model.cutestmodel import CUTEstModel
model = CUTEstModel('ROSENBR')

I obtain the following error

cp: cutest/model/src/cutest.pyx: No such file or directory
cp: cutest/model/src/cutest.pxd: No such file or directory
[Errno 2] No such file or directory: '/private/var/folders/m6/qrrd98hx2gsfrv1cbknm292h0000gn/T/tmpfPvpGJ/ROSENBR.pyx'

 Problem name: ROSENBR

 Double precision version will be formed

 The objective function uses 1 linear group
 The objective function uses 1 nonlinear group

 There are 2 free variables

 File successfully decoded
clang: error: no such file or directory: 'ROSENBR.c'
clang: error: no input files
clang: error: no such file or directory: 'ROSENBR.o'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cutest/model/cutestmodel.py", line 35, in __init__
  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named ROSENBR

[kenjydem]

The fact that you have this message seems to mean that the source file has not been copied in your Python during the installation.

I suggest you to execute once again 'python setup.py install' and check if you have the script cutest.pyx, cutest.pxd in your repository CUTEst.py-0.0.1-py2.7.egg/cutest from your python site-package,

I'm sorry that you have all these difficulties. As I told you, I suggest you to complete manually the procedure defined in the function compile_SIF. At the end of the process, you should have a dynamic library containing the CUTEst problem loadable in Python.

[hjmshi]

The weird thing is that this didn't happen before - let me play with this and the environment variables and get back to you.

[hjmshi]

OK, I believe I have fixed compile_SIF - I need to add ".SIF" to the end of the problem name when calling the sifdecoder, otherwise it does not work. The error I'm now encountering has to do with importing the Cython library, I believe.

  File "<ipython-input-5-02baaee68558>", line 1, in <module>
    runfile('/Users/hjmshi/Downloads/temp', wdir='/Users/hjmshi/Downloads')

  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 705, in runfile
    execfile(filename, namespace)

  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 94, in execfile
    builtins.execfile(filename, *where)

  File "/Users/hjmshi/Downloads/temp", line 11, in <module>
    model = CUTEstModel('ROSENBR.SIF')

  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/CUTEst.py-0.0.1-py2.7.egg/cutest/model/cutestmodel.py", line 35, in __init__
    cc = importlib.import_module(cython_lib_name)

  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)

  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/pyximport/pyximport.py", line 458, in load_module
    language_level=self.language_level)

  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/pyximport/pyximport.py", line 233, in load_module
    exec("raise exc, None, tb", {'exc': exc, 'tb': tb})

  File "/Users/hjmshi/anaconda/envs/python2/lib/python2.7/site-packages/pyximport/pyximport.py", line 216, in load_module
    mod = imp.load_dynamic(name, so_path)

ImportError: Building module ROSENBR failed: ['ImportError: dlopen(/Users/hjmshi/.pyxbld/lib.macosx-10.6-x86_64-2.7/ROSENBR.so.reload3, 2): Symbol not found: _cutest_cdh_\n  Referenced from: /Users/hjmshi/.pyxbld/lib.macosx-10.6-x86_64-2.7/ROSENBR.so.reload3\n  Expected in: flat namespace\n in /Users/hjmshi/.pyxbld/lib.macosx-10.6-x86_64-2.7/ROSENBR.so.reload3\n']

Any ideas on how to resolve this?

[kenjydem]

I suggest you to compile manually the Cython library.

If you look at the code, once you have decode the problem with Sifdecoder, you should have cythonize the Cython file which create a C file.

So the next step is to compile the C file with cutest, Python and numpy (l. 93) Then, the code compile some Fortran codes at the line 96 and link them to the cutest problem and cutest library (line 100). And finally, the dynamic library which is callable from python is create at line 113.

Could tell me which one of these step is not executed well? I'm pretty sure all that those difficulties are due to one of those manipulation.

[hjmshi]

At line 113, I receive the following warnings:

ld: warning: could not create compact unwind for ___cutest_MOD_cutest_assemble_hessian_pattern: stack subq instruction is too different from dwarf stack size
ld: warning: could not create compact unwind for ___cutest_MOD_cutest_assemble_hessian: stack subq instruction is too different from dwarf stack size

Everything else appears to work fine when I manually compile the Cython library (going line-by-line).

[Vonny0123]

I'm currently trying to use pycutest on Google Colab but I'm having an issue setting the environment variables CUTEST and MYARCH correctly, based on the above I can't tell what they should be set to. I get the error:

RuntimeError: Could not find CUTEST installation - have CUTEST and MYARCH environment variables been set correctly?

[lambe]

@Vonny0123 Step 2 in this README explains what the environment variables should be.