Symbol not found: _CCtsp_solve_dat - Error after installing pyconcorde

[michelgokan]

I'm using Python 3.8.1 on Mac OS X 10.15.2 (brew-based) with Xcode 11.3.1. I've succefully installed pyconcorde without any errors but when I import TSPSolver using from concorde.tsp import TSPSolver I'm getting the "Symbol not found: _CCtsp_solve_dat" error. It seems concorde/tsp.py cannot import _CCtsp_solve_dat from concorde._concorde in line 10.

What I run: >>> from concorde.tsp import TSPSolver What I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/a/pyconcorde/concorde/tsp.py", line 10, in <module>
    from concorde._concorde import _CCutil_gettsplib, _CCtsp_solve_dat
ImportError: dlopen(/Users/a/pyconcorde/concorde/_concorde.cpython-38-darwin.so, 2): Symbol not found: _CCtsp_solve_dat
  Referenced from: /Users/a/pyconcorde/concorde/_concorde.cpython-38-darwin.so
  Expected in: flat namespace
 in /Users/a/pyconcorde/concorde/_concorde.cpython-38-darwin.so

[michelgokan]

New update: Downgrading to 3.7.6 didn't fix the issue

[michelgokan]

Update 2: Using Python 2.7 didn't fix the issue either.

Python 2.7.17 (default, Dec 23 2019, 21:25:33) 
[GCC 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from concorde.tsp import TSPSolver
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "concorde/tsp.py", line 10, in <module>
    from concorde._concorde import _CCutil_gettsplib, _CCtsp_solve_dat
ImportError: dlopen(concorde/_concorde.so, 2): Symbol not found: _CCtsp_solve_dat
  Referenced from: concorde/_concorde.so
  Expected in: flat namespace
 in concorde/_concorde.so

[michelgokan]

Update 3: I've tried the exact same procedure on a fresh Ubuntu 18.04 installation and everything works pretty well! It might be the case that there are some inconsistencies in my Mac's libraries, I'm not sure. Have you ever tested it on the latest version of Mac OS? If it works fine then do you have any guides on how to debug and fix the problem in my machine?

[jvkersch]

@michelgokan Thanks for reporting the problem and for doing the investigation. I currently don't have access to a Mac to test, but this is worrisome since Mac OS is supposed to be a supported platform. I'll try to reproduce in the next few days.

[iliailmer]

Hi, have the same issue but running on an M1 macbook pro with arm64 architecture, seems like it requires a different concorde build?

[jvkersch]

@iliailmer Thanks for the follow-up. I am not sure about how to support the M1. One thing to note is that I am planning to move to a new API that runs Concorde in a subprocess (see https://github.com/jvkersch/pyconcorde/issues/28#issuecomment-770354999). If you can get Concorde to compile and run on your platform, then this new API should be compatible with it. I'd be very interested in hearing your experiences.

[iliailmer]

@jvkersch Hey, thanks for the response. I will check it out and let you know.

[iliailmer]

@jvkersch Hey does this require downloading QS_opt or CPLEX as well?

Sorry, had to remove previous question, the executable is the TSP/concorde file after running make. I placed in the $HOME/bin/ directory but i still get

ConcordeNotFoundError: Could not run the concorde executable at 'concorde'. Check that the concorde executable is present and can be run.

the $HOME/bin is added to PATH

Had to restart my shell to make it work.

[samvanderpoel]

Hi, I am having a similar problem on MacOS (Intel chip, and software update 12.2.1 -> 12.3 did not help). I got the following warnings before building

WARNING: Missing build requirements in pyproject.toml for file:///Users/samvanderpoel/Desktop/Projects/TSP/test_dir/pyconcorde. WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'wheel'.

Then in the Traceback:

File ".../pyconcorde/concorde/tsp.py", line 10, in <module> from concorde._concorde import _CCutil_gettsplib, _CCtsp_solve_dat

ImportError: dlopen(/Users/samvanderpoel/Desktop/Projects/TSP/test_dir/pyconcorde/concorde/_concorde.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_QSadd_cols'

when trying to import.

It is strange because I have previously installed and used pyconcorde extensively on the same machine. I am now trying to install in a new conda environment.

Any updates on how to solve this problem? Thank you!

[DangerousDyl]

Hi I am receiving the same issue:

dlopen(/Users/dylanjohnson/gnn-tsp/pyconcorde/concorde/_concorde.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_QSadd_cols'

Thanks, Dylan

[Technophobe01]

I was able to resolve this problem. This error appears to be caused by an incompatibility between compiling and linking libraries compiled with clang and GCC, that and the underlying Concorde library has no config for MacOSX.

1. Before running pip install -e . you must go into the ./pyconcorde/build/concorde directory and manually configure it to build and compile Concorde.
2. It is important to unalias gcc, and then set the location of GCC export CC="/usr/local/Cellar/gcc/12.2.0/bin/gcc-12"
3. At this point, you can set the export QSOPTDIR=pwd/QSOPT and export CFLAGS="-g -O3 -arch x86_64", run configure and build.
4. Now you can go back and set the locations of Concorde, and QSOPT libraries and run pip install -e .
5. Load Python and use Concorde wrapper

Reproduction of Error:

git clone https://github.com/jvkersch/pyconcorde.git
cd pyconcorde
pip install -e .                                                                                 [2.7.2]
Obtaining file:///Users/.../test/pyconcorde
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: cython>=0.22.0 in /Users/.../venv/lib/python3.9/site-packages (from pyconcorde==0.1.0) (0.29.33)
Requirement already satisfied: numpy>=1.10.0 in /Users/.../venv/lib/python3.9/site-packages (from pyconcorde==0.1.0) (1.24.2)
Building wheels for collected packages: pyconcorde
  Building editable for pyconcorde (pyproject.toml) ... done
  Created wheel for pyconcorde: filename=pyconcorde-0.1.0-0.editable-cp39-cp39-macosx_13_0_x86_64.whl size=3596 sha256=ea7722c0faad213b7eef0d80b7d7fb0582dd2ff94aa99c7b022658eb35a99dce
  Stored in directory: /private/var/folders/0m/scybmx2x643b9hw9_yjtrky00000gn/T/pip-ephem-wheel-cache-twt6ev3o/wheels/01/52/9c/470b44bc70ceef5ac677fc113e6af8de1a54bd719607b55284
Successfully built pyconcorde
Installing collected packages: pyconcorde
  Attempting uninstall: pyconcorde
    Found existing installation: pyconcorde 0.1.0
    Uninstalling pyconcorde-0.1.0:
      Successfully uninstalled pyconcorde-0.1.0
Successfully installed pyconcorde-0.1.0
(venv).../test/pyconcorde (master●)$ python3                                                                                         [2.7.2]
Python 3.9.16 (main, Dec  7 2022, 10:16:11)
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from concorde.tsp import TSPSolver
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/.../test/pyconcorde/concorde/tsp.py", line 10, in <module>
    from concorde._concorde import _CCutil_gettsplib, _CCtsp_solve_dat
ImportError: dlopen(/Users/.../test/pyconcorde/concorde/_concorde.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_QSadd_cols'
>>>

Resolution of Error:

unalias gcc
export CC="/usr/local/Cellar/gcc/12.2.0/bin/gcc-12"
cd build/concorde
mkdir QSOPT;cd QSOPT
wget 'https://www.math.uwaterloo.ca/~bico/qsopt/downloads/codes/mac64/qsopt.gz'
wget 'https://www.math.uwaterloo.ca/~bico/qsopt/downloads/codes/mac64/qsopt.a.gz'
wget 'https://www.math.uwaterloo.ca/~bico/qsopt/downloads/codes/mac64/qsopt.h'
gunzip qsopt.a.gz
cd ..
export QSOPTDIR=`pwd`/QSOPT
export CFLAGS="-g -O3 -arch x86_64"
./configure --with-qsopt=$QSOPTDIR --host=darwin
make

# Check build succeeded
ls -la TSP/concorde

# go back to pyconcorde directory
cd ../../
export QSOPT_DIR=`pwd`/build/concorde/QSOPT
export CONCORDE_DIR=`pwd`/build/concorde/
pip install -e . 
Obtaining file:///Users/pkjar/Cloud/Dropbox/dev/stackoverflow/75792660/test/pyconcorde
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: cython>=0.22.0 in /usr/local/lib/python3.11/site-packages (from pyconcorde==0.1.0) (0.29.33)
Requirement already satisfied: numpy>=1.10.0 in /usr/local/lib/python3.11/site-packages (from pyconcorde==0.1.0) (1.24.2)
Installing collected packages: pyconcorde
  Running setup.py develop for pyconcorde
Successfully installed pyconcorde-0.1.0

python3
Python 3.11.2 (main, Mar 24 2023, 00:31:37) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from concorde.tsp import TSPSolver
>>>

[Entman]

Just wanted to chime an and confirm that the above solution works, one note - if the "unalias gcc" command isn't working for you chances are you just simply don't have gcc installed like I did - in this case you just simply need to install it with Homebrew. @Technophobe01 thanks for the fix. I am toying with an idea of making a youtube video that would show how to install this library on mac, given the fact that I've spent a good few hours fighting with this shit lel

[rgan19]

I was stuck trying to install this for a couple of hours on my Mac and came across this. Also had this error: symbol not found in flat namespace '_QSadd_cols'. Had to install gcc with homebrew (brew install gcc), find the path then export CC="/usr/local/Cellar/gcc/13.1.0/bin/gcc-13". This line is really important.

@Technophobe01 Thanks for the fix, saved the day.

It finally worked.

[jvkersch]

This is very strange; the build process should normally pull in QSOpt and link against it correctly. I have no idea why this happens and I cannot reproduce it on my machine either. It would be very helpful if someone who encountered this problem in the past could find the Concorde shared library (compiled with the instructions in the README, not with the workaround) in the pyconcorde/concorde directory and run otool and nm on it. For example, I get the following:

$ otool -L _concorde.cpython-310-darwin.so                         (pyconcorde)
_concorde.cpython-310-darwin.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

and

$ nm -g _concorde.cpython-310-darwin.so | grep _QSadd_cols         (pyconcorde)
0000000000053974 T _QSadd_cols

The latter tells me that the symbol _QSadd_cols is present in the compiled library; I am interested to see what it would look like for one of the problems installs.

Also, anything else that you could find to pinpoint the problem would be helpful. I am in the process of moving the library to a subprocess-based approach, which will make this issue go away in due course, but I would like to resolve this issue right now to avoid having people lose more time on it.