Status messages and compilation settings

smmaurer commented 5 years ago

This PR includes a few pieces of maintenance that I'm working on in parallel:

1. Updates the compilation settings in `setup.py`

Recent versions of OS X weren't able to install Pandana from source because of outdated compiler settings. I think i've fixed it. This procedure is working on multiple machines:

check that you have Xcode command line tools installed: xcode-select --install
set up a clean Python environment (i've been using conda and python 3.6)
install numpy and cython, which are required to build the C++ extension
python setup.py develop

For multi-threaded installation, see comment below (and issue #104).

2. Restores contraction hierarchy status messages from Pandana 0.3

Pandana used to print status messages when you built a network:

Generating contraction hierarchies with 1 threads.
Setting CH node vector of size 1498
Setting CH edge vector of size 1702
Range graph removed 1900 edges of 3404
. 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%

These were removed in Pandana 0.4: https://github.com/UDST/pandana/commit/989efcc8e6c4206b02bfbd986eac8075a08ed75c#diff-b5456dd6223748303b0929471b1231ed https://github.com/UDST/pandana/commit/6544efbbfc47bd81919ef4b7254ac035107d10fe#diff-47d7b75a6b197df3c859e0497e0b3e2c

But i think they're useful, so this PR adds them back. This restores the thread count diagnostic message requested in issue #104.

3. Resolves some deprecation warnings

Per issue #103, this PR updates network.py to use .values instead of .as_matrix().

To do before merging:

[x] add more environments to the Travis build, or manually test installation in mac, win, py27
[x] get reviews from folks more familiar with the C++ code

smmaurer commented 5 years ago

Draft instructions for multi-threaded installation from source in OS X:

Recommended procedure (installs compilers using Conda)

Confirm that you have OS X command line tools installed xcode-select --install
Install the optional "macOS_SDK_headers" package open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

(Not sure if this is necessary pre-10.14, and if so the file name would be different. Let me know if you encounter this.)
Clone the Pandana repo and check out this branch (mac-compilation)
Set up a clean Python environment conda create --name pandana-testing python=3.6 numpy cython source activate pandana-testing

Numpy and Cython are needed to compile Pandana.
Install compiler tools from Conda conda install llvm-openmp clang
Set some temporary environment variables export CC=clang export USEOPENMP=1

This lets setup.py know to use Clang instead of the built-in OS X compiler, and to include the right flags for OpenMP multi-threading.
Install Pandana python setup.py develop

If you get a compilation error like "'wchar.h' file not found", it's because the headers from step 2 are missing.

Run the Pandana test script python examples/simple_example.py

It should produce output like this:

Generating contraction hierarchies with 8 threads.
Setting CH node vector of size 1498
Setting CH edge vector of size 1702
Range graph removed 1900 edges of 3404
. 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%

Alternate procedure (for people not using Conda)

Replace steps 5 and 6 above with the following:

Install Homebrew, then brew install llvm
Set environment variables export CC=/usr/local/opt/llvm/bin/clang export USEOPENMP=1

Discussion

Earlier multi-threaded installation instructions suggested getting GCC, but Clang seems to be more standard on OS X. The default compiler is Clang (but an earlier version that doesn't support OpenMP), and Conda Forge uses Clang to support multi-threaded builds for Mac. The compilation flags have diverged, so Mac users who install GCC would need to change the extra arguments in setup.py.

It looks like we'll be able to create Conda builds that are automatically multi-threaded now, which is great -- see discussion here.

smmaurer commented 5 years ago

@federicofernandez Done. Thanks for taking a look at this!

coveralls commented 5 years ago

Coverage remained the same at 91.667% when pulling 00b5d0ddd5c5e5221da4542c3d92f4dadc0b60b6 on mac-compilation into 381fef8391074148494b2db0791ab71496bee44e on develop.

knaaptime commented 5 years ago

I'm sure this is something to raise with the conda folks specifically but thought i would share here too...

Following the above instructions works great in my base environment and I get multithreaded pandana working. But if i do the same in a different environment, I get

(general) knaaptime@daedalus ~udst/pandana mac-compilation $ python examples/simple_example.py
Traceback (most recent call last):
  File "examples/simple_example.py", line 23, in <module>
    import pandana.network as pdna
  File "/Users/knaaptime/udst/pandana/pandana/__init__.py", line 1, in <module>
    from .network import Network
  File "/Users/knaaptime/udst/pandana/pandana/network.py", line 11, in <module>
    from .cyaccess import cyaccess
ImportError: dlopen(/Users/knaaptime/udst/pandana/pandana/cyaccess.cpython-37m-darwin.so, 2): Symbol not found: ___kmpc_barrier
  Referenced from: /Users/knaaptime/udst/pandana/pandana/cyaccess.cpython-37m-darwin.so
  Expected in: flat namespace
 in /Users/knaaptime/udst/pandana/pandana/cyaccess.cpython-37m-darwin.so

I'm guessing this is a linker issue with openmp, because I can install the single-threaded version fine. Again, this seems like an issue for the conda folks, since technically all the pandana pieces are working and what seems to be missing is a properly set DYLD_LIBRARY_PATH or something, but I figured 1) you might have some idea how to resolve, and 2) this might help leave a trail in case others face this issue. I havent tested whether this happens on platforms other than osx

Pardon my complete lack of c/++ knowledge

knaaptime commented 5 years ago

i do not have this problem on my linux machine

smmaurer commented 5 years ago

@knaaptime Thanks for sharing this. I'm not sure what's going on here, but glad to hear that the procedure is working in at least some cases.

I agree that this seems related to openmp, and from Stack Overflow i get the sense that it might be caused by python trying to mix-and-match code that was compiled in different ways. Apologies for not having a more specific suggestion here, but it might be worth completely removing pandana from the environment and installing it again?

I've been using Conda environments for testing this on my Mac, so it definitely works in principle..

knaaptime commented 5 years ago

yup. works in a clean environment, so disregard. thanks for the quick response!

UDST / pandana