JohnLangford
commented
7 years ago @gramhagen should this be working?
Closed acepor closed 7 years ago
JohnLangford
commented
7 years ago @gramhagen should this be working?
Scott-Graham-Bose
commented
7 years ago Using Python 3.5
/usr/bin/g++ -std=c++0x -I/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I /usr/local/include -I ../rapidjson/include -fPIC -c pylibvw.cc -o pylibvw.o
/usr/bin/g++ -shared pylibvw.o -L /usr/local/lib -L/usr/local/opt/python/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -ldl -framework CoreFoundation ../vowpalwabbit/libvw.a ../vowpalwabbit/liballreduce.a -L /usr/local/lib -lboost_program_options-mt -lboost_serialization-mt -l pthread -l z -l boost_python3-mt -o pylibvw.soIt does not seem to be using the appropriate includes/ldflags when building the python so. That points to some mixup between the setup.py and makefile passing the python version or retrieving those params.
What do you get when you run these?
python3.5-config --includes
python3.5-config --ldflags
acepor
commented
7 years ago @Scott-Graham-Bose
➜ ~ python3.5-config --includes -I/Library/Frameworks/Python.framework/Versions/3.5/include/python3.5m -I/Library/Frameworks/Python.framework/Versions/3.5/include/python3.5m ➜ ~ python3.5-config --ldflags -L/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/config-3.5m -lpython3.5m -ldl -framework CoreFoundation ➜ ~
Scott-Graham-Bose
commented
7 years ago ok, there's a weird condition in the makefile that changes the flags when installing under a virtualenv, i suspect that is what is happening.
could you try changing the Makefile in the python directory?
ifeq ($(VIRTUAL_ENV), )
PYTHON_INCLUDE = $(shell python$(PYTHON_VERSION)-config --includes)
PYTHON_LDFLAGS = $(shell python$(PYTHON_VERSION)-config --ldflags)
else
PYTHON_INCLUDE = $(shell python-config --includes)
PYTHON_LDFLAGS = $(shell python-config --ldflags)
endifto this:
PYTHON_INCLUDE = $(shell python$(PYTHON_VERSION)-config --includes)
PYTHON_LDFLAGS = $(shell python$(PYTHON_VERSION)-config --ldflags)then do the following from the python directory:
python3.5 setup.py clean
python3.5 setup.py install@Scott-Graham-Bose Thanks a lot. It worked!
Scott-Graham-Bose
commented
7 years ago glad to hear it. I'd like to remove the offending code, but i don't understand what use case it was fixing.
@jotterbach can you comment on whether this code is necessary and if there may be a more specific way to solve it that doesn't impact everyone running in a virtualenv?
https://github.com/JohnLangford/vowpal_wabbit/commit/3f273c421d93ffe84d261cd6f2d2d44159404e8b
acepor
commented
7 years ago @Scott-Graham-Bose Sorry, I may need further helps.
When I tryed importing pyvw in iPython, it showed the info:
import pyvw ImportError Traceback (most recent call last)
in () ----> 1 import pyvw /usr/local/lib/python3.6/site-packages/pyvw.py in () 1 import sys ----> 2 import pylibvw 3 4 class SearchTask(): 5 def __init__(self, vw, sch, num_actions): ImportError: dynamic module does not define module export function (PyInit_pylibvw)
Yes, there was a similar issue with the solution for Python2: How to generate pylibvw.py
Basically, we can copy the pylibvw.so to another DIR as you mentioned:
cp python/pylibvw.so
/lib/python/2.7/lib-dynload/
However, there is no such DIR in my Python3 folder.
How can this be solved? Thanks very much.
Scott-Graham-Bose
commented
7 years ago this error can be caused by a few different things: the name of the shared object not being consistent with the name of the module pylibvw.so not being on the pythonpath or the pylibvw that is found first is actually compiled for a different version of python (e.g. python2.7)
I don't think it's the first one, and most likely it is the last. can you try the following commands:
python -c 'import imp; print imp.find_module("pyvw"); print imp.find_module("pylibvw")'@Scott-Graham-Bose
Traceback (most recent call last): File "
", line 1, in ImportError: No module named pyvw
Scott-Graham-Bose
commented
7 years ago oops, i guess that would be python3 ? whichever python you used to install vw
acepor
commented
7 years ago I cloned the git repo, and compiled vw in python 2.7.13, although the default environment is Python 3.6.0. I tried to install Python 3, but always failed. Thanks again.
florinb
commented
7 years ago Indeed the problem seems to be related to the selection of different params between the setup.py and Makefile. I was able to fix the problem mentioned above on an Ubuntu 14.04.5 VM with the latest Anaconda 4.31 (the Python 3.6 version 64-bit installer, with ~anaconda3/bin added to PATH) and the latest code from master with the following changes:
if system == 'Linux':
# use version suffix if present
boost_lib = 'boost_python-py{v[0]}{v[1]}'.format(v=sys.version_info)with
if system == 'Linux':
# use version suffix if present
boost_lib = 'boost_python-py{v[0]}{v[1]}'.format(v=sys.version_info)
if sys.version_info.major == 3:
if find_library('boost_python-py36'):
boost_lib = 'boost_python-py36'
elif find_library('boost_python-py35'):
boost_lib = 'boost_python-py35'
elif find_library('boost_python-py34'):
boost_lib = 'boost_python-py34'
else:
boost_lib = 'boost_python-py{v[0]}{v[1]}'.format(v=sys.version_info)ifeq ($(VIRTUAL_ENV), )
...
endifwith
PYTHON_INCLUDE = $(shell python$(PYTHON_VERSION)-config --includes)
PYTHON_LDFLAGS = $(shell python$(PYTHON_VERSION)-config --ldflags)With the above changes in place one could then run in Terminal on Ubuntu14.04 in python directory the following:
export PYTHON_VERSION=3.6
export PYTHON_LIBS="-l boost_python-py34"
python setup.py clean
python setup.py installRunning tests/test_*.py after that returned no error.
Detecting the PYTHON_VERSION could be done in Makefile, too, but I'm not sure about PYTHON_LIBS.
Update Detect the PYTHON_VERSION and PYTHON_LIBS in Makefile (replaced lines 13 to 19, tested on Ubuntu only) without using the export... mentioned above:
ifeq ($(PYTHON_VERSION),)
PYTHON_VER_FULL = $(wordlist 2,4,$(subst ., ,$(shell python --version 2>&1)))
PYTHON_VERSION = $(word 1,$(PYTHON_VER_FULL)).$(word 2,$(PYTHON_VER_FULL))
PYTHON_VER_MAJOR = $(word 1,$(PYTHON_VER_FULL))
PYTHON_VER_BOOST = $(word 1,$(PYTHON_VER_FULL))$(word 2,$(PYTHON_VER_FULL))
endif
ifeq ($(PYTHON_LIBS),)
PYTHON_LIBS = -l boost_python
ifeq ($(PYTHON_VER_MAJOR), 3)
ifneq ($(notdir $(shell find /usr/lib -name libboost_python-py36.so* -type f 2>&1)), )
PYTHON_LIBS = -l boost_python-py36
else ifneq ($(notdir $(shell find /usr/lib -name libboost_python-py35.so* -type f 2>&1)), )
PYTHON_LIBS = -l boost_python-py35
else ifneq ($(notdir $(shell find /usr/lib -name libboost_python-py34.so* -type f 2>&1)), )
PYTHON_LIBS = -l boost_python-py34
else
PYTHON_LIBS = -l boost_python-py$(PYTHON_VER_BOOST)
endif
endif
endifthis is great, thanks @florinb, could you make a pull request with these changes?
florinb
commented
7 years ago Yes, will do!
Scott-Graham-Bose
commented
7 years ago fwiw, I was playing with anaconda and vowpalwabbit last night and was able to build using the following procedure on Mac OSX:
conda create --name vw boost
source activate vw
git clone https://github.com/JohnLangford/vowpal_wabbit.git
cd vowpal_wabbit
# edit Makefile in repo root
# change BOOST_INCLUDE to use anaconda env dir: /anaconda/envs/vw/include
# change BOOST_LIBRARY to use anaconda lib dir: /andaconda/envs/vw/lib
cd python
# edit Makefile in python dir to match what is in #1230
python setup.py installNow that I have a working process I think the next step is to create an conda forge project so vw can be installed into an anaconda environment directly, but in the meantime those using anaconda now have an option to install the python package without too much fuss
JohnLangford
commented
7 years ago Maybe add to wiki?
-John
On Mon, May 1, 2017 at 9:07 AM, Scott Graham notifications@github.com wrote:
fwiw, I was playing with anaconda and vowpalwabbit last night and was able to build using the following procedure on Mac OSX:
conda create --name vw boost source activate snowflakes git clone https://github.com/JohnLangford/vowpal_wabbit.git cd vowpal_wabbit
edit Makefile in repo root
change BOOST_INCLUDE to use anaconda env dir: /anaconda/envs/vw/include
change BOOST_LIBRARY to use anaconda lib dir: /andaconda/envs/vw/lib
cd python
edit Makefile in python dir to match what is in #1230
python setup.py install
Now that I have a working process I think the next step is to create an conda forge project so vw can be installed into an anaconda environment directly, but in the meantime those using anaconda now have an option to install the python package without too much fuss
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/JohnLangford/vowpal_wabbit/issues/1208#issuecomment-298330757, or mute the thread https://github.com/notifications/unsubscribe-auth/AAE25iLcaAF8mu8p6bvnoRzMuDPZKgoJks5r1dkUgaJpZM4MTyqP .
jotterbach
commented
7 years ago Not sure if this is still relevant comment @Scott-Graham-Bose , but I introduced the code to be able to install VW within a local virtual environment. I was using VW as a part of a larger project that I wanted to virtualize and I couldn't access it since it installed VW in the system python even when within a venv.
gramhagen
commented
7 years ago Ok. Installing with pip or setup.py under your vitualenv should now work without requiring that special handling.
On Fri, May 26, 2017, 9:27 PM Johannes Otterbach notifications@github.com wrote:
Not sure if this is still relevant comment @Scott-Graham-Bose https://github.com/scott-graham-bose , but I introduced the code to be able to install VW within a local virtual environment. I was using VW as a part of a larger project that I wanted to virtualize and I couldn't access it since it installed VW in the system python even when within a venv.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JohnLangford/vowpal_wabbit/issues/1208#issuecomment-304417470, or mute the thread https://github.com/notifications/unsubscribe-auth/AFdJ2QN43THGNZ3hMHmtQHfa99heoAfbks5r93vtgaJpZM4MTyqP .
acepor
commented
7 years ago @Scott-Graham-Bose many thanks!
juWuBabaaaa
commented
6 years ago How to change the Boost_INCLUDE and Boost_LIBRARY? very puzzled at this
Scott-Graham-Bose
commented
6 years ago those are defined in the Makefile in the main directory
Hi,
I followed the installation instruction to install vowpal_wabbit on MacOS 10.12.3 with Python 3.6.0/3.5.2, and neither succeeded. However, when I installed it with Python 2.7.13, everything was smooth.
Can anyone offer any help? Thanks you so much!