search

KxSystems / pyq

PyQ — Python for kdb+
http://code.kx.com/q/interfaces
Apache License 2.0
190 stars 49 forks source link

pyq installation looks for 32 bit executable in 64 bit architecture #78

Closed karthikmurali closed 6 years ago

karthikmurali commented 6 years ago

Questions

Steps to reproduce the issue

We are trying out pyq and running into installation errors – I am trying to point it to the 64bit commercial version, but it always installs the pyq components for the 32bit and then cannot find the 32bit q executable. 

$TARGET =  $VIRTUAL_ENV = /home/user/pyq3

(pyq3) $ pyq
qbinpath = /home/user/pyq3/q/l32/q
qbinpath = /home/user/pyq3/q/l32/q
qbinpath = /home/user/pyq3/q/l32/q
qbinpath = /home/user/q/l32/q

but this works just fine

(pyq3) $ QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/l64/q
KDB+ 3.5 2018.04.25 Copyright (C) 1993-2018 Kx Systems
l64/ 4()core 32104MB 

q)

this is the snippet of code used to create the virtualenv

pip install  --target=/tmp/pyqenvtmp virtualenv

python3 /tmp/pyqenvtmp/virtualenv.py $TARGET

rm -rf /tmp/pyqenvtmp

source $TARGET/bin/activate
mkdir $VIRTUAL_ENV/q
cp -r $KDB_DIR/* $VIRTUAL_ENV/q //$KDB_DIR is where 64bit q is stored
echo "export QHOME=$VIRTUAL_ENV/q" >> $VIRTUAL_ENV/bin/activate
echo "export QLIC=$VIRTUAL_ENV/q/lic/licfile" >> $VIRTUAL_ENV/bin/activate
pip install  pyq

I also tried running get_q_arch( os.environ[‘QHOME’]) from setup.py within the virtual environment and it gives me l64

Expected result

pyq should run with 64bit q

Actual result

pyq looks for 32bit q

Workaround

If you know workaround, please provide it here.

abalkin commented 6 years ago

Please read installing into a virtual environment instructions carefully. You need to install kdb+ in $VIRTUAL_ENV/q and your QHOME should be either unset or set to $VIRTUAL_ENV/q.

If it still does not work, please run pip install in verbose mode

pip install -vvv  pyq

and post the results.

karthikmurali commented 6 years ago

I installed again - $QHOME is in $VIRTUAL_ENV/q

Yes - q is in $VIRTUAL_ENV/q (pyq4) $ echo $QHOME /home/user/pyq4/q (pyq4) $ echo $VIRTUAL_ENV /home/user/pyq

Output of pip install -i $INTERNAL_PYPI_URL --trusted_host $INTERNAL_PYPI_HOST -vvv pyq

Collecting pyqââââââââââââ 1 location(s) to search for versions of pyq:

Successfully installed pyq-4.1.4

abalkin commented 6 years ago

Did you resolve your issue?

karthikmurali commented 6 years ago

No , the same issue still exists. It installed successfully, but fails looking for 32bit q instead of 64bit.

abalkin commented 6 years ago

You really have to tell us more than that. What command do you run? What output do you expect and what output do you get? One problem I can see in your log is this:

Using cached wheel link: file:///home/user/.cache/pip/wheels/.../pyq-4.1.4-cp36-cp36m-linux_x86_64.whl

You have a .whl file left from a command that you don't show us.

Also, we have no way to know what $INTERNAL_PYPI_URL or $INTERNAL_PYPI_HOST variables are.

If you prefer following someone else's instructions instead of those at code.kx.com, you should ask those who instructed you for help.

karthikmurali commented 6 years ago

Sorry, I thought I gave you all this information in the first comment - the exact commands I run are below. I have tried to stay pretty close to what is at code.kx.com. The main differences are , a) public pypi is blocked, so we install from a pypi within the corporate network b) there is already a directory with the kdb installation and we copy q from there into the virtual env. $INTERNAL_PYPI_HOST and $INTERNAL_PYPI_URL are just pointers to the internal pypi installation which is just a copy of public pypi.

What I expect: When I run pyq - it starts up pointing to l64/q

What actually happens: I cannot start pyq, it looks for l32/q which is not there and fails $ source /home/user/pyq4/bin/activate (pyq4) $ pyq qbinpath = /home/user/pyq4/q/l32/q qbinpath = /home/user/pyq4/q/l32/q qbinpath = /home/user/pyq4/q/l32/q qbinpath = /home/user/q/l32/q

Exact commands I run

pip install -i $INTERNAL_PYPI_URL --trusted_host $INTERNAL_PYPI_HOST -target=/tmp/pyqenvtmp virtualenv

python3 /tmp/pyqenvtmp/virtualenv.py /home/user/pyq4

rm -rf /tmp/pyqenvtmp

source $/home/user/pyq4/bin/activate mkdir $VIRTUAL_ENV/q cp -r $KDB_DIR/* $VIRTUAL_ENV/q //$KDB_DIR is where 64bit q is stored echo "export QHOME=$VIRTUAL_ENV/q" >> $VIRTUAL_ENV/bin/activate echo "export QLIC=$VIRTUAL_ENV/q/lic/licfile" >> $VIRTUAL_ENV/bin/activate pip install -i $INTERNAL_PYPI_URL --trusted_host $INTERNAL_PYPI_HOST pyq

abalkin commented 6 years ago

OK, we are getting somewhere, but you still did not provide a clean build log. Please run

pip install -vvv --no-binary pyq pyq

You don't have to post the entire story of finding the sources on pypi. Try to find the relevant lines.

You should be looking for a line with -DQARCH in it. If it shows l32 it means you are building pyq with a 32-bit python. Check what your pip command uses:

head -1 $(which pip)
karthikmurali commented 6 years ago

I was just able to fix this by running pip with --no-cache-dir. Thank you for the pointer about the cached wheel file. I think a previous attempt at installation had left some files around which was being picked up by this.

Thanks a lot for your help!