Closed jljs closed 6 years ago
Dear Jeffrey,
Unfortunately you ignored the questionnaire in the issue template and told us nothing about your system.
From the little information that you gave, it looks like you've installed PyQ in a system location without using a virtual environment. While such configuration may work, it usually requires substantial expertise to get right. For this reason we strongly recommend using a virtual environment.
As for your question, no PyQ does not override the SSL library. You can find out which library is used by PyQ as follows:
$ ldd $(pyq -c "import _ssl;print(_ssl.__file__)")| grep libssl
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f87048a8000)
If the same command with pyq replaced by python gives a different result - you have a misconfiguration.
Can you post the output of the following commands:
ldd $(which python)
and
cat $QHOME/pyq-config.q
where $QHOME
is the location of your kdb+ installation.
Apologize for my under specification. Please see my system setup below.
Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?
Red Hat Enterprise Linux Server release 6.9 (Santiago)
Which version of PyQ are you running? You can use pyq --versions or pip list | grep pyq.
PyQ 4.1.3 Which version of kdb+ are you using, is it 32-bit or 64-bit? KDB+ 3.4 (2016.10.10) l64 If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows. QLIC=/opt/kx/3.4 Did you use virtual environment to install PyQ? If not, why?
Yes with Anaconda
Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows.
QHOME=/opt/kx/3.4 Do you use Conda? If so, what version?
Yes. conda 3.14.1
Hi,
I'm am using Conda as my virtual environment.
the output of
ldd $(which python)
is
(py36an440)[svc_tpmdb_research@pridbtsskdb02 bin]$ ldd $(which python) linux-vdso.so.1 => (0x00007fff87d6c000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003df9400000) libc.so.6 => /lib64/libc.so.6 (0x0000003df9000000) /lib64/ld-linux-x86-64.so.2 (0x0000003df8c00000)
cat $QHOME/pyq-config.q
is
\d .p python_dll:"/home/svc_tpmdb_research/.conda/envs/py36an440/bin/../lib/libpython3.6m.so.1.0\000" pyq_executable:"/home/svc_tpmdb_research/.conda/envs/py36an440/bin/pyq"
$QHOME location
/opt/kx/3.4
More information as I dig into this more. I found some difference when running the following two commands.
(py36an440)[svc_tpmdb_research@pridbtsskdb02 bin]$ ldd $(which ~/.conda/envs/py36an440/bin/python3.6_org) linux-vdso.so.1 => (0x00007ffd251e2000) libpython3.6m.so.1.0 => /home/svc_tpmdb_research/.conda/envs/py36an440/bin/../lib/libpython3.6m.so.1.0 (0x00007fa5978b2000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003df9400000) libdl.so.2 => /lib64/libdl.so.2 (0x0000003df9800000) libutil.so.1 => /lib64/libutil.so.1 (0x0000003dfac00000) librt.so.1 => /lib64/librt.so.1 (0x0000003dfa000000) libm.so.6 => /lib64/libm.so.6 (0x0000003dfa400000) libc.so.6 => /lib64/libc.so.6 (0x0000003df9000000) /lib64/ld-linux-x86-64.so.2 (0x0000003df8c00000) (py36an440)[svc_tpmdb_research@pridbtsskdb02 bin]$ ldd $(which pyq) linux-vdso.so.1 => (0x00007ffee79c9000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003df9400000) libc.so.6 => /lib64/libc.so.6 (0x0000003df9000000) /lib64/ld-linux-x86-64.so.2 (0x0000003df8c00000)
Before we spend more time troubleshooting, since you are using conda, you may want to try our recently released conda package:
conda install -c enlnt -c kx pyq
Did you try
ldd $(pyq -c "import _ssl;print(_ssl.__file__)")| grep libssl
and compare it with
ldd $(python -c "import _ssl;print(_ssl.__file__)")| grep libssl
?
Note that ldd $(which pyq)
is rather pointless because pyq is just glorified exec q python.q
.
Yes and I see the following differences.
ldd $(pyq -c "import _ssl;print(_ssl.file)")| grep libssl
linux-vdso.so.1 => (0x00007ffee79c9000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003df9400000) libc.so.6 => /lib64/libc.so.6 (0x0000003df9000000) /lib64/ld-linux-x86-64.so.2 (0x0000003df8c00000)
and compare it with
ldd $(python -c "import _ssl;print(_ssl.file)")| grep libssl linux-vdso.so.1 => (0x00007ffd251e2000) libpython3.6m.so.1.0 => /home/svc_tpmdb_research/.conda/envs/py36an440/bin/../lib/libpython3.6m.so.1.0 (0x00007fa5978b2000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003df9400000) libdl.so.2 => /lib64/libdl.so.2 (0x0000003df9800000) libutil.so.1 => /lib64/libutil.so.1 (0x0000003dfac00000) librt.so.1 => /lib64/librt.so.1 (0x0000003dfa000000) libm.so.6 => /lib64/libm.so.6 (0x0000003dfa400000) libc.so.6 => /lib64/libc.so.6 (0x0000003df9000000) /lib64/ld-linux-x86-64.so.2 (0x0000003df8c00000)
Note that ldd $(which pyq) is rather pointless because pyq is just glorified exec q python.q.
OK, now, let's see the paths to _ssl:
pyq -c "import _ssl;print(_ssl.__file__)"
and
python -c "import _ssl;print(_ssl.__file__)"
but I already suspect that the problem is with your QHOME location. We recommend setting QHOME to $CONDA_PREFIX/q.
You should still be able to fix your setup by setting PYTHONHOME and/or PYTHONPATH.
Not able to get the path to _ssl for pyq due to import error. But the path to _ssl for python is below
/home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so
What is your $CONDA_PREFIX set to? I don't have this variable defined in my env.
If you don't have CONDA_PREFIX set - you don't have an activated conda env. Did you run conda activate command? Please consult conda documentation.
Please run
PYTHONVERBOSE=2 pyq -c "import _ssl" 2>&1 | grep ssl
got the following when running the above command
trying /home/svc_tpmdb_research/.conda/envs/py36an440/bin/_ssl.cpython-36m-x86_64-linux-gnu.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/bin/_ssl.abi3.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/bin/_ssl.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/bin/_ssl.py trying /home/svc_tpmdb_research/.conda/envs/py36an440/bin/_ssl.pyc trying /tpm/research/qmf-tss/servers/qmf-tpm/etl/_ssl.cpython-36m-x86_64-linux-gnu.so trying /tpm/research/qmf-tss/servers/qmf-tpm/etl/_ssl.abi3.so trying /tpm/research/qmf-tss/servers/qmf-tpm/etl/_ssl.so trying /tpm/research/qmf-tss/servers/qmf-tpm/etl/_ssl.py trying /tpm/research/qmf-tss/servers/qmf-tpm/etl/_ssl.pyc trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/_ssl.cpython-36m-x86_64-linux-gnu.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/_ssl.abi3.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/_ssl.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/_ssl.py trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/_ssl.pyc trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so ImportError: /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so: undefined symbol: SSL_CTX_set_alpn_protos clear[2] openssl_md5 clear[2] openssl_sha1 clear[2] openssl_sha224 clear[2] openssl_sha256 clear[2] openssl_sha384 clear[2] openssl_sha512 clear[2] openssl_md_meth_names
That's interesting. So both pyq and python load the same _ssl module, but get different libssl's. We have too look deeper. Do you haves strace installed? Try
strace pyq -c 'import _ssl' 2>&1 | grep libssl
and the same with python.
Thank you for your help. I think we are slowly seeing something. Please see the trace results for both pyq and python
strace python -c 'Import _ssl' 2>&1 | grep libssl execve("/home/svc_tpmdb_research/.conda/envs/py36an440/bin/python3.6_org", ["/home/svc_tpmdb_research/.conda/"..., "-c", "Import _ssl"], [/ 86 vars /]) = 0 Import _ssl
strace pyq -c 'Import _ssl' 2>&1 | grep libssl execve("/home/svc_tpmdb_research/.conda/envs/py36an440/bin/python3.6", ["/home/svc_tpmdb_research/.conda/"..., "-c", "Import _ssl"], [/ 86 vars /]) = 0 execve("/opt/kx/3.4/l64/q", ["/home/svc_tpmdb_research/.conda/"..., "python.q", "-c@", "Import _ssl", "-q"], [/ 87 vars /]) = 0 open("/opt/common/external/anaconda/2.3.0/lib/libssl.so", O_RDONLY) = 3 Import _ssl
Why does pyq get libssl.so from root conda "/opt/common/external/anaconda/2.3.0" instead of the activate conda "/home/svc_tpmdb_research/.conda/envs/py36an440"?
Is pyq using a specific env variable for those conda prefix? Conda keeps changing their prefix env. The CONDA_PREFIX is something newly added. Our conda version is older, so I think this is why I don't have this env variable set.
It is strange that you see anything because "Import" should start with a lower-case "I". (It's my fault - I made this mistake in my comment.)
Did you try installing our conda package as I suggested above? Even if you decide to proceed with your own build, it will be good to know if a standard install works.
okay. I just did that and get the following error.
pyq: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by pyq)
Thanks for trying. It looks like you have a rather old system. What is the output of
ldd --version
?
We should probably recompile our conda packages to lower the required glibc version.
Thank you for your help.
my version. ldd (GNU libc) 2.12
in the meantime, how do I revert back to my working pyq before?
how do I revert back to my working pyq before?
conda uninstall kdb # should uninstall both q and pyq
pip install pyq
@jljs Thanks for the info. Please try again to install PyQ from conda.
conda uninstall pyq
conda install -c enlnt -c kx pyq
This will installl PyQ and kdb+ 3.6 for you.
Thank you for your help. I'm getting a new error when running pyq now.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/site-packages/pyq/__init__.py", line 22, in <module>
from ._k import K as _K, error as kerr, Q_VERSION, Q_DATE, Q_OS
ImportError: /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/site-packages/pyq/_k.cpython-36m-x86_64-linux-gnu.so: undefined symbol: ee
'2018.07.06T15:23:28.223 python
@
"k"
"p)from pyq import q"
Sorry, could you please explain why I need to get ondemand license? I'll get our kdb admin to do it shortly.
@jljs it looks like you have old kdb+ installation. Please check when you run q
you get a banner saying that you have kdb+ 3.6.
why I need to get ondemand license?
You need kdb+ version >= 3.5 in order to use pyq from conda. If you follow our instructions, you should get kdb on demand v. 3.6, but you should only use it on your personal computer. Since you have a kdb admin, you should probably ask them to handle pyq installation.
I have resolved my issue above by setting LD_LIBRARY_PATH to include conda's virtual env path. I think the current version of pyq may not have correctly setup to use the conda environment properly. Hopefully the new version mentioned above can fix this. I can not verify this right now, as I can only use kdb verserion 3.4.
Thank you for your help
Hi,
I am getting a ssl import error when importing _ssl using PyQ. Please see the error below.
import _ssl
ImportError:/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so: undefined symbol: SSL_CTX_set_alpn_protos
I don't have the same issue if I were to use Python executable directly to run my code while keeping all environment variable the same. Are you seeing the same issue in the past? Just wondering if any SSL library is overridden in PyQ?
Thanks
Jeffrey