search

KxSystems / pyq

PyQ — Python for kdb+
http://code.kx.com/q/interfaces
Apache License 2.0
190 stars 49 forks source link

Error importing SSL in PyQ #76

Closed jljs closed 6 years ago

jljs commented 6 years ago

Hi,

I am getting a ssl import error when importing _ssl using PyQ. Please see the error below.

import _ssl

ImportError: /lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so: undefined symbol: SSL_CTX_set_alpn_protos

I don't have the same issue if I were to use Python executable directly to run my code while keeping all environment variable the same. Are you seeing the same issue in the past? Just wondering if any SSL library is overridden in PyQ?

Thanks

Jeffrey

abalkin commented 6 years ago

Dear Jeffrey,

Unfortunately you ignored the questionnaire in the issue template and told us nothing about your system.

From the little information that you gave, it looks like you've installed PyQ in a system location without using a virtual environment. While such configuration may work, it usually requires substantial expertise to get right. For this reason we strongly recommend using a virtual environment.

As for your question, no PyQ does not override the SSL library. You can find out which library is used by PyQ as follows:

$ ldd $(pyq -c "import _ssl;print(_ssl.__file__)")| grep libssl
    libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f87048a8000)

If the same command with pyq replaced by python gives a different result - you have a misconfiguration.

Can you post the output of the following commands:

ldd $(which python)

and

cat $QHOME/pyq-config.q

where $QHOME is the location of your kdb+ installation.

jljs commented 6 years ago

Apologize for my under specification. Please see my system setup below.

Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?

Red Hat Enterprise Linux Server release 6.9 (Santiago)

Which version of PyQ are you running? You can use pyq --versions or pip list | grep pyq.

PyQ 4.1.3 Which version of kdb+ are you using, is it 32-bit or 64-bit? KDB+ 3.4 (2016.10.10) l64 If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows. QLIC=/opt/kx/3.4 Did you use virtual environment to install PyQ? If not, why?

Yes with Anaconda

Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows.

QHOME=/opt/kx/3.4 Do you use Conda? If so, what version?

Yes. conda 3.14.1

Hi,

I'm am using Conda as my virtual environment.

the output of

ldd $(which python)

is

(py36an440)[svc_tpmdb_research@pridbtsskdb02 bin]$ ldd $(which python) linux-vdso.so.1 => (0x00007fff87d6c000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003df9400000) libc.so.6 => /lib64/libc.so.6 (0x0000003df9000000) /lib64/ld-linux-x86-64.so.2 (0x0000003df8c00000)

cat $QHOME/pyq-config.q

is

\d .p python_dll:"/home/svc_tpmdb_research/.conda/envs/py36an440/bin/../lib/libpython3.6m.so.1.0\000" pyq_executable:"/home/svc_tpmdb_research/.conda/envs/py36an440/bin/pyq"

$QHOME location

/opt/kx/3.4

jljs commented 6 years ago

More information as I dig into this more. I found some difference when running the following two commands.

(py36an440)[svc_tpmdb_research@pridbtsskdb02 bin]$ ldd $(which ~/.conda/envs/py36an440/bin/python3.6_org) linux-vdso.so.1 => (0x00007ffd251e2000) libpython3.6m.so.1.0 => /home/svc_tpmdb_research/.conda/envs/py36an440/bin/../lib/libpython3.6m.so.1.0 (0x00007fa5978b2000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003df9400000) libdl.so.2 => /lib64/libdl.so.2 (0x0000003df9800000) libutil.so.1 => /lib64/libutil.so.1 (0x0000003dfac00000) librt.so.1 => /lib64/librt.so.1 (0x0000003dfa000000) libm.so.6 => /lib64/libm.so.6 (0x0000003dfa400000) libc.so.6 => /lib64/libc.so.6 (0x0000003df9000000) /lib64/ld-linux-x86-64.so.2 (0x0000003df8c00000) (py36an440)[svc_tpmdb_research@pridbtsskdb02 bin]$ ldd $(which pyq) linux-vdso.so.1 => (0x00007ffee79c9000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003df9400000) libc.so.6 => /lib64/libc.so.6 (0x0000003df9000000) /lib64/ld-linux-x86-64.so.2 (0x0000003df8c00000)

abalkin commented 6 years ago

Before we spend more time troubleshooting, since you are using conda, you may want to try our recently released conda package:

conda install -c enlnt -c kx pyq
abalkin commented 6 years ago

Did you try

ldd $(pyq -c "import _ssl;print(_ssl.__file__)")| grep libssl

and compare it with

ldd $(python -c "import _ssl;print(_ssl.__file__)")| grep libssl

?

Note that ldd $(which pyq) is rather pointless because pyq is just glorified exec q python.q.

jljs commented 6 years ago

Yes and I see the following differences.

ldd $(pyq -c "import _ssl;print(_ssl.file)")| grep libssl

linux-vdso.so.1 => (0x00007ffee79c9000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003df9400000) libc.so.6 => /lib64/libc.so.6 (0x0000003df9000000) /lib64/ld-linux-x86-64.so.2 (0x0000003df8c00000)

and compare it with

ldd $(python -c "import _ssl;print(_ssl.file)")| grep libssl linux-vdso.so.1 => (0x00007ffd251e2000) libpython3.6m.so.1.0 => /home/svc_tpmdb_research/.conda/envs/py36an440/bin/../lib/libpython3.6m.so.1.0 (0x00007fa5978b2000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003df9400000) libdl.so.2 => /lib64/libdl.so.2 (0x0000003df9800000) libutil.so.1 => /lib64/libutil.so.1 (0x0000003dfac00000) librt.so.1 => /lib64/librt.so.1 (0x0000003dfa000000) libm.so.6 => /lib64/libm.so.6 (0x0000003dfa400000) libc.so.6 => /lib64/libc.so.6 (0x0000003df9000000) /lib64/ld-linux-x86-64.so.2 (0x0000003df8c00000)

Note that ldd $(which pyq) is rather pointless because pyq is just glorified exec q python.q.

abalkin commented 6 years ago

OK, now, let's see the paths to _ssl:

pyq -c "import _ssl;print(_ssl.__file__)"

and

python -c "import _ssl;print(_ssl.__file__)"

but I already suspect that the problem is with your QHOME location. We recommend setting QHOME to $CONDA_PREFIX/q.

You should still be able to fix your setup by setting PYTHONHOME and/or PYTHONPATH.

jljs commented 6 years ago

Not able to get the path to _ssl for pyq due to import error. But the path to _ssl for python is below

/home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so

jljs commented 6 years ago

What is your $CONDA_PREFIX set to? I don't have this variable defined in my env.

abalkin commented 6 years ago

If you don't have CONDA_PREFIX set - you don't have an activated conda env. Did you run conda activate command? Please consult conda documentation.

abalkin commented 6 years ago

Please run

PYTHONVERBOSE=2 pyq -c "import _ssl" 2>&1 | grep ssl
jljs commented 6 years ago

got the following when running the above command

trying /home/svc_tpmdb_research/.conda/envs/py36an440/bin/_ssl.cpython-36m-x86_64-linux-gnu.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/bin/_ssl.abi3.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/bin/_ssl.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/bin/_ssl.py trying /home/svc_tpmdb_research/.conda/envs/py36an440/bin/_ssl.pyc trying /tpm/research/qmf-tss/servers/qmf-tpm/etl/_ssl.cpython-36m-x86_64-linux-gnu.so trying /tpm/research/qmf-tss/servers/qmf-tpm/etl/_ssl.abi3.so trying /tpm/research/qmf-tss/servers/qmf-tpm/etl/_ssl.so trying /tpm/research/qmf-tss/servers/qmf-tpm/etl/_ssl.py trying /tpm/research/qmf-tss/servers/qmf-tpm/etl/_ssl.pyc trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/_ssl.cpython-36m-x86_64-linux-gnu.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/_ssl.abi3.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/_ssl.so trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/_ssl.py trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/_ssl.pyc trying /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so ImportError: /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so: undefined symbol: SSL_CTX_set_alpn_protos clear[2] openssl_md5 clear[2] openssl_sha1 clear[2] openssl_sha224 clear[2] openssl_sha256 clear[2] openssl_sha384 clear[2] openssl_sha512 clear[2] openssl_md_meth_names

abalkin commented 6 years ago

That's interesting. So both pyq and python load the same _ssl module, but get different libssl's. We have too look deeper. Do you haves strace installed? Try

strace pyq -c 'import _ssl' 2>&1 | grep libssl

and the same with python.

jljs commented 6 years ago

Thank you for your help. I think we are slowly seeing something. Please see the trace results for both pyq and python

strace python -c 'Import _ssl' 2>&1 | grep libssl execve("/home/svc_tpmdb_research/.conda/envs/py36an440/bin/python3.6_org", ["/home/svc_tpmdb_research/.conda/"..., "-c", "Import _ssl"], [/ 86 vars /]) = 0 Import _ssl

strace pyq -c 'Import _ssl' 2>&1 | grep libssl execve("/home/svc_tpmdb_research/.conda/envs/py36an440/bin/python3.6", ["/home/svc_tpmdb_research/.conda/"..., "-c", "Import _ssl"], [/ 86 vars /]) = 0 execve("/opt/kx/3.4/l64/q", ["/home/svc_tpmdb_research/.conda/"..., "python.q", "-c@", "Import _ssl", "-q"], [/ 87 vars /]) = 0 open("/opt/common/external/anaconda/2.3.0/lib/libssl.so", O_RDONLY) = 3 Import _ssl

jljs commented 6 years ago

Why does pyq get libssl.so from root conda "/opt/common/external/anaconda/2.3.0" instead of the activate conda "/home/svc_tpmdb_research/.conda/envs/py36an440"?

Is pyq using a specific env variable for those conda prefix? Conda keeps changing their prefix env. The CONDA_PREFIX is something newly added. Our conda version is older, so I think this is why I don't have this env variable set.

abalkin commented 6 years ago

It is strange that you see anything because "Import" should start with a lower-case "I". (It's my fault - I made this mistake in my comment.)

abalkin commented 6 years ago

Did you try installing our conda package as I suggested above? Even if you decide to proceed with your own build, it will be good to know if a standard install works.

jljs commented 6 years ago

okay. I just did that and get the following error.

pyq: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by pyq)

abalkin commented 6 years ago

Thanks for trying. It looks like you have a rather old system. What is the output of

ldd --version

?

We should probably recompile our conda packages to lower the required glibc version.

jljs commented 6 years ago

Thank you for your help.

my version. ldd (GNU libc) 2.12

jljs commented 6 years ago

in the meantime, how do I revert back to my working pyq before?

abalkin commented 6 years ago

how do I revert back to my working pyq before?

conda uninstall kdb  # should uninstall both q and pyq
pip install pyq
sashkab commented 6 years ago

@jljs Thanks for the info. Please try again to install PyQ from conda.

conda uninstall pyq
conda install -c enlnt -c kx pyq

This will installl PyQ and kdb+ 3.6 for you.

jljs commented 6 years ago

Thank you for your help. I'm getting a new error when running pyq now.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/site-packages/pyq/__init__.py", line 22, in <module>
    from ._k import K as _K, error as kerr, Q_VERSION, Q_DATE, Q_OS
ImportError: /home/svc_tpmdb_research/.conda/envs/py36an440/lib/python3.6/site-packages/pyq/_k.cpython-36m-x86_64-linux-gnu.so: undefined symbol: ee
'2018.07.06T15:23:28.223 python
@
"k"
"p)from pyq import q"
jljs commented 6 years ago

Sorry, could you please explain why I need to get ondemand license? I'll get our kdb admin to do it shortly.

sashkab commented 6 years ago

@jljs it looks like you have old kdb+ installation. Please check when you run q you get a banner saying that you have kdb+ 3.6.

abalkin commented 6 years ago

why I need to get ondemand license?

You need kdb+ version >= 3.5 in order to use pyq from conda. If you follow our instructions, you should get kdb on demand v. 3.6, but you should only use it on your personal computer. Since you have a kdb admin, you should probably ask them to handle pyq installation.

jljs commented 6 years ago

I have resolved my issue above by setting LD_LIBRARY_PATH to include conda's virtual env path. I think the current version of pyq may not have correctly setup to use the conda environment properly. Hopefully the new version mentioned above can fix this. I can not verify this right now, as I can only use kdb verserion 3.4.

Thank you for your help