search

KxSystems / pyq

PyQ — Python for kdb+
http://code.kx.com/q/interfaces
Apache License 2.0
191 stars 49 forks source link

Python 3.10 compatiblity #145

Open sashkab opened 2 years ago

sashkab commented 2 years ago

Questions

Steps to reproduce the issue

$ brew install python@3.10
$  /usr/local/opt/python@3.10/bin/python3 -mvenv .virtualenvs/310
$ source .virtualenvs/310/bin/activate
$ python3.10 -mpip install -U pip setuptools wheel
$ unzip ~/Downloads/m64.zip -d "${VIRTUAL_ENV}/q"
$ (unset PIP_NO_INDEX; python3.10 -mpip install pyq)
...
Successfully built pyq
Installing collected packages: pyq
Successfully installed pyq-5.0.0
$ pyq --versions
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py", line 4, in <module>
    from collections import Mapping as _Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)
'2021.10.06T12:38:40.403 python

Expected result

pyq --versions will output versions installed on system

Actual result

$ pyq --versions
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py", line 4, in <module>
    from collections import Mapping as _Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)
'2021.10.06T12:38:40.403 python

Workaround

$ cp -pv /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py{,.orig}
/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py -> /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py.orig
$ vi /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py
$ diff /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py{.orig,}
4c4,7
< from collections import Mapping as _Mapping
---
> try:
>     from collections import Mapping as _Mapping
> except ImportError:
>     from collections.abc import Mapping as _Mapping
$ pyq --versions
PyQ 5.0.0
KDB+ 3.5 (2019.01.31) m64
Python 3.10.0 (default, Oct  6 2021, 01:11:32) [Clang 13.0.0 (clang-1300.0.29.3)]
sashkab commented 2 years ago

After first problem fixed, there is second problem. 

$ python3.10 -mpip list
Package    Version
---------- -------
attrs      21.2.0
coverage   6.0
iniconfig  1.1.1
numpy      1.21.2
packaging  21.0
pip        21.2.4
pluggy     1.0.0
py         1.10.0
pyparsing  2.4.7
pyq        5.0.0
pytest     6.2.5
pytest-cov 3.0.0
pytest-pyq 1.2.0
setuptools 58.2.0
toml       0.10.2
tomli      1.2.1
wheel      0.37.0
$ pyq -mpip list
/Users/user/.virtualenvs/310/bin/pyq: No module named pip

I believe we saw this kind of issues before, but can't recall if it was addressed. It was related to the virutalenv and venv modules, how they are now implemented... which breaks logic PyQ relies on.

sashkab commented 2 years ago

Interestingly:

$ QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q
...
q)p)import numpy
q)p)import pytest
q)p)import pip
q)p)print(pip.__file__)
/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pip/__init__.py
q)p)print(sys.executable)
/Users/user/.virtualenvs/310/bin/pyq
q)p)print(sys.path)
['/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python310.zip', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload', '/Users/user/.virtualenvs/310/lib/python3.10/site-packages']
q)p)print(sys.prefix)
/Users/user/.virtualenvs/310
q)p)print(sys.exec_prefix)
/Users/user/.virtualenvs/310

but

$ pyq
Python 3.10.0 (default, Oct  6 2021, 01:11:32) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/Users/user/.virtualenvs/310/bin/pyq'
>>> sys.path
['', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python310.zip', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload']
>>> sys.prefix
'/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10'
>>> sys.exec_prefix
'/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10'
sashkab commented 2 years ago

NB: mapping import was fixed in the internal MR 746, which is planned to be released as 5.0.1...

MurrMack commented 1 year ago

Hi Aleks - is there a plan for 5.0.1 to be released any time soon? the default version of python on ubuntu 22.04 LTS is 3.10 so others may start running into this issue as upgrades begin to occur. Cheers!

b-nln commented 1 year ago

There are no plans to release 5.0.1, as there is no full python 3.10 support yet.

MurrMack commented 1 year ago

Is there anyway to force pyq to use a different version of python? My default version of python is 3.9 using update-alternatives however when I run pyq it tries to still use 3.10

CPUS=24 pyq                                                                                                                                                                         
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/murray/.local/lib/python3.10/site-packages/pyq/__init__.py", line 4, in <module>
    from collections import Mapping as _Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
'2022.08.15T08:58:06.931 python
  [5]  /home/murray/q/p.k:7: p)from pyq import q
                             ^
  [2]  /home/murray/q/python.q:1: \l p.k

python --version  
Python 3.9.13

python3 --version
Python 3.9.13
b-nln commented 1 year ago

Use virtualenv with python3.9 and install PyQ in there as described in the installation guide.

sashkab commented 1 year ago

After spending hours earlier this week in order to rewrite Python initialization for 3.10 and failing, I came up with a quick workaround for the importing modules installed in the virtual environment when using Python 3.10 and PyQ:

$ pyq
Python 3.10.5 (main, Jul 25 2022, 15:52:08) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import markdown
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.virtualenvs/3/lib/python3.10/site-packages/pyq/__init__.py", line 820, in __import__
    m = _imp(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'markdown'

Unfortunately, due to Python Initialization changes in 3.10, sys.path is not properly initialized when Python is loaded from kdb+. Dirty workaround is to extend sys.path to what it should have been:

>>> import sys
>>> sys.path.extend(['/home/user/.virtualenvs/3/lib/python3.10/site-packages', '/usr/lib/python3.10/site-packages'])
>>> import markdown
>>> markdown.__version__
'3.3.4'

Basically, first path is your $VIRTUAL_ENV/lib/python3.10/site-packages, and second path is where system python 3 packages are installed.