xiangpeng2008 commented 4 years ago

Questions

[ ] Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?

macOS Catalina 10.15.4
[ ] Which version of PyQ are you running? Please provide output of pyq --versions, if PyQ isn't operational, please provide Python interpreter version and PyQ version python -V; python3 -V; pip list | grep pyq:

4.2.1
[ ] Which version of kdb+ are you using, is it 32-bit or 64-bit?

3.6 64-bit
- [ ] If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows.

``No, but my license file is under$VIRTUAL_EVN/q````

[ ] Did you use virtual environment to install PyQ? If not, why?
- yes
[ ] Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows. QHOME=/Users/xiangpeng/venv37/q
[ ] Do you use Conda? If so, what version? No

Steps to reproduce the issue

source venv37/bin/activate export QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q q)p)import numpy as np

Expected result

Actual result

KDB+ 3.6 2018.06.14 Copyright (C) 1993-2018 Kx Systems
m64/ 4(16)core 24576MB xiangpeng imac 192.168.1.11 EXPIRE 2021.02.22 zhangxiangpeng2008@gmail.com KOD #4169312

q)p)import numpy as np
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyq/__init__.py", line 841, in __import__
    m = _imp(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'numpy'
'python
q)\\

Workaround

I just updated my python3.7 and re-created virtual env following: https://github.com/KxSystems/pyq/issues/131#issuecomment-627361984 It looks like python tries to look for package from system instead of inside virtual env. Meanwhile, pyq has no problem to import package from virtual env.

If you know workaround, please provide it here.

sashkab commented 4 years ago

Unable to reproduce.

$ QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q
KDB+ 3.5 ...
m64/ ...
q)p)import numpy
q)p)print(numpy.zeros(5))
[0. 0. 0. 0. 0.]

sashkab commented 4 years ago

Please provide output of:

pyq -mpip list | grep -E 'pyq|numpy'
python -mpip list | grep -E 'pyq|numpy'
python3.7 -mpip list | grep -E 'pyq|numpy'

xiangpeng2008 commented 4 years ago

@sashkab Thanks a lot for your reply ! here's the output

(venv37) iMac /Users/xiangpeng $ pyq -mpip list | grep -E 'pyq|numpy'
numpy           1.18.4
pyq             4.2.1
(venv37) iMac /Users/xiangpeng $ python -mpip list | grep -E 'pyq|numpy'
numpy           1.18.4
pyq             4.2.1
(venv37) iMac /Users/xiangpeng $ python3.7 -mpip list | grep -E 'pyq|numpy'
numpy           1.18.4
pyq             4.2.1

xiangpeng2008 commented 4 years ago

Does numpy package exist in your /usr/local/Cellar/python/...... as well ? What if you try with some package only existing in your virtual env ?

sashkab commented 4 years ago

All packages are installed for me in virtualenv, and nothing is installed to the homebrew site-packages. But your PyQ is installed into main homebrew site-packages -- sorry, missed it on first look.

Please do following:

deactivate
python3.7 -mpip uninstall pyq

Then re-install PyQ into virtualenv.

source venv37/bin/activate
python3.7 -mpip install pyq

xiangpeng2008 commented 4 years ago

@sashkab thanks a lot for your advice. Indeed, I really have PyQ installed in main homebrew site-packages. However after doing

deactivate
python3.7 -mpip uninstall pyq

q)p)print(1)

stops working, even after I redo what you asked https://github.com/KxSystems/pyq/issues/131#issuecomment-627361984

iMac /Users/xiangpeng $ rm -fr pyq
iMac /Users/xiangpeng $ brew upgrade
iMac /Users/xiangpeng $ python3.7 -mvenv pyq
iMac /Users/xiangpeng $ source pyq/bin/activate
(pyq) iMac /Users/xiangpeng $ cp -r q64 $VIRTUAL_ENV/q
(pyq) iMac /Users/xiangpeng $ python3.7 -mpip install pyq
Collecting pyq
  Using cached https://files.pythonhosted.org/packages/55/ef/c0ce6cfd39dc599cba09e1d889dc750fea83d2ab47afca328df46f3441cf/pyq-4.2.1.tar.gz
Installing collected packages: pyq
  Running setup.py install for pyq ... done
Successfully installed pyq-4.2.1
WARNING: You are using pip version 19.2.3, however version 20.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(pyq) iMac /Users/xiangpeng $ pyq --versions
PyQ 4.2.1
KDB+ 3.6 (2018.06.14) m64
Python 3.7.7 (default, May 12 2020, 22:03:45)
[Clang 11.0.0 (clang-1100.0.33.17)]
(pyq) iMac /Users/xiangpeng $ QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q
KDB+ 3.6 2018.06.14 Copyright (C) 1993-2018 Kx Systems
m64/ 4(16)core 24576MB xiangpeng imac 192.168.1.11 EXPIRE 2021.02.22 zhangxiangpeng2008@gmail.com KOD #4169312

q)p)print(1)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pyq'
'python
  [0]  (<load>)

  )

sashkab commented 4 years ago

You didn't follow my instructions 100%. You used cp q64 ..., but I suggested to use clean macos.zip to extract. I don't know what's inside your q64 directory.

Also, after creating virtual env and prior to installing PyQ, please upgrade pip and other pypa tools to the latest versions:

python3.7 -mpip install -U pip setuptools wheel

Thanks.

xiangpeng2008 commented 4 years ago

@sashkab I tried several more times just as you told me, it's really really wired I believe you wanted say setuptools instead of setup and tools ?

rm -fr pyq
brew upgrade
python3.7 -mvenv pyq
source pyq/bin/activate
mkdir $VIRTUAL_ENV/q
unzip Downloads/m64.zip -d $VIRTUAL_ENV/q
python3.7 -mpip install -U pip setuptools wheel
cp q64/kc.lic $VIRTUAL_ENV/q/
python3.7 -mpip install pyq
QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q
QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q
p)1

Restored session: Wed May 13 21:18:31 +08 2020
iMac /Users/xiangpeng $ rm -fr pyq
iMac /Users/xiangpeng $ brew upgrade
iMac /Users/xiangpeng $ python3.7 -mvenv pyq
iMac /Users/xiangpeng $ source pyq/bin/activate
(pyq) iMac /Users/xiangpeng $ mkdir $VIRTUAL_ENV/q
(pyq) iMac /Users/xiangpeng $ unzip Downloads/m64.zip -d $VIRTUAL_ENV/q
Archive:  Downloads/m64.zip
  inflating: /Users/xiangpeng/pyq/q/m64/q
  inflating: /Users/xiangpeng/pyq/q/q.k
(pyq) iMac /Users/xiangpeng $ python3.7 -mpip install -U pip setuptools wheel
Collecting pip
  Using cached https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/73/22/156aac338c4ac8ffb1c3c6c3d4841611fdd4e40a9be7518bf5be432fd38c/setuptools-46.2.0-py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, wheel
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
  Found existing installation: setuptools 41.2.0
    Uninstalling setuptools-41.2.0:
      Successfully uninstalled setuptools-41.2.0
Successfully installed pip-20.1 setuptools-46.2.0 wheel-0.34.2
(pyq) iMac /Users/xiangpeng $ cp q64/kc.lic $VIRTUAL_ENV/q/
(pyq) iMac /Users/xiangpeng $ python3.7 -mpip install pyq
Processing ./Library/Caches/pip/wheels/eb/82/3d/7caf530ccb243dd5f55b260ddc247d87283840118e86983734/pyq-4.2.1-cp37-cp37m-macosx_10_15_x86_64.whl
Installing collected packages: pyq
Successfully installed pyq-4.2.1
(pyq) iMac /Users/xiangpeng $ QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q
Killed: 9
(pyq) iMac /Users/xiangpeng $ QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q
KDB+ 3.6 2019.09.19 Copyright (C) 1993-2019 Kx Systems
m64/ 4(16)core 24576MB xiangpeng imac 192.168.1.11 EXPIRE 2021.02.22 zhangxiangpeng2008@gmail.com KOD #4169312

q)p)1
'dlopen(/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/Python, 10): image not found
  [2]  /Users/xiangpeng/pyq/q/p.k:2: (`:./pyq 2:(`p_init;1)).pyq.python_dll
                                     ^
  [0]  (<load>)

  )

Meanwhile, I don't have python version 3.7.6_1 after brew upgrade. Is it because pyq doesn't support python3.7.7 yet ? Thanks !

xiangpeng2008 commented 4 years ago

I tried again on another mac, which gave me very similar error

MacBookPro /Users/xiangpeng $ rm -fr pyq
MacBookPro /Users/xiangpeng $ brew upgrade
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/cask).
==> Updated Casks
brave-browser

MacBookPro /Users/xiangpeng $ python3.7 -mvenv pyq
source pyq/bin/activate
mkdir $VIRTUAL_ENV/q
MacBookPro /Users/xiangpeng $ source pyq/bin/activate
(pyq) MacBookPro /Users/xiangpeng $ mkdir $VIRTUAL_ENV/q
(pyq) MacBookPro /Users/xiangpeng $ cp -r Downloads/m64-2/* $VIRTUAL_ENV/q/
(pyq) MacBookPro /Users/xiangpeng $ ls $VIRTUAL_ENV/q/
./   ../  m64/ q.k
(pyq) MacBookPro /Users/xiangpeng $ python3.7 -mpip install -U pip setuptools wheel
Collecting pip
  Using cached https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/09/27/98ef5f13ca9c63d9d8601e739817839dde55baeb105a6c32fc8f24daddc2/setuptools-46.3.0-py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, wheel
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
  Found existing installation: setuptools 41.2.0
    Uninstalling setuptools-41.2.0:
      Successfully uninstalled setuptools-41.2.0
Successfully installed pip-20.1 setuptools-46.3.0 wheel-0.34.2
(pyq) MacBookPro /Users/xiangpeng $ cp venv37/q/kc.lic $VIRTUAL_ENV/q/
(pyq) MacBookPro /Users/xiangpeng $ python3.7 -mpip install pyq
Processing ./Library/Caches/pip/wheels/ac/e1/86/336491a8401046eed5f231db7a49e1c15555d0ad10a49be74f/pyq-4.2.1-cp37-cp37m-macosx_10_14_x86_64.whl
Installing collected packages: pyq
Successfully installed pyq-4.2.1
(pyq) MacBookPro /Users/xiangpeng $ QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q
Killed: 9
(pyq) MacBookPro /Users/xiangpeng $ QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q
KDB+ 3.6 2019.09.19 Copyright (C) 1993-2019 Kx Systems
m64/ 4(16)core 8192MB xiangpeng macbookpro 192.168.1.7 EXPIRE 2021.02.22 zhangxiangpeng2008@gmail.com KOD #4169312

q)p)1
'dlopen(/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/Python, 10): image not found
  [2]  /Users/xiangpeng/pyq/q/p.k:2: (`:./pyq 2:(`p_init;1)).pyq.python_dll
                                     ^
  [0]  (<load>)

  )

The difference is it looks python3.7.4 now which is the python version before brew upgrade

sashkab commented 4 years ago

I believe you wanted say setuptools instead of setup and tools ?

Correct, fixed comment.

I will look into this further, still unable to reproduce on my system.

xiangpeng2008 commented 4 years ago

Are you with maxOS Catalina 10.15.4 too ?

sashkab commented 4 years ago

I was able to reproduce this, and opened internal issue 1068. No ETA on this.

xiangpeng2008 commented 4 years ago

Is it because of macOS version ? I can downgrade it in this case ?

sashkab commented 4 years ago

No. Only workaround I can provide to you is to use python 3.6 and older version of the virutalenv to create venv.

brew install sashkab/python/python@3.6
/usr/local/opt/python@3.6/bin/python3.6 -mpip install -U virtualenv==16.7.9
/usr/local/opt/python@3.6/bin/python3.6 -mvirtualenv pyq
source pyq/bin/activate
python3.6 -mpip install -U pip setuptools wheel
unzip Downloads/m64.zip -d $VIRTUAL_ENV/q
cp q64/kc.lic $VIRTUAL_ENV/q/
python3.6 -mpip install pyq
QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q

xiangpeng2008 commented 4 years ago

This works perfectly ! And it loads python package from virtual env correctly ! python 3.6 or 3.7 don't make any difference for me, I will stick to 3.6 until I have to upgrade. Thanks a lot ! As you have created an internal issue, I close this one.

sashkab commented 4 years ago

I'm going to keep this issue open until it is resolved for Python 3.7+.

sashkab commented 4 years ago

This has been resolved in PyQ 5.0. Please update to it using pip install -U pyq.

xiangpeng2008 commented 4 years ago

Yes, it works !

juanez commented 3 years ago

same/similar issue on linux (xubuntu 20.04) and python 3.8

can work around by installing pyq inside the virtual env AND outside ...

sashkab commented 3 years ago

@juanez please open new issue and provide detailed information. same/similar doesn't help to troubleshoot.

KxSystems / pyq

q can't find python package in virtual env #132

Questions

Steps to reproduce the issue

Expected result

Actual result

Workaround