Cant Import Python Modules

[Andrew-Pynch]

Today I installed "swift-tensorflow-RELEASE-0.9-cuda10.2-cudnn7-ubuntu18.04.tar.gz" using the installation instructions on "installation.md". Unfortunately, when trying to use python libraries such as numpy with the following code from the "Python interoperability" I receive the following errors.

CODE

#if canImport(PythonKit)
    import PythonKit
#else
    import Python
#endif
print(Python.version)

let np = Python.import("numpy")
print(np)
let zeros = np.ones([2, 3])
print(zeros)

ERRORS 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0] Fatal error: 'try!' expression unexpectedly raised an error: Python exception: No module named 'numpy': file /swift-base/PythonKit/PythonKit/Python.swift, line 673 Current stack trace: 0 libswiftCore.so 0x00007f7a23474120 swift_reportError + 50 1 libswiftCore.so 0x00007f7a234e6be0 _swift_stdlib_reportFatalErrorInFile + 115 2 libswiftCore.so 0x00007f7a2318bcb5 + 1506485 3 libswiftCore.so 0x00007f7a2318b8f7 + 1505527 4 libswiftCore.so 0x00007f7a2318be92 + 1506962 5 libswiftCore.so 0x00007f7a2318a2e0 assertionFailure(:_:file:line:flags:) + 517 6 libswiftCore.so 0x00007f7a231d47b9 + 1804217 7 libPythonKit.so 0x00007f7a23fd699f + 92575 9 swift 0x000000000059ae63 + 1683043 10 swift 0x0000000000587af0 + 1604336 11 swift 0x000000000056df1a + 1498906 12 swift 0x00000000005611f5 + 1446389 13 swift 0x000000000055df79 + 1433465 14 swift 0x00000000004e8788 + 952200 15 libc.so.6 0x00007f7a25099ab0 __libc_start_main + 231 16 swift 0x00000000004e832a + 951082 Stack dump:

0. Program arguments: /home/andrew/Downloads/usr/bin/swift -frontend -interpret t.swift -disable-objc-interop -color-diagnostics -module-name t
1. Swift version 5.3-dev (LLVM 5a342fdfac, Swift ec4315266b)
2. While running user code "t.swift" /home/andrew/Downloads/usr/bin/swift[0x51d65b4] /home/andrew/Downloads/usr/bin/swift[0x51d41ae] /home/andrew/Downloads/usr/bin/swift[0x51d688c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f7a26a4b890] /home/andrew/Downloads/usr/lib/swift/linux/libswiftCore.so($ss17_assertionFailure4file4line5flagss5NeverOs12StaticStringV_SSAHSus6UInt32VtF+0x212)[0x7f7a2318a4f2] /home/andrew/Downloads/usr/lib/swift/linux/libswiftCore.so(+0x1b87b9)[0x7f7a231d47b9] /home/andrew/Downloads/usr/lib/swift/linux/libPythonKit.so(+0x1699f)[0x7f7a23fd699f] [0x7f7a26e7818d] /home/andrew/Downloads/usr/bin/swift[0x59ae63] /home/andrew/Downloads/usr/bin/swift[0x587af0] /home/andrew/Downloads/usr/bin/swift[0x56df1a] /home/andrew/Downloads/usr/bin/swift[0x5611f5] /home/andrew/Downloads/usr/bin/swift[0x55df79] /home/andrew/Downloads/usr/bin/swift[0x4e8788] /lib/x86_64-linux-gnu/libc.so.6(libc_start_main+0xe7)[0x7f7a25099b97] /home/andrew/Downloads/usr/bin/swift[0x4e832a] Illegal instruction (core dumped)

My nvidia-smi profile is as follows

[Andrew-Pynch]

Apologies, nvidia-smi profile screenshot didn't display as expected. Here is the raw text copied and pasted

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.59       Driver Version: 440.59       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:25:00.0  On |                  N/A |
| 16%   55C    P0    62W / 250W |   1037MiB / 11175MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2254      G   /usr/lib/xorg/Xorg                            41MiB |
|    0      2398      G   /usr/bin/gnome-shell                          49MiB |
|    0      2746      G   /usr/lib/xorg/Xorg                           532MiB |
|    0      2880      G   /usr/bin/gnome-shell                         167MiB |
|    0      3284      G   ...AAAAAAAAAAAACAAAAAAAAAA= --shared-files   119MiB |
|    0     14694      G   ...quest-channel-token=1115807171952670491    14MiB |
|    0     17360      G   /proc/self/exe                                85MiB |
+-----------------------------------------------------------------------------+

[BradLarson]

To confirm: if you fire up a python3 and import numpy as np or similar, numpy is visible to your Python interpreter in your current virtual environment?

If so, would you be able to try the same code with the 0.8 Swift for TensorFlow release toolchain to see if this is a regression in the 0.9 toolchain? Our Python interoperability Colab notebook still seems to be able to import numpy correctly with Python 3.6.9 and the 0.9 toolchain.

[Andrew-Pynch]

It might have something to do with me using conda? If I fire up python and type the same commands as I did in swift this is the output I receive.

(base) andrew@andrewpc:~/Downloads$ python3
Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> print(np)
<module 'numpy' from '/home/andrew/anaconda3/lib/python3.7/site-packages/numpy/__init__.py'>
>>> zeros = np.ones([2, 3])
>>> print(zeros)
[[1. 1. 1.]
 [1. 1. 1.]]
>>> 

I uninstalled the 0.9 toolchain and installed the 0.8 toolchain and attempted to run the same code from my first post

CODE

#if canImport(PythonKit)
    import PythonKit
#else
    import Python
#endif
print(Python.version)

let np = Python.import("numpy")
print(np)
let zeros = np.ones([2, 3])
print(zeros)

OUTPUT

(base) andrew@andrewpc:~/Downloads$ swift t.swift 
3.6.9 (default, Apr 18 2020, 01:56:04) 
[GCC 8.4.0]
Fatal error: 'try!' expression unexpectedly raised an error: Python exception: No module named 'numpy': file /swift-base/PythonKit/PythonKit/Python.swift, line 678
Current stack trace:
0    libswiftCore.so                    0x00007f7a7d014910 swift_reportError + 50
1    libswiftCore.so                    0x00007f7a7d085b90 _swift_stdlib_reportFatalErrorInFile + 115
2    libswiftCore.so                    0x00007f7a7cd65e31 <unavailable> + 1461809
3    libswiftCore.so                    0x00007f7a7cd65a77 <unavailable> + 1460855
4    libswiftCore.so                    0x00007f7a7cd66012 <unavailable> + 1462290
5    libswiftCore.so                    0x00007f7a7cd64460 _assertionFailure(_:_:file:line:flags:) + 517
6    libswiftCore.so                    0x00007f7a7cda8819 <unavailable> + 1734681
7    libPythonKit.so                    0x00007f7a7c2b197f <unavailable> + 92543
9    swift                              0x00000000005393a9 <unavailable> + 1282985
10   swift                              0x000000000053d9c0 <unavailable> + 1300928
11   swift                              0x000000000052a23e <unavailable> + 1221182
12   swift                              0x0000000000514d66 <unavailable> + 1133926
13   swift                              0x00000000005091c3 <unavailable> + 1085891
14   swift                              0x0000000000505f9a <unavailable> + 1073050
15   swift                              0x000000000048e568 <unavailable> + 583016
16   libc.so.6                          0x00007f7a7e7b0ab0 __libc_start_main + 231
17   swift                              0x000000000048e11a <unavailable> + 581914
Stack dump:
0.      Program arguments: /home/andrew/Downloads/usr/bin/swift -frontend -interpret t.swift -disable-objc-interop -color-diagnostics -module-name t 
1.      Swift version 5.2-dev (LLVM b3057cffb6, Swift da7410955d)
2.      While running user code "t.swift"
/home/andrew/Downloads/usr/bin/swift[0x505d4a4]
/home/andrew/Downloads/usr/bin/swift[0x505b09e]
/home/andrew/Downloads/usr/bin/swift[0x505d75c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f7a80162890]
/home/andrew/Downloads/usr/lib/swift/linux/libswiftCore.so($ss17_assertionFailure__4file4line5flagss5NeverOs12StaticStringV_SSAHSus6UInt32VtF+0x212)[0x7f7a7cd64672]
/home/andrew/Downloads/usr/lib/swift/linux/libswiftCore.so(+0x1a7819)[0x7f7a7cda8819]
/home/andrew/Downloads/usr/lib/swift/linux/libPythonKit.so(+0x1697f)[0x7f7a7c2b197f]
[0x7f7a8058d18d]
/home/andrew/Downloads/usr/bin/swift[0x5393a9]
/home/andrew/Downloads/usr/bin/swift[0x53d9c0]
/home/andrew/Downloads/usr/bin/swift[0x52a23e]
/home/andrew/Downloads/usr/bin/swift[0x514d66]
/home/andrew/Downloads/usr/bin/swift[0x5091c3]
/home/andrew/Downloads/usr/bin/swift[0x505f9a]
/home/andrew/Downloads/usr/bin/swift[0x48e568]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f7a7e7b0b97]
/home/andrew/Downloads/usr/bin/swift[0x48e11a]
Illegal instruction (core dumped)

[brettkoonce]

the first call is pulling in conda python3, the second one is pulling in system python, and then you don't have numpy in the system python install. find the system python3 and do

/hard/coded/path/to/system/python3 -m pip install numpy

and things should work for your demo. alternatively, there's probably some way to point pythonkit at your conda install.

[Andrew-Pynch]

You were right. Installing the packages on the sys python3 resolved the issue. I would like to figure out a way to point swift to anaconda as the default python. @BradLarson Any thoughts?

[Andrew-Pynch]

The solution was to install packages on sys python since all my data science packages were installed on anaconda

EXAMPLE requirements.txt

numpy
pandas
matplotlib
scipy
scikit-learn
fastai
tensorflow
keras
pillow
opencv-python

INSTALL

pip3 install -r requirements.txt

[s-udhaya]

@Andrew-Pynch Adding the conda env python lib path in the environment variable(export PYTHON_LIBRARY="~/anaconda3/lib/libpython3.7m.so") solved the issue for me. Pls look the link posted by Brad for reference

[PrashantKT]

Facing the same issue

PythonKitBasics[44550:459091] PythonKit/Python.swift:674: Fatal error: 'try!' expression unexpectedly raised an error: Python exception: No module named youtube_dl

can anyone help ? https://github.com/anupamchugh/iowncode/tree/master/PythonKitBasics

[Andrew-Pynch]

@PrashantKT I don't have a solution to this particular problem. A quick heads up, swift for TensorFlow is deprecated and no longer being worked on to my knowledge.