wapm-packages / python

Python ported to Wasm + WASI
Apache License 2.0
51 stars 2 forks source link

Use python.wasm with NumPy #2

Open ygina opened 4 years ago

ygina commented 4 years ago

I downloaded python.wasm from WAPM and am trying to run a Python file with NumPy.

# run.py
import sys  # Default python.wasm sys.path does not include ./lib/python3.6/site-packages
sys.path = ['', './lib/python3.6', './lib/python3.6/site-packages', './lib/python3.6/lib-dynload']
import numpy as np
print(np.ones(10))

I created a custom lib/ folder with default libraries and the NumPy library from site-packages/:

$ virtualenv env -p python3.6
$ source env/bin/activate
$ pip install numpy
$ deactivate
$ wapm install python
$ cp -r wapm_packages/_/python@0.1.0/lib .
$ cp -r env/lib/python3.6/site-packages lib/python3.6/

I can use this lib successfully with regular Python since I set sys.path:

$ PATH="" env/bin/python3.6 run.py
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]

But it fails with python.wasm (see full stacktrace):

$ wasmer run python.wasm --dir . -- sys.py
Traceback (most recent call last):
  File "./lib/python3.6/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "./lib/python3.6/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "./lib/python3.6/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sys.py", line 11, in <module>
    import numpy as np
  File "./lib/python3.6/site-packages/numpy/__init__.py", line 140, in <module>
    from . import core
  File "./lib/python3.6/site-packages/numpy/core/__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.6 from ""
  * The NumPy version is: "1.19.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

For some reason, overrides.py fails to import numpy.core._multiarray_umath from the file lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-x86_64-linux-gnu.so?

taybenlor commented 3 years ago

Hi @ygina - I just stumbled across this issue and thought I might leave a comment for posterity's sake.

The reason this is not working is because your numpy package is compiled for x86_64-linux. That's because numpy includes C code that needs to be compiled for a particular platform. To get it to work with WASM you would need to compile it to Web Assembly WASI.

Instead I would recommend looking at the Pyodide project. It's a version of python compiled to WASM that includes the scientific stack (things like numpy). https://pyodide.org/en/stable/