question about pyimport, numpy, and implementation details

[emin63]

Thanks for creating this package. It looks really neat.

I was particularly intrigued by the pyimport function you mentioned in the README. Having worked with some other javascript to python translators, I was surprised to see this work. Not only that it even works on something like the os module! I guess what you are doing is not translating targets of pyimport into javascript but marking them as untranslated python when you translate javascript to python.

Is that right?

That brings me to the question of numpy. When I try and use pyimport with numpy in the following snippet

import js2py
js2py.eval_js('pyimport numpy;\nconsole.log(numpy.mean([1,2,3,4]));')

it fails and I get an exception.

Digging into the exception a bit suggests that numpy.mean is trying to look at the type of its input and dispatch on that. But since the type of [1, 2, 3, 4] is a javascript object, numpy gets confused and dies. Various other attempts (including using EvalJs to put numpy.array and list into javascript) failed.

Any tips on how one could use js2py with numpy, pandas, etc.?

Thanks.

[PiotrDabkowski]

Hi, thanks, the pyimport is actually quite easy as the Js gets translated to Py and the python modules do not need translation at all, the translated code simply calls python functions directly (with some simple wrapping of arguments and the return value). This simple wrapping is exactly the reason why your np.mean function does not work. As you can see in the conversion table in readme, the Js types are converted to Py types as follows:

Boolean -> bool
String -> unicode (str in Python 3)
Number -> float (or int/long if whole number)
undefined -> None
null -> None
OTHER -> JsObjectWrapper

Hence the value received by the numpy.mean is actually a JsObjectWrapper, the reason why this is done this way can become clearer after checking the readme file. I have just added a hacky opt-in config to convert js arrays and objects to python lists and dicts implicitly:

>>> import js2py
>>> js2py.base.PyJs.CONVERT_TO_PY_PRIMITIVES = True
>>> js2py.eval_js('pyimport numpy;var np=numpy;var a = np.array([1,2,3]);var b = np.array([5,1,3]);a.__add__(b)')
True
array([6, 3, 6])

[abalter]

This did not seem to work for me, or I'm not doing it right.

(js2py) balter@spectre3:~$ cat hellojs2py.py
import js2py
js2py.base.PyJs.CONVERT_TO_PY_PRIMITIVES = True

x = 'console.log("Hello World!");'
js2py.eval_js(x)

y = """
var o = {a:1, b:"b"};
console.log(o);
"""
js2py.eval_js(y)

(js2py) balter@spectre3:~$ python hellojs2py.py
'Hello World!'
Traceback (most recent call last):
  File "hellojs2py.py", line 11, in <module>
    js2py.eval_js(y)
  File "/home/balter/conda/envs/js2py/lib/python3.7/site-packages/js2py/evaljs.py", line 115, in eval_js
    return e.eval(js)
  File "/home/balter/conda/envs/js2py/lib/python3.7/site-packages/js2py/evaljs.py", line 190, in eval
    self.execute(code, use_compilation_plan=use_compilation_plan)
  File "/home/balter/conda/envs/js2py/lib/python3.7/site-packages/js2py/evaljs.py", line 185, in execute
    exec (compiled, self._context)
  File "<EvalJS snippet>", line 2, in <module>
  File "/home/balter/conda/envs/js2py/lib/python3.7/site-packages/js2py/base.py", line 949, in __call__
    return self.call(self.GlobalObject, args)
  File "/home/balter/conda/envs/js2py/lib/python3.7/site-packages/js2py/base.py", line 1463, in call
    return Js(self.code(*args))
  File "/home/balter/conda/envs/js2py/lib/python3.7/site-packages/js2py/host/jseval.py", line 45, in Eval
    executor(py_code)
  File "/home/balter/conda/envs/js2py/lib/python3.7/site-packages/js2py/host/jseval.py", line 51, in executor
    exec (code, globals())
  File "<string>", line 4, in <module>
  File "/home/balter/conda/envs/js2py/lib/python3.7/site-packages/js2py/base.py", line 995, in callprop
    return cand.call(self, args)
  File "/home/balter/conda/envs/js2py/lib/python3.7/site-packages/js2py/base.py", line 1463, in call
    return Js(self.code(*args))
  File "/home/balter/conda/envs/js2py/lib/python3.7/site-packages/js2py/host/console.py", line 9, in log
    print(arguments[0])
  File "/home/balter/conda/envs/js2py/lib/python3.7/site-packages/js2py/base.py", line 1365, in __repr__
    return repr(self.to_python().to_dict())
AttributeError: 'dict' object has no attribute 'to_dict'

[PiotrDabkowski]

Hey @abalter, sorry it looks like there is a bug and js2py.base.PyJs.CONVERT_TO_PY_PRIMITIVES = True breaks the repr method (so the console.log will not work). to_python normally returns a JsObjectWrapper wrapper that has the to_dict method. However, with CONVERT_TO_PY_PRIMITIVES feature enabled it just converts everything to pure python objects that do not have this method. Fix seems to be easy though.

[abalter]

I'd like to try doing some stuff with pandas and statsmodels with javascript to see if it will work. What else do you think would need to happen to get there? If you can give me a few instructions I would take a stab at it.

[abalter]

@PiotrDabkowski Can you describe what a fix might be, and what source file would need the edit?

[PiotrDabkowski]

To fix that all the to_python().to_list() and to_python().to_dict() calls would need to be adjusted (in base.py). For example this one: https://github.com/PiotrDabkowski/Js2Py/blob/master/js2py/base.py#L1365

For your application, I believe it will work but it will be painful to use :) Js2Py integrates Js and Python, but in some cases this is not easily possible. For example, you would not be able to write (np.array([1,2,3))+ 3) in javascript, you would need to do (np.array([1,2,3]).__add__(3)).

[abalter]

I wonder if there would be a way to do a global operator overload wither in JS or Python to fix situations like this. Like perhaps when importing np, overload .__ad__ in EVERY object. Or, add prototype.__add__ to every np object in JS. Would something like that be even feasible?

[PiotrDabkowski]

@abalter Yes, this could be done, but will require some time implementing. Not sure why there would be much benefit from using python modules from Js? Python is a very nice language :) Hence Js2Py mostly allows to easily use JS modules from Python. The other way around is also supported, but is more problematic (as you observed).

[abalter]

Python is a great language. But it does have some shortcomings, one of the main being the lack of a robust method for anonymous functions, which I feel hampers it in terms of functional programming. I'm finding I used that more and more in R programming. Also, I think JS is likely to get a pipeline operator sooner than Python--something R already has. Also, I really like the idea of being able to do rich visualization, widgeting, and dashboarding in the same language as computation, and JS simply excels at those.

Maybe it's a pipe dream (no pun intended), but if it was an easy thing to play around with, I thought it would be a fun experiment.

Perhaps I'll take a stab at the overloading :)

Thanks!

[PiotrDabkowski]

Ok :) I just think using Js2Py in your case will just slow you down and it would be faster to just switch to Python, but you can give it a go. You will in this weird development environment where you execute JS code via Python interpreter... You will not be able to execute this JS code from your browser obviously. Also, there are JS libraries, like for example numjs that you may use directly

[abalter]

Nice, but no datatables or stats :)

I see what you mean about executing JS by Python. I'm probably thinking in reverse of what makes the most sense. It would probably be better to create an API to the python libraries I want in Python and then call them from node via https://github.com/extrabacon/python-shell or something like it.