IPython magic-like construct for scripts

[marius311]

This is a proposal for something I can implement, but wanted to get some feedback first.

Basically, I find the IPython magic incredibly convenient to pass values back-and-forth between Julia/Python. It'd be nice to have something that works in scripts too. I propose having the function jl which takes in a string and executes it just like the IPython magics do. The choice of jl is to have a nice symmetry between Julia where you do,

py" ...Python code... "

and Python where you'd do,

jl(" ...Julia code... ")

The behavior with regards to scope / interpolation / etc would be identical to the Julia magic. Any thoughts?

[tkf]

I don't think it's a good idea to promote magic behavior outside IPython (e.g., it breaks pyflakes). I rather prefer to expose a lower-level API Main.eval(src, dict) and let users explicitly do Main.eval(src, locals()). You can then get the jl function with the sys._getframe hack with just a few lines.

[marius311]

I rather prefer to expose a lower-level API

Yea, that's what I mean.

[marius311]

To be clear, what I want is basically to do all the examples from the docs in a script and without depending on IPython, so it'd be something like e.g.

# file: myscript.py

from julia import jl

arr = [1, 2, 3]

jl("$arr .+ 1")

jl('sum(py"[x**2 for x in arr]")')

etc..., and this would be done by just hooking into the lower-level stuff that already exists, not the IPython wrapper. If that sounds reasonable I'll go ahead and draft something up!

[tkf]

I don't think we should be helping users to write code expecting sys._getframe hack. For example, it breaks pyflakes. Refactoring JuliaMainModule to add the 2-arg version is OK. You can then create a simple Python package with the jl function on top of it.

[marius311]

Which part of the sys._getframe do you not like? Is it just the use of sys._getframe at all? Or is it the search for the parent frame that we do for the magic:

https://github.com/JuliaPy/pyjulia/blob/0c0d74f72c473c60ea530b8f3c105aba6c903f27/src/julia/magic.py#L112-L118

Because for my proposed jl() function, you don't need the latter thing, since there you'll be guaranteed that one frame before jl is the caller's frame.

In any case, I will start drafting up the other parts of this, I do agree that with the internals set up right it will be trivial for a user to implement jl() themselves using sys._getframe, although I do think it'll be nice for this package to have something like jl() working out of the box.

[tkf]

Using sys._getframe like this breaks usual Python semantics. That's why analzyer like pyflakes does not work. sys._getframe should be used only for "secondary" purposes; e.g., generating useful error messages.

I am somewhat OK with the use of _getframe in the IPython magic as this is something we could implement by using IPython API. Maybe we should get rid of this too, as this depends on IPython internals (and a CPython internal); i.e., it would stop working if IPython starts using a higher-order function from an external library.

I do think it'll be nice for this package to have something like jl() working out of the box.

I believe a function like jl degrades maintainability of non-trivial Python code.

[marius311]

What about something roughly like this so we fail gracefully (and allow a workaround) if for whatever reason _getframe fails (like if we're not on CPython) ?

def jl(src, locals=None, globals=None):
    """ Execute some Julia code """

    def getcallerframe():
        try:
            return sys._getframe(2)
        except:
            raise Error('Unable to get caller frame. Please provide locals/globals explicitly `jl("...", locals(), globals())`')

    if locals is None:
        locals = getcallerframe().f_locals

    if globals is None: 
        globals = getcallerframe().f_globals

    src = unicode(src)
    return_value = "nothing" if src.strip().endswith(";") else ""

    return julia.Main.eval(
        """
        _PyJuliaHelper.@prepare_for_pyjulia_call begin
            begin %s end
            %s
        end
        """
        % (src, return_value)
    )(globals, locals)

This is a functioning version of what I'm suggesting above basically. I do get that its mildly hacky but isn't it exactly symmetric with what is done on the Julia end with py"..." which seems ok?