Open orenbenkiki opened 3 months ago
I don't think you can do this with the with
statement - that just runs the code in the with
block immediately, whereas you need to create a function to pass to Julia. It's very different from the do
syntax in Julia which does create a function.
You'll have to do something like
outer_vector = np.array([0])
def callback(inner_vector):
inner_vector[0] = 1
jl.MyModule.caller(callback, outer_vector)
assert outer_vector[0] == 1
First, with
statements do not run the code "immediately", they run it when the yield
statement is invoked in the body of the contextmanager
.
Second and more importantly, even putting this aside, a plain callback doesn't work. For example:
jl.seval("""
function jl_caller(jl_called::Function)
return jl_called("foo")
end
""")
def py_called(text: str) -> str:
return text + "bar"
print(jl.jl_caller(py_called))
Gives the error message:
Traceback (most recent call last):
File "/Users/obk/projects/Daf.py/callback.py", line 12, in <module>
print(jl.jl_caller(py_called))
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/obk/.julia/packages/PythonCall/wXfah/src/jlwrap/any.jl", line 208, in __call__
return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Julia: MethodError: no method matching jl_caller(::Py)
Closest candidates are:
jl_caller(!Matched::Function)
@ Main none:1
It seems that a Python function object is not converted to a Julia function, instead it is wrapped into a generic Py
object, so callbacks just aren't supported?
Julia can call Python and Pythin can call Julia so this is possible. The workaround is somewhat convoluted:
cat callback.py
from contextlib import contextmanager
from juliacall import Main as jl # type: ignore
jl.seval("""
function jl_caller(jl_called::Function)::Any
return jl_called("foo")
end
""")
jl.seval("""
function py_to_function(py_object::Py)::Function
return (args...; kwargs...) -> pycall(py_object, args...; kwargs...)
end
""")
# Pass callback as an argument:
def py_called(text: str) -> str:
return text + "bar"
print(jl.jl_caller(jl.py_to_function(py_called)))
# Use with statement:
@contextmanager
def py_caller() -> None:
def capture(text):
yield text
# yield from capture("foo")
yield from jl.jl_caller(jl.py_to_function(capture))
with py_caller() as text:
print(text + "bar")
Running this prints foobar
twice as expected. Nice.
So, back to the feature request: Can we have a built-in conversion rule that takes plain-old Python functions and lambdas and wraps them as Julia functions. This would allow passing functions as arguments without having to use the above workaround (that is, remove the need for defining and using py_to_function
).
It seems that a Python function object is not converted to a Julia function, instead it is wrapped into a generic Py object, so callbacks just aren't supported?
Well no, they are wrapped as a generic Py
object, but those are still callable, so can be used as callbacks.
Your issue is simply that you've got a ::Function
type annotation on jl_caller
when py_called
is received as a Py
. If you remove it then the simple version works:
>>> jl.seval("""
... function jl_caller(jl_called)
... return jl_called("foo")
... end
... """)
Julia: jl_caller (generic function with 1 method)
>>> def py_called(text: str) -> str:
... return text + "bar"
...
>>> print(jl.jl_caller(py_called))
foobar
I'm pretty sure we were talking at cross purposes about the with
statement. In your original post it looked a lot like you were trying to use with
in the same way as Julia's do
, but in your later posts it seems that's not the case. Anyway that's all tangential to the main issue.
Yes, there are two issues - Py
vs. Function
and with
vs. do
.
My later post showed a workaround around both issues which requires writing manual wrappers.
So it is possible to do achieve what I want (given writing the manual wrappers), which is great!
That said, ideally one should not have to write such wrappers:
Python functions "should" be converted to some PyFunction
type which is a Julia Function
, so they would work even if the Julia function specified ::Function
for the callback argument.
The juliacall
Python module should provide a context
wrapper function so one could, in Python, say:
with juliacall.context(jl.MyModule.foo)(...args...) as ...:
...
Makes sense?
I'm happy to consider the PyFunction idea - feel free to make a separate issue about that.
I don't understand what you want juliacall.context
to do?
Something along the lines of the following (up to bikeshedding on the names and exact syntax):
from contextlib import contextmanager
from juliacall import Main as jl # type: ignore
from typing import Any
from typing import Callable
from typing import Iterator
#: This would not be needed if/when issue #477 is resolved.
jl.seval("""
function py_function_to_fulia_function(py_object::Py)::Function
return (args...; kwargs...) -> pycall(py_object, args...; kwargs...)
end
""")
# Example Julia caller function.
jl.seval("""
function jl_caller(callback::Function, positional:: AbstractString; named:: AbstractString)::Any
extra = 1
return callback(positional, named, extra) # All must be positional.
end
""")
# Example Python callback function.
def py_callback(first: str, second: str, third: int) -> Any:
print(f"first: {first}")
print(f"second: {second}")
print(f"third: {third}")
return 7
# Pass a callback as an explicit Function parameter. Return value is available.
returned = jl.jl_caller(jl.py_function_to_fulia_function(py_callback), "positional", named ="named")
print(f"returned: {returned}")
# Proposed addition to `juliacall`, converts Python `with` to work similarly to Julia's `do`.
@contextmanager
def jl_do(jl_caller: Callable, *args: Any, **kwargs: Any) -> Iterator[Any]:
def capture(*args: Any) -> Iterator[Any]:
if len(args) == 1:
yield args[0]
else:
yield args
yield from jl_caller(jl.py_function_to_fulia_function(capture), *args, **kwargs)
# Use in `with` statement. No return value.
with jl_do(jl.jl_caller, "positional", named = "named") as args:
print(f"args: {args}")
Could you explain some more how this is useful? I don't understand the utility of jl_do
- as far as I can tell it has very little similarity to Julia's do
syntax.
Consider Julia do
:
jl_caller("positional", named="named") do first, second, third
println("first: $(first)")
println("second: $(second)")
println("third: $(third)")
end
Compared to Python with
:
with jl_do(jl.jl_caller, "positional", named="named") as (first, second, third):
print(f"first: {first}")
print(f"second: {second}")
print(f"third: {third}")
Looks mighty similar to me.
If I have a Julia function which takes a callback (artificial example here):
And I'd like to call it from Python - it seems not possible to do so? Ideally:
I have a Julia package that uses callbacks for various functions (for example, initializing arrays), and I'm trying to wrap it with a Python interface. Being able to zero-copy pass around numpy arrays is a godsend, but it seems that callbacks of the above type are not supported. Looking at the code I see the tests for "callback" are empty...
Is there some manual workaround I could use in my code instead of direct support for the above? Any way at all, as long as I can bury the boilerplate code in my Python wrappers so the end user can use the
with
statement.