Structural data construction and anonymous functions without closures

[Eliah-Lakhin]

Hello!

I'm very impressed with Rhai project, and especially because of it's minimalistic design, transparent interoperability with Rust and active development. In my opinion Rhai is the best choice for Rust embedding scripting among other options we currently have in Rust ecosystem. And I'm going to use Rhai in my personal project of 3D procedural modelling toolkit based on ray marching of signed-distance fields. The scripting will be used for high-level scene live coding in my project.

The scene itself consists of a hierarchy of domain specific objects describing various 3D primitives, their combinations and a lot of effects applied on top of them. And is currently constructing using Rust functions that I would like to export(at least partially) into Rhai context.

There are practically two issues that I faced between my design and Rhai interoperability limitations. Let me show an example of a scene code in Rust:

// starting scene description
// "space" is a context of the current subspace of the scene
Primitive::scene(|space| {

    space
        .view() // receiving &mut reference to the model view related to the current "space"
        .translate(1.5); // Mutating model view matrix

    space
        .add_spape(&Ellipsoid::sphere(1.0)) // creating a Sphere in the current context
        .union(|space| { // entering into a new inner context related to the just added shape of the outer context
            ... // similar operation with the inner context
        });

});

So, the first issue is that in Rhai we can't return &mut reference to the object. This is actually not a big deal as I can provide a setter instead of "&mut" getter to redefine(or update) owned property of the space object.

But the second issue is actual hierarchical structures encoding. Entering into a new "space" scope as shown in my example above as far as I understand is not achievable in Rhai, because it doesn't have anonymous functions. In #159 @schungx mentioned that it would be easy to add such functions without closures, but he said without closures they are not really useful. I want to say that in my particular case they would be VERY useful even without closures. I don't practically use closures to the upper context that much. So, if you can include simplified syntax version in prior-1.0 release that would be very helpful.

Also, as I understood from The Book, it is possible to deal with the context through the this variable inside a "method" function, but I didn't realize how to access(and especially how to "shadow") this in the registered functions if it's even possible.

Since my project is in relatively early stage, I'm fine to slightly change my API design to better fit Rhai design requirements. I will be appreciated for your help and advices. My goal is to have a long-term cooperation with the project teams my project is related to. So if you need my contribution to Rhai, I will be glad to participate too.

Thanks in advance!

Ilya.

[Eliah-Lakhin]

To explain what I'm trying to achieve(at least with the current syntax) is something like this:

#[derive(Clone, Debug)]
struct Foo {}

#[derive(Clone, Debug)]
struct Bar {}

fn main() {
    let mut engine = Engine::new();
    let mut scope = Scope::new();

    engine.registe_type::<Foo>();
    engine.registe_type::<Bar>();

    engine.register_fn("bar", |foo: &mut Foo, callback: CallbackType| {
        callback.set_this_to_bar_and_call_with_param.execute(foo.create_bar(), 100);
    });

    engine.eval_with_scope::<_>(&mut scope, r#"
        fn do_action(x) { print(this.data); }

        let foo = foo_new();

        foo.bar(Fn("do_action"));
    "#);
}

[schungx]

You'd need to register a Rust function that takes a "function pointer" as parameter (i.e. your CallbackType). You can't do that right now because, currently, Rhai has no built-in functionalities to call a script-defined function within a Rust function.

I'm just guessing here, how about something like this:

    let ast = engine.compile(... the script ...)?;

    engine.register_result_fn("bar", |foo: &mut Foo, callback: &str|
        engine.call_fn(&mut Scope::new(), &ast, callback, ( ... parameters ... ))
    });

    engine.eval_ast_with_scope::<_>(&mut scope, &ast)?;

You leverage the following properties of Rhai:

• Engine is re-entrant
• AST can be shared

[schungx]

OK, I tried it. Not quite simple as this. The closure given to register_result_fn must be 'static. Therefore it cannot hold references to engine, ast etc.

Something like this will compile:

    let ast = engine.compile(... the script ...)?;

    let ast_copy = ast.clone();
    let engine2 = Engine::new();

    engine.register_result_fn("bar", move |foo: &mut Foo, callback: &str|
        engine2.call_fn(&mut Scope::new(), ast_copy, callback, ( foo, ... parameters ... ))
    });

    engine.eval_ast_with_scope::<_>(&mut scope, &ast)?;

Unfortunately you cannot bind to the this pointer in the script.

For later versions, I'll modify the function Engine::call_fn_dynamic so that it can optionally pass a value to bind to this.

[Eliah-Lakhin]

@schungx , thank you for you reply!

Well, it seems that using functions in form of callbacks is not that simple in Rhai :) If I have to create a new engine in the "register(_result)_fn" every time, it also implies that I will have to enrich(re-register) that new engine with all other functions related to my framework that already registered in the top Engine instance. Also, referencing the named functions for callbacks is dangerous because the end-user can recreate the function with the same name, means that engine2.call_fn will call wrong reference.

To sum up, I'm afraid that callback concept is not that useful in Rhai for my architecture at least in this stage of Rhai development. I will try to reformat my framework to bypass callbacks completely, or maybe choose some other scripting language for now until Rhai stabilized.

Anyway, I will keep an eye on Rhai as the project has many other advantages that other solutions don't have. Thank you for your great work!

[schungx]

No prob! You should use whatever fits your application the best!

In Rhai, you can simply create a package with all the functions you want to add to an Engine. The using Engine::new_raw() and Engine::load_package() you can make engine creation extremely cheap.

I'm readying a couple of new low-level API's that will actually pass an Engine instance for a user function to use.

[Eliah-Lakhin]

@schungx After thinking a little further I decided to give Rhai another try. The idea is that since it's difficult to adapt closures-based API in Rhai, perhaps I can implement a flat adapter for the Scene constructor that will use internal stack to enter and leave construction context. I think it should work fine in most cases. And I will try to implement it as a set of rhai Packages.

So, for the Package(Module) what I didn't realize is how to set up user-define types. Within an Engine instance I can do it this way: engine.register_type::<Point2<f64>>();, but I didn't find an alternative for the Module instance. In fact I'm not sure if the register_type is needed at all. All interactions with my crate types defined through the public methods(including constructors), and there are no pub attributes of the types. They all encapsulated. So, do I really need register_type in this case? Does it provide any advantages except the direct access to the struct properties?

[schungx]

If you can pull from this PR: https://github.com/jonathandturner/rhai/pull/176

With this version you can do this:

engine.register_raw_fn_2::<Foo, ImmutableString>("bar",
    move |engine: &Engine, lib: &Module, args: &mut [&mut Dynamic]| {
        // `args` are guaranteed to have the correct number and of the correct types
        // so you don't need to check for errors.  Use `unwrap` galore!
        let callback = args[1].clone().cast::<ImmutableString>();
        let this_ptr = args.get_mut(0).unwrap();

        // `engine` is the same instance of the running engine, so it contains all your configurations and settings.
        // 'lib' is a collection of all functions in the AST.
        engine.call_fn_dynamic(&mut Scope::new(), lib, &callback,
                    Some(this_ptr), [ ... parameters as Dynamic ... ])
    }
});

engine.eval_with_scope::<_>(&mut scope, "... script ...")?;

The new, low-level API Engine::register_raw_fn_XXX makes it happen. But you'll have to go thru some hoops to use that.

[schungx]

Also, referencing the named functions for callbacks is dangerous because the end-user can recreate the function with the same name, means that engine2.call_fn will call wrong reference.

You're right. Rhai takes a very free form approach wrt functions, based only on name resolution. It gives users tremendous flexibility to override functions.

If you do not want to allow users to override certain system functions (registered in Rust), what you can do is:

1. Compile the user script into AST
2. Use AST::retain_functions to filter off all user functions that you don't want.
3. Give a message back to the user warning him/her that the functions are junked.

[schungx]

So, do I really need register_type in this case? Does it provide any advantages except the direct access to the struct properties?

No, you do not need register_type. Rhai transparently works with any Rust type.

Engine::register_type is here to complete the API. Right now it does nothing, other than pretty-print the type name if you use Engine::register_type_with_name.

[Eliah-Lakhin]

@schungx Thank you very much for the fast response!

If you can pull from this PR: #176

Yes, this is something I will definitely utilize in my API architecture. I think I will implement both approaches. First is a flat version when the user can enter and leave inner tree construction context using exported enter/leave-like functions with an internal stack implemented. And the second is using the callbacks approach with register_raw_fn_XXX.

For the further improvements it would be nice if these methods will be available on the Module API too(unless it's already implemented), and also it would be really good if you can implement a syntax for anonymous functions even without closures, like this:

    let engine = Engine::new();
    let scope = Scope::new();
    //... initialization
    engine.eval_with_scope::<..>(&mut scope, r#"
        bar.foo(|x, y| {
            x + y
        });
    "#);

The anonymous function here perhaps could be introduced by adding some unique name into the Scope like $anon_<hash> or something like this that cannot be introduced by the user manually, but fits for the Scope programmatic looking up. And the expression result will be a Ref/ImmutableString referring this name. This way we can avoid clashes with the functions shadowing problem.

[schungx]

This seems to be a good idea... anonymous functions. Hoisted to the global namespace but with a name that users can never override...

I can expand this idea into support anonymous functions in Fn declarations:

let f = Fn(|x, y| x + y);

let f = Fn(x, y | x + y);     // alternative syntax

let f = Fn(fn (x, y) { x + y });     // allow fn

let f = |x, y| x + y;       // maps to Fn(...)

becomes

fn $anonymous_fn_1$ (x, y) { x + y }

let f = Fn("$anonymous_fn_1$");

However, what should happen when we pass this function pointer to a function? If it is a Rhai function, it is fine:

fn bar(f) { f.call(1, 2) }

bar(f)

but for Rust functions...

bar(f)                 // auto convert Fn -> string? or

bar(f.name)       // force to use string?

[Eliah-Lakhin]

@schungx For Rust interoperability actually you can provide both options:

1) Simple f.name() will return function's name to be used within engine.call_fn_dynamic as suggested in the snippet above. It will be a low-level approach when the user needs more control. 2) f.call_<n><_mut>(<optional this argument>, <other arguments>) shortcut that will do the job of that snippet under the hood: automatically converting arguments to Dynamic, binding Scope/Module(if possible) and calling call_fn_dynamic of the current Engine in use. This form also will be consistent with the JavaScript function call convention that Rhai's syntax mimics too.

[schungx]

f.call_<n><_mut>(<optional this argument>, <other arguments>) shortcut that will do the job of that snippet under the hood: automatically converting arguments to Dynamic, binding Scope/Module(if possible) and calling call_fn_dynamic of the current Engine in use. This form also will be consistent with the JavaScript function call convention that Rhai's syntax mimics too.

Unfortunately we can't quite do this in native Rust. There must be engine and lib parameters, otherwise there is no way to pass the current execution context to the callback function.

[schungx]

But which one of the anonymous func syntax do you like best? Keep it Rust-like?

[Eliah-Lakhin]

But which one of the anonymous func syntax do you like best? Keep it Rust-like?

You mean f or f.name? Personally I prefer explicit behavior over implicit, so f.name looks better to me.

[schungx]

No, I mean the syntax for declaring anonymous functions.

Both f.name and f.name() work. Both property getter and method are declared for FnPtr.

[Eliah-Lakhin]

Oh, you mean Rhai syntax. I would go with Rust-like syntax: let f = |x, y| x + y; // maps to Fn(...) IMO, it would be consistent and transparent for understanding for the user.

[schungx]

OK, maybe let's solicit more comments before implementing something that would be difficult to remove/change in the future.

I'll put it under a separate branch first.

[Eliah-Lakhin]

@schungx For the engine.register_raw_fn_<x> series of functions. Is there a way(or will it be implemented in the future?) to access these methods from the Module instance?

[schungx]

Actually Engine::register_raw_fn_XXX are implemented on top of Module. Everything is a module in Rhai. A script has a global scope module.

The function you want is Module::set_fn_var_args. Actually, this is Engine::register_raw_fn_XXX:

self.global_module.set_fn_var_args(name, arg_types, func);

That's it. it basically re-wraps Module::set_fn_var_args.

In Module, there is a series of convenient API, for example Module::set_fn_1, Module::set_fn_2_mut etc.

There is a simple write-up in the book (chapter "Create a Module from Rust"), but for details you'll have to look at the Module documentation.

EDIT: Ah, ma bad. Module::set_fn_var_args is internal. I can make it public!

[Eliah-Lakhin]

In Module, there is a series of convenient API, for example Module::set_fn_1, Module::set_fn_2_mut etc.

Yes, I already use them, but none of them provide low-level stuff such as Engine and Module too.

[schungx]

I'll upload a new commit soon with Module::set_raw_fn public.

[schungx]

The latest version changes Engine::register_raw_fn so now it can return any type, not just Dynamic. It'll do the conversion automatically.

Also, rhai::FnPtr is exposed as a valid data type which encapsulates function pointer.

So now the code can become:

engine.register_raw_fn_2::<Foo, FnPtr>("bar",
    move |engine: &Engine, lib: &Module, args: &mut [&mut Dynamic]| {
        let fp = args[1].clone().cast::<FnPtr>();
        let this_ptr = args.get_mut(0).unwrap();
        engine.call_fn_dynamic(&mut Scope::new(), lib, fp.fn_name(), Some(this_ptr), [ ... parameters as Dynamic ... ])
    }
});

The script can call bar with Fn parameter:

foo.bar(Fn("do_action"));

[schungx]

I have an implementation of anonymous functions which is in the latest master.

Syntax: same as Rust closure, i.e. |x, y, ... | expr or |x, y, ... | { stmts }.

This is just syntactic sugar for Fn("anonymous_function_1234567") where the function names are hash-generated.

[Eliah-Lakhin]

@schungx Just pulled from your branch and tested this new syntax with Module::set_raw_fn function register. Works perfectly! Thank you so much!

[Eliah-Lakhin]

@schungx Just out of curiosity. The Module::set_raw_fn's callback accepting &Engine, &Module and a mutable array of &mut Dynamic arguments. Are there any particular reasons why this function cannot also accept the current Scope from which the registered function was called? Having this feature one could mimic closures quite easily. At least in case when the anonymous function was created from the same context. Of course this cannot be a treated as a general solution, but anyway.

In fact I'm going to use anonymous functions for the blocks of code separation purposes. In other words for the end-user to construct a tree-like structure in form of:

let x = 200;

root.add_branch(|| {
  // "this" refers to a new Branch object created in place when the "add_branch" was called.
  // That object will be added to the "root" once the function call finished.
  this.some_branch_value = 100;

  // would be good to access "x" here too.

  this.add_branch(|| {
    //adding a subbranch

    // and accessing "x" from the "root"'s Scope as well
  });

  // continuing middle branch construction
});

So if any of the callbacks could run in the same Scope from which it was called it would cover the "closures" capabilities quite well.

[schungx]

Are there any particular reasons why this function cannot also accept the current Scope from which the registered function was called?

No reason that I can see, except for the fact that it is extremely hard to predict what that scope will be at function invocation time. Basically, it will be the scope at the time of the function's calling, instead of normal closures where the scope is actually captured into the function/closure itself.

Having a function run depending on the environment at the time of calling will be quite unlike any other language out there... and I think will be extremely confusing for the user.

[schungx]

Looking at your example more closely, this is exactly the reason why closures capture their environment - that is to pass values into closures without resorting to function parameters. Actually, Rust closure captured variables are parameters to the underlying functions, just syntactic sugar...

You're not going to be able to do that easily. For example, x will disappear inside a function call - a function call always starts with a clean scope with only the parameters in it. In order to refer to variables outside, I need to keep a "scope-chain" like JavaScript.

[Eliah-Lakhin]

@schungx ok, as a workaround and to prevent end-user confusion I would suggest to introduce a different approach. Something similar to JavaScript's bind method for Rhai functions:

let x = 100;
let f = (|x| { x = x + 50; x }).bind(x); // this call returns another anonymous function with zero parameters that will call original wrapped function under the hood with a copy/clone from x value at a moment of `bind` call.

print(f.call()); // prints 150
print(x); // prints 100. A function is still pure, it didn't mutate x outside of it's scope.

x = 200;

print(f.call()); // prints 150 as "x" was bound to the previous x value

I guess such feature could be implemented on top of already existing Rhai API through registered functions(by registering such bind function with this semantic), but I'm not sure how to create anonymous function as result of registered Rust function call.

[schungx]

If you pull from my latest master, there is already a version of the call method that binds the this pointer.

In your example, it would be:

let x = 100;
let f = |v| { this += v; this };

x.call(f, 50);     // x => 150

You can even mutate the this binding.

[Eliah-Lakhin]

@schungx Well, the thing is I would like to use this to be bound to the Rust object created in place by registered function. In other words I want the end-user to use callback as a "constructor"/"configurator" of the Rust object. I will use this pattern not everywhere, sometimes I will provide a user a way to construct an object in ordinary way(through the ordinary constructor functions and getters/setters), but sometimes the described approach might be handy too. The end-user will construct complex hierarchical structures, so having both approach might be good.

Anyway, I will figure out how to do this with Rhai, the current Rhai capabilities are already quite flexible. Thank you very much for the quick responses and fixes! I'm using your master branch extensively. Your work is really great, and it helps my project a lot.

[Eliah-Lakhin]

fyi, this is an example of a work-in progress of a Rhai scripting in my project: https://gist.github.com/Eliah-Lakhin/a42caebfc0bac8b3369825f11c1b565e :)

[schungx]

let f = (|x| { x = x + 50; x }).bind(x); // this call returns another anonymous function with zero parameters that will call original wrapped function under the hood with a copy/clone from x value at a moment of bind call.

Looks like you're currying the original function. This is akin to passing state into a closure which requires "real" closures...

I would like to use this to be bound to the Rust object created in place by registered function

Not very sure what you mean by this... can you elaborate?

Thank you very much for the quick responses and fixes!

You're welcome! I am already spending way more time than I actually should on this... so mind as well do it right and have something useful to the community!

[Eliah-Lakhin]

@schungx ok. We are actually already discussing a different topic now: closures and currying. The original topic of providing anonymous functions syntax without closures is fully covered and implemented. So I consider to close this Issue.

As of closures, not sure how they could be implemented in the current Rhai environment right way, but maybe I will try to implement and pull a PR with a syntax for currying myself later.

[schungx]

Might be possible to extend the FnPtr data type to include captured argument values...

When running a function, first find it by adding the number of arguments to the number of captured arguments.

Then fill in the missing arguments from the captured values...

It may turn out to be quite simple to add.

Have to think about this some more.

[schungx]

Yes, I'm right. Took me a few hours to add currying support.

Do it via the curry keyword:

fn mul(x, y) {                  // function with two parameters
    x * y
}

let func = Fn("mul");

func.call(21, 2) == 42;         // two arguments are required for 'mul'

let curried = func.curry(21);   // currying produces a new function pointer which
                                // carries 21 as the first argument

let curried = curry(func, 21);  // function-call style also works

curried.call(2) == 42;          // <- de-sugars to 'func.call(21, 2)'
                                //    only one argument is now required

So your example now becomes this:

let x = 100;
let f = |x| { x = x + 50; x }.curry(x);

print(f.call()); // prints 150
print(x); // prints 100. A function is still pure, it didn't mutate x outside of it's scope.

x = 200;

print(f.call()); // prints 150 as "x" was bound to the previous x value

[Eliah-Lakhin]

@schungx Just checked out from your branch. Seems like curry-ed arguments ignored when the curry-ed function called from registered function.

Registered function:

    lib.set_raw_fn(
        "registered_func",
        &[TypeId::of::<FnPtr>()],
        |engine: &Engine, module: &Module, args: &mut [&mut Dynamic]| {
            let callback = std::mem::take(args[0])
                .cast::<FnPtr>()
                .fn_name()
                .to_string();

            let _ = engine.call_fn_dynamic(
                &mut Scope::new(),
                module,
                &callback,
                None,
                [Dynamic::from(10)],
            )?;

            Ok(())
        },
    );

This Rhai code prints 10 instead of 100:

let curried = |x| { print(x) }.curry(100);
registered_func(curried);

Please notice that I explicitly passed an argument([Dynamic::from(10)]) to the call_fn_dynamic invocation. Removing this argument([]) leads to runtime error in form of:

  ErrorInFunctionCall(
    "anon$1672011711214155021",
    ErrorFunctionNotFound(
        "anon$6271522687537960879",
        ...,
    ),
    ...,

So, it clearly shows that call_fn_dynamic calls original function instead of a curry-ed form.

[Eliah-Lakhin]

@schungx I think I found a workaround for this: https://github.com/schungx/rhai/pull/17 Please review my PR, when you have time. Thank you!

[schungx]

Just off-hand, I think you can avoid an allocation with:

let callback = std::mem::take(args[0]).cast::<FnPtr>().fn_name();

let _ = engine.call_fn_dynamic(
                &mut Scope::new(),
                module,
                callback,
                None,
                [Dynamic::from(10)],
            )?;