Add Value::Opaque type

[Qix-]

This allows the host application to pass around references to Rust objects without allowing the Lisp environment to interact with them (as Foreign specifies, for example).

I've gone back and forth about whether or not it's a good idea to allow true Any (which means any non-crate types) so any opinions would be cool :) If not, then the OpaqueValue trait can go away.

[brundonsmith]

Hmm

It seems to me like Foreign covers this use-case, no? The main benefit to Opaque would be what it doesn't allow you to do, but this being a dynamically-typed scripting language, I'm not sure that's a high priority. And if the goal is to make sure a value is eg. immutable when processed with some lisp code, the user could always just implement the ForeignValue trait but not wire up any commands for it

There is one other advantage to Opaque, which is that it doesn't require an explicit trait implementation for each type you might put in that slot. Though I think we could narrow that gap by:

1. Giving ForeignValue's command method a default implementation that just does nothing (allowing a value to be passed but not sent commands)
2. Possibly doing some blanket-implementations of ForeignValue for common types, so users only have to impl it for their custom types

Thoughts?

[brundonsmith]

I guess maybe Opaque also allows unpacking of values back into their full types? (I just learned about Any recently and haven't played with it yet, so I'm not positive about what all it can do)

That could plausibly be useful, though I'd still be curious if you have a specific use-case in mind

[Qix-]

I guess maybe Opaque also allows unpacking of values back into their full types?

That's the main benefit, and yeah looking back the trait implementation requirement on Opaque doesn't make sense because then you can't store types outside of the implementor's own crate. I'll remove that part and just use an Any.

But yeah, the reason why Foreign didn't work for me was because I wouldn't be able to cast it back to the concrete type. Plus I didn't want to be able to interact with it from lisp.

This is very similar to Lua's lightuserdata type by the way. It allows for the host program to expose internal things go the Lisp environment to allow the lisp code to make decisions about resources without needing to directly interact with them.

In my case, the statements I'm iterating over return configuration commands that are run against a global state object, and those commands are Opaque values returned by functions. They can't be used directly.

[brundonsmith]

Alright, the use-case is coming into focus

Here's my proposal:

• I still think this should fall under the ForeignValue trait; I don't think the use-case is different enough to be worth adding a whole separate concept
• ForeignValue should have a default implementation of command. That way you can just do impl ForeignValue for Foo { } when you don't need to send it commands, without manually implementing an empty command method
• An Rc<dyn ForeignValue> should have the ability to be downcasted back into the underlying value. Not sure exactly what this will need to look like, possibly it will use Any, but it should be doable
• I don't think "preventing interaction with the value from lisp" is a goal that's worth trying to serve here, especially since you can achieve that by just not implementing any actual commands for the type

How does that sound?

[brundonsmith]

Follow-up: it's possible the thing to do is just to put Any in the Foreign variant, and have the trait be an optional add-on. Not sure.

[Qix-]

That way you can just do impl ForeignValue for Foo { } when you don't need to send it commands, without manually implementing an empty command method

The problem with this is if Foo is a type that originates outside the implementing crate, then you can't implement ForeignValue for it. You'd have to do a newtype pattern to do so, which sort of adds a lot of boilerplate. This is also why I mentioned I'd remove the trait bound (though currently out of the country for the holidays so best I can do at the moment is comment 🙃).

The other thing is, if memory serves, Any adds more runtime overhead than the current ForeignValue implementation. I could be wrong though.

[Qix-]

Nevermind, found some time to push the rebase/trait removal today :) Still interested to see how Foreign could be adapted.

[brundonsmith]

The problem with this is if Foo is a type that originates outside the implementing crate, then you can't implement ForeignValue for it

Agh I forgot about that limitation, that's a shame

Any adds more runtime overhead than the current ForeignValue implementation

Not as far as I'm aware; any dynamic trait object (dyn) adds a small amount of overhead, but we have to do that for ForeignValue anyway, and I don't know of any additional overhead for Any

I took another swing at this today, and gave it a lot of thought- turns out it's hard or impossible to represent "an Any which might also be downcast to a ForeignValue" without a second layer of heap allocation (a Box<dyn Any> that holds a Box<dyn ForeignValue>, which- performance isn't the highest priority given this is a scripting language, but even then, this is pretty egregious

But then I realized... the whole idea of ForeignValue originated before I knew that Any existed (I learned about it like a week before your PR was made 🙂)

I think we could actually drop ForeignValue and the "command" concept entirely. Value::Foreign would just hold an Any, and instead of implementing commands for each foreign type, you'd just add regular functions to the environment that downcast them and work with them directly

So, I think I'll go ahead and implement that!

[brundonsmith]

The above has been implemented: https://github.com/brundonsmith/rust_lisp/commit/893fc8bf0502376d9df90787116e251a768eb7f3

Value::Foreign now holds a Rc<dyn Any>; if the value needs to be mutable it can be a RefCell<>. It can be interacted with via lisp code by adding env functions that extract data from it and/or mutate it

[Qix-]

Awesome, I think that's a much more elegant design anyway. Thanks for looking into this :)