Allow arbitrary creation of tables from C++ functions.

[Rapptz]

At the moment the only way to create a table from C++ and returning it from a binded function is to use sol::table as the return type like so:

sol::table stuff() {
}

However, the only way to actually create a sol::table is through sol::state::create_table which is very awkward to use with free functions. So you typically have to use a lambda or something like so:

int main() {
    sol::state lua;
    lua.set_function("stuff", [&lua](int x) {
        auto&& t = lua.create_table();
        // do stuff with table
    });

It's very annoying and not very intuitive with current C++ code.

Instead, it would be nice if C++ containers returned tables as well.

Example:

std::vector<int> stuff() {
    return { 1, 2, 3, 4, 5 };
}

int main() {
    sol::state lua;
    lua.set_function("stuff", stuff);
}

And then from the lua side:

local t = stuff()
for i = 1, #t do
    print(t[i])
end

Would print 1 2 3 4 5

[Rapptz]

Implemented.

[princesstrash]

This issue is closed, but I had to ask...

Consider the following snippet

#include <sol.hpp>
#include <vector>

std::vector<int> make () {
    return { 1, 2, 3, 4, 5 };
}

void take ( std::vector<int> mreow ) {
    (void)mreow;
}

int main () {
    sol::state lua;
    lua.open_libraries( sol::lib::base );

    lua.set_function( "make", make );
    lua.set_function( "take", take );
    lua.script( "a = make()\n"
                "take(a)"
                );
}

Upon calling take the program will segfault. Is this to be expected? Should I simply try to find a work around?

[Rapptz]

It's definitely a well known issue. I encountered it when implementing it. The work around at the moment is to take in sol::table instead of std::vector. A lot of information is lost when going back and forth and it's hard to keep all this stuff transparent. It's essentially a form of type erasure so it's hard to fix.

[princesstrash]

I see...

I have a bit of free time. Perhaps I can take a look at trying to fix this, if you wouldn't mind? It'd give me a chance to learn to flex my templates.

[Rapptz]

You're welcome to send in a pull request. I have no issues with that.

[princesstrash]

Sounds like a plan...

Initial looking over the code suggests that there's potential in using the userdata class to represent these kinds of classes...

The problem is I'm not sure if it's possible to hook operators as I understand them in C++ (operator[] and the like) to lua, just by using userdata's constructor. I'll give it a shot, though!

[princesstrash]

I made some progress, but just to keep you up to date with what I'm doing...

It seems that the kind of SFINAE (if I'm learning and using the term correctly) that you're applying to pusher<T> had inferior (if that's the right word...) capabilities, namely for the ability to override functionality later on. Consider the lines for this class.

Notice that the SFINAE is entirely internal to the class itself. This is fine, but becomes inadequate when you want to overload pusher<T> not based on a partial specialization (e.g. pusher<std::vector<T>>) but based on whether or not that T meets a criteria (e.g. pusher<has_begin_and_end<T>>.

Some digging led me to believe that modifying pusher's signature to include an extra template parameter for SFINAE could solve the problem. I'm taking that approach right now, and it seems to compile and behave as expected (e.g. with template <typename T, typename X> struct pusher, and template<T> struct pusher<T, has_begin_and_end<T>> (there's a std::enable_if in there somewhere as well)).

This template stuff is hard but I can see the appeal in it greatly. If I can make that pusher<T, has_begin_and_end<T>> struct, then internally I can use the traditional SFINAE already in the repository to differentiate between kinds of containers (e.g. with the has_key_value_pair and the like).

Most notably, however, this enables me to define the pusher<T, (SFINAE Stuff)> outside of the stack.hpp. The reason this is necessary is because in order to preserve type information, it looks like I have to go the route of userdata<T>. However, I don't want to keep all the type information: a container<T> specialized for certain traits of the underlying container T can provide a way to hook into the lua functionality (by overriding the meta table or whatever) but also preserve the original container's semantics. This might also mean refactoring get into something that mirrors pusher to also allow it to be overloaded based on SFINAE traits.

This is a long comment, but I think that explains what I'm trying to do. I'll get back to it!

[Rapptz]

I recommend not taking the route of the extra parameter. I already differentiate between the two different type of containers just fine using the EnableIf alias. You can already define pusher<T> outside of stack.hpp (I've done so before with relatively good success). Should be worth noting that pusher works just fine as it is right now without issue.

[princesstrash]

I don't quite follow...

Perhaps I didn't explain well, but my problem isn't necessarily defining pusher<T> outside of stack.hpp: you can do that with partial specialization just fine.

My issue was that I was unable to specialize pusher<T> outside of stack.hpp while also trying to make that specialization apply to the two types of containers. The reason I can't leave the two container specialization you're doing in the default struct pusher { ... }; definition you have is because I want to use userdata.hpp, and stack.hpp is included in userdata.hpp and I can't use userdata<T> until it's defined.

Because C++ doesn't allow things like partial class declarations, I can't go to the bottom of userdata.hpp and define pusher<T> with the sample template parameters are before and depend on the functions to internally EnableIf/DisableIf and pick...

Deepest apologies if I'm not explaining myself right. Or, if there's some other way to specialize pusher<T> for the two container types after userdata<T>, I'd love to know how to do that as well.

[princesstrash]

I have made progress and completed the implementation. I had to make an avalanche of changes. I hope you don't mind how big the pull request might turn out, but I'm at a bit of an impasse now.

The tests currently won't compile. The semantics of has_key_value_pair seem to disagree with how C++ (or how I) think about the container it comes from. std::vector<std::pair<...>> triggers the has_key_value_pair semantic, and while from the tests I understand this point of view I do not know how to appropriately map it to a std::vector<std::pair<...>> whose return type should be a pair and whose key value should always be an integer.

Nonetheless, while on my way I also managed to solve issue 25 by introducing a pointer-based component to all userdata<T>, to support references (which are stored as pointers and retrieved as pointers). You can make the system have lua permanently own the data by passing back a T&&/T (no reference/pointer) instead (this might need to be double-checked with container).

I don't want to make a pull request without figuring out how to make the tests here compile, specifically with regard to the way b.three and such are treated. Please let me know what I should do.

[princesstrash]

As a final note, std::function can be returned from a function now: I did this to balance the inequality of not being able to set an anonymous routine to something in lua (e.g. local a = cpp_call_makes_function()).

You can also push arbitrary functions using the pusher<function_t> type. I was considering potentially doing a SFINAE overload with pusher<T, IsFunction<T>> but I'm not sure if that's a too far-overreaching or overzealous overload (consider pusher<lua_CFunction>, albeit that's usually only used in lower-level places and perhaps the explicit overload should be enough to prevent the compiler from picking/choosing the wrong thing?).

[Rapptz]

Well. I'm looking over your code now. I've left comments (and I'm still commenting) in places where I think it's kind of weird/strange.

For some reason you're using pusher<T>{}.push syntax -- please note that the functions should be static instead. That way you can define pusher<T>::push for consistency with everything. Is there a particular reason why you chose not to do this?

[Rapptz]

In general, I like the changes so far. I don't particularly like the design of container myself. It seems too finicky. Your issue is most likely due to that. I could pull your fork and try to fix the issue myself if you'd like.

Edit:

I decided against fixing it myself -- it would make the std::vector<std::pair<...>> implementation have O(N) lookup instead of the old O(1) lookup it had in the lua side. So I'm pretty opposed to it as a result.

[princesstrash]

I use pusher<T>{}.push as a change to allow for any kind of pusher that might expect to be able to do things in its constructor before push is called. With most pusher<T> not having constructor, it is the same as static call anyhow (I still need to go through to change all functions I wrote to be static anyways, even using syntax). I was trying to follow the user of std::allocator from before, where the object is made and then construct() is called on it. It might just be me using wrong idioms anyways.

I understand not fixing it. I thought that maybe I could leave old push semantics if someone explicitly does push<sol::table>( my_paired_vector ), and keep the container semantics for the default case, but... I don't know. I'm not sure what to do with it, albeit I do like passing things in and then getting them out, as per this snippet:

#include <sol.hpp>
#include <vector>
#include <map>

std::vector<int> make() {
    return { 1, 2, 3, 4, 5 };
}

void take(std::vector<int> arf) {
    if(arf[0] != 10)
        throw 0;
}

std::map<int, int> make2() {
    return { {0, 1},
        {1, 2},
        {2, 3},
        {3, 4},
        {4, 5} };
}

void take2(std::map<int, int> arf) {
    if(arf[0] != 10)
        throw 0;
}

int main() {
    sol::state lua;
    lua.open_libraries(sol::lib::base);

    lua.set_function("make", make);
    lua.set_function("take", take);
    lua.set_function("make2", make2);
    lua.set_function("take2", take2);
    lua.script("a = make()\n"
               "a[0]=10\n"
               "take(a)\n"
               "print(#a)\n"
               "print(a[0])\n"
               "b = make2()\n"
               "b[0]=10\n"
               "take2(b)\n"
               "print(#b)\n"
               "print(b[0])\n"
    );

}

[Rapptz]

I don't believe sol::stack::pusher should have any state -- I haven't ran into a situation where it needs it. Though you're welcome to point me to a situation where this is desired.

As for your inquiry about enjoying the acceptance and creation of a certain standard container, I agree that this is indeed nice to have. However the implementation to support such semantics are definitely hack-ish. I don't believe the implementation as-is supports cases that worked before, i.e. std::vector<std::tuple<int, int, int>> where the result would be local a, b, c = stuff[1] etc. My reasoning was that std::tuple would be the equivalent of returning multiple values and std::pair would be a key-value pair similar to Lua's hash tables.

I don't believe that acceptance of a C++ container should result in a lower subset of features, it seems to me that this entire issue has caused issues of inconsistencies of how the rest of the code is handled and it'll be hard to fix. Also, leaving the old push semantics is pretty useless -- I don't think the end user will ever have a reason to go out of their way to get the old semantics that aren't even documented anywhere.

The original reason I didn't want to support having the second parameter for SFINAE with pusher was because allowing the user to specify generic parameters can be a bit troublesome. For example, suppose in my code that I have pusher<T, has_begin_end<T>> and in the user code someone has pusher<T, has_cbegin_and_cend<T>>. If a type is passed in that meets both of these requirements -- a compiler error will ensue. Something that I kind of want to avoid because now whenever I write additional constraints the user code might be polluted with compiler errors that are out of my reach.

With regards to your code, there are still some unexplained idiosyncrasies which I'm interested in knowing, such as preferring std::addressof(&str[0]) instead of str.c_str() and a couple more that I can't recall. I do think most of your changes are good though. I like them, keep it up.

[princesstrash]

I do std::addressof(str[0]) to get the first character of a string, as I pointed out in a response to your comment here. I saw std::addressof used elsewhere, and I've read it's better to do that for generic code, thought it was proper to do it that way. Albeit, std::addressof seems overkill when I could just do &str[0]...

I think at the moment have to stick by my current use of pusher<T, X>. If we throw out the changes I wrote for the hackish container<T> layer in the codebase... that will leave only partial specializations and explicit specializations for the pusher<T>. This leaves the library pure by perhaps extensible by someone who wants to replace X with some template strangeness they want. Of course, we don't have to afford the user this change and we can just leave it at pusher<typename T> and remove the extra typename.

As far as container goes, I gave it my best dance. In the end, I learned a lot, so I don't mind removing all the container-based stuff and sending a pull request. The function stuff and the userdata stuff alone I think are good and worthy contributions.

[Rapptz]

If we end up removing the container stuff, the issue of "What happens when someone passes in a container" is still there. The issue is very troublesome indeed. Creating arbitrary tables is helpful though. Arguably a little more helpful than accepting a container as a proxy to a lua table. There's only so much you can do with switching languages without resulting in type erasure. It's finicky and disappointing.

The plus side of having it back to back is that regular C++ code works seamlessly. This is a big plus and is consistent with nearly everything sol has. However from day 1 of implementing this I didn't expect to have perfect 1 to 1 translation. I think having to bind through sol::table isn't exactly too bad. If I added iterators to it (which was originally intended though proved to be impossible) it could have been incredibly similar to a C++ container.

[princesstrash]

Well, I think my current container implementation would actually work very well to make that happen... except in the case where something like std::vector<std::pair<>> becomes a key-value-pair rather than an indexed list containing a bunch of pairs that would do local a, b = paired_vector[1]. Either way, I've reverted the container changes for now and left everything else intact. All tests passed in my last compilation and run. I had fun helping out. Let me know if you want to think over trying to do C++ -> Lua -> C++ containers again.