Support for method templates (in declarations and in definitions).

[jll63]

Thanks!

Can you tell me what project(s) you use yomm2 in? Are any available as open source? I'm trying to make a list of projects to help promote the library.

OK, to your question...

Right now there is no (official) way of creating method and specialization templates, although it could be done by examining the code generated by declare_method and define_method.

It would make a lot of sense to support templates though. For example, the matrix used in the examples screams template.

There are a couple of difficulties.

1. Instantiating the templates

Suppose you have:

template<typename T>
declare_method(string, to_json, (virtual_<const matrix<T>&>));

template<typename T>
define_method(string, to_json, (const dense_matrix<T>& m)) {
    return "json for dense matrix...";
}

This syntax, by the way, would not fly. I'll come to that in point #2. Who will instantiate the required (template) specializations of the (method) specializations? At the very least this would require a post-link stage that would find all the instantiations of the base method, then re-run the compiler (a la Cfront) to instantiate the necessary method definition templates. Of course, if (when?) open methods are part of the language, the compiler takes care of that. This still doesn't work perfectly though: in presence of dynamic loading, the user will still have to help - as it is already needed for ordinary templates, that's one of the reasons why there is a syntax for explicitly instantiating a template.

2. Syntax

The syntax used above has no chance of working. It is because the declare/define macros generate several declarations and definitions. If you are interested, make a debug build of yomm2 then look for the *.ii files. It will show you how what the macros do. Probably you want to run clang-format on the files first, to preserve your sanity.

I think that the messiest the syntax can get is something like:

declare_method_template(
    ((typename, T)), 
    string, to_json, (virtual_<const matrix<T>&>));

define_method_template(
    ((typename, T)), 
    string, to_json, (const dense_matrix<T>& m)) {
    return "json for dense matrix...";
}

It might be that we can get away with just saying (typename T) (or (typename T1, typename T2), etc) but I doubt it, at least until I experiment a bit.

The reason why the messier syntax has good chances of working is that it allows extracting all the parts of the template parameters. Hmmm, but we also need to support variadic parameters and default values. So we may be dragged into something like:

    ((typename, , T, )) // template<typename T>

...so we can say, if needed:

    ((typename, , A, int), (typename, ..., B, )) // template<typename A = int, typename... B>

So the whole exercise here is to come up with ways of making the syntax lighter. For example, we could move the ellipsis and default value arguments to the end, so the example with ((typename, T)) works. But it would be at the cost of a less natural order when we do want either the ellipsis or the default

Oh, and it has to play ball with method containers.

I am going to work on this but it will take time. Unless you want to work on it?

[happycr]

Thank you for taking the time to reply and writing such a detailed response!

About my project:

I am currently working on creating a new programming language which is named c+++ (that name might change though). c+++ aims to be a high-level language which improves upon c++ by being simpler (removing low-level features inherited from c, simplifying syntax and more) and adding what I think are much needed features ( no manual memory management, support for runtime checks for undefined behaviour in debug mode, null safety....). I am using your library as I think open virtual methods are a killer feature which I plan to include in my programming language. It is not open source as my repository is a mess and is still under development, although I think I will make the source code publicly available once everything is cleaned up and there is a good foundation to add new features. Moreover I don't really know how open-source projects work as programming is still really a hobby for me ( I am 15 yo and only started programming 2 years ago).

About open template virtual functions:

I don't understand why open template virtual functions would need to be resolved at a post-link time? Your open methods can be overloaded (I think?), so couldn't the compiler generate overloads just like it does for normal template functions? I agree that this would be impossible for you ( as a library ) to do, but if open methods were part of the language would this be a suitable approach? Moreover, the language could take care of the syntax.

By the way, is there any proposal/plans for open methods to become part of c++ ? Do they have any chance to make it in c++23 ?

[jll63]

I am currently working on creating a new programming language which is named c+++ (that name might change though). c+++ aims to be a high-level language which improves upon c++ by being simpler (removing low-level features inherited from c, simplifying syntax and more) and adding what I think are much needed features ( no manual memory management, support for runtime checks for undefined behaviour in debug mode, null safety....).

Sounds a lot like D ;-)

I don't understand why open template virtual functions would need to be resolved at a post-link time? Your open methods can be overloaded (I think?),

Yes they can. The entry point / method dispatcher generated by declare_method is a normal inline function.

so couldn't the compiler generate overloads just like it does for normal template functions?

The method specializations (aka definitions) are not directly referenced. The compiler will see no reason to instantiate them. Even if the rest of your code does reference some of them (using method containers), the compiler will not instantiate the static constructors that registers them with the method. The method dispatcher will be instantiated, but it will not find any specializations.

I agree that this would be impossible for you ( as a library ) to do, but if open methods were part of the language would this be a suitable approach? Moreover, the language could take care of the syntax.

But you have to realize that what happens with template instantiation is all but trivial.

The initial implementation (in Cfront) worked like this: a linker wrapper first linked the program, then looked at undefined symbol errors (if any), and, if the missing symbols looked like they could come from a template, it picked a few translation units, compiled them again, while writing instructions in a file that told the compiler which templates to instantiate. The cycle would repeat until all the necessary instantiations were made, or no more symbols could be generated from templates.

Later a different approach was developed: instantiate everything as weak symbols, and let the linker discard all but one copy when the instantiation is referenced, and all of them if not. In the end the result is the same.

At one point there was yet another approach, that supported separate compilation of templates, but it was complex to implement and did not gain traction.

By the way, is there any proposal/plans for open methods to become part of c++ ? Do they have any chance to make it in c++23 ?

Stroustrup and co submitted a proposal in 2007. It was rejected, on the basis that the language was already too complex (remember the Vasa!) IIRC. Since then, the people who set the tune in the C++ community have become allergic to runtime polymorphism, especially when based on inheritance. Only compile time stuff is cool these days. And now language complexity does not seem to be an issue anymore. Go figure...

[happycr]

Sounds a lot like D ;-)

Yeah, it does. I find it's a shame that D is not doing really well and I think it's mainly because of its GC and poorer performance. Although my language will just be a simple preprocessor over c++ and thus will have complete compatibility with it and zero-overhead. Even if it doesn't work out it's a good learning experience and a good project to show to universities anyway.

The method specializations (aka definitions) are not directly referenced.

Really? How are they referenced? Isn't the inline method called directly and can't that be made a template?

But you have to realize that what happens with template instantiation is all but trivial.

Well, we have template functions, template classes, template methods, template variables, template lambdas... Now that compilers know how to handle templates, it can't be that hard for c++ to do template open virtual functions, if they are added to the language.

It was rejected, on the basis that the language was already too complex.

What a shame. In my opinion, one of the most needed features of c++ are extension methods, like in kotlin, which would go hand in hand with open virtual methods having only one virtual parameter.

have become allergic to runtime polymorphism, especially when based on inheritance

That's completely stupid. Runtime polymorphism is essential because the information is often only present at runtime, and inheritance is by far the best way I know to achieve this.

Only compile time stuff is cool these days.

Compile time stuff is pretty useless, in my opinion (although concepts are pretty cool). constexpr and consteval were not really made for performance, as compilers already had an optimisation for that: constant folding. Rather they are used when for non-type template parameters for template meta programming, which only experts use.

Anyway it was very nice talking to you; you're the first ever c++ programmer I've ever talked to! Also, should I close this issue and mark it as something else as it's not really an issue anymore? I'm really new to github so I don't really know how things work.

[jll63]

Let's keep this open as a place for discussing method templates. I labeled the "issue" as "enhancement", which still makes it look like an issue but I can live with it.

[jll63]

I am making good progress, cleaning up the internals, and moving things out the the detail namespace. I am pretty sure at this point that I will figure out a workable solution (albeit not fully automatic).

[jll63]

@derpda Are you still stuck with a very old version of Boost? I am experimenting with a solution for templatized methods, and in the process I am considering adopting Boost.Mp11, which appeared in Boost 1.68.0.

[derpda]

I'm leaving university by the end of this month, and will thus be off the project. I'll direct the younger student who is taking over here! For what it's worth, I don't think it environment has changed so we are probably still stuff with it.

[delta-leader]

Hi, I am the younger student that he mentioned. We've actually been talking about using templates in our library as well, but so far it has been disregarded because of the limited support by YOMM. So actually this feature would be very helpful to us as well. I asked the other members of the project and they said switching to a newer boost version should not be a problem.

[jll63]

Hello younger student-san ;-) Welcome to our small community.

We've actually been talking about using templates in our library as well

I am using a mock matrix library as a use case for my research on the matter. It is close to complete. Most of the work on the code is done. I have to finish the tutorial (still in flux) and write documentation for the newly exposed internals.

but so far it has been disregarded because of the limited support by YOMM. So actually this feature would be very helpful to us as well. I asked the other members of the project

It's great to have people try this, who actually know what they are talking about when they talk about matrices :-D

switching to a newer boost version should not be a problem

Great!

[delta-leader]

I am using a mock matrix library as a use case for my research on the matter. It is close to complete. Most of the work on the code is done. I have to finish the tutorial (still in flux) and write documentation for the newly exposed internals

That sounds great, thanks for sharing the tutorial, I am in the process of reading through. It seems like your scope goes far beyond our needs (since out library is limited by the availability of LAPACK/BLAS functions, so a "closed" approach probably works fine for us). I'll try to raise the topic at the next meeting again.

[jll63]

@happycr @delta-leader It was long in coming but finally here it is. If you are still around, let me know what you think. I have not released a new minor version yet, so things can still be changed at this point.