Feature modular/configurable runtime

[ParticlePeter]

This Feature idea is inspired by John Blows "programming language for games" and Vulkans" more responsibility for the developer". I will post more details about the idea in D forum at some time. There are people (like e.g. John Blow) who don't like/trust a GC language, especially for games. The idea (if feasible at all) is to configure the game project wide runtime to not use the GC but instead ref counted or plain new and delete allocators (preferably in code as pragmas). The later one would require project wide @system declaration as unsafe.

What do you think?

[Wallbraker]

The short answer is, yes I like it and something we should strive for. But will require a lot of work.

Just making sure you know Volt isn't D, :) and the D forums shouldn't be used to discuss Volt features. Tho if you phrase the question in the context of D its probably welcome. Also the question is welcome here just as long as it covers Volt. Thanks!

The long answer:

If you want to be sure that the GC isn't used in a project a flag to the compiler would probably be best (--no-gc. Project wide @system might be avoidable if reference counted classes and arrays are done correct. There might be places where you use @system to deal with manual memory management that you encapsulate in classes/structs.

We also need to make sure that Watt is buildable with that flag. Some functions will probably be removed when --no-gc is given. So we also need to make sure it has have enough functions left to be useful.

There are also some basic language designs changes we can do to make Volt more usable when no gc is present. Like changing array literals to be scoped (and stack/malloc allocations as such) by default and making gc allocation optin. Scoped [42, 64], gc allcoated new [42, 64]. Tho with --no-gc AA's becomes nearly impossible to do, tho it might be do able.

Cheers, Jakob.

[Wallbraker]

There is a lot more to it then just these few points that I brought up here. The design of the language, standard lib, compiler, buildsystem all play a part.

[ParticlePeter]

Nice to hear that :-) and yes I am aware of that this is not D. Today I tried glad the first time, noticed the possible output to Volt and got curious. I like your project, especially the break backwards compatibility part. What I meant was that I will present the same idea to the D community, but I doubt that the reception will be that positive.

The runtime configurability should not be limited to memory management, another example would be exceptions vs. implicit function error codes (also inspired by Jon Blow). Or the old virtual/final class discussions from d forums. You could end up with a lot of compiler switches if more options come up. But I must admit that pragmas are my personal favorite.

I am not so much of a language architect but what would be the problem in implementing DAs and AAs same way as std.container.array/rbtree or emsi containers dynamic array/hashmap?

[Wallbraker]

Okay excellent, nice to see Volt is getting some minor spread even tho we have done zero advertisements for it. Right then I understand let me know when you posted it, I want to follow it. Yeah I'm happy to that we choose to break backwards compat. If you have time you are welcome to help out with the compiler, stdlib and runtime in anyway you can, we are more the happy to answer questions. Also sorry for being late in getting back to you, had exams this week.

I'm not sure if we want more flags tho, gc is special. I'm of the opinion that if its not default people are not going to test it or write code for it. So none of the libraries for volt will work with any of the extra flags.

Can you expand a bit on exceptions vs implicit function error codes? While I really like C like error codes, exceptions are faster if they are not expected. We removed exceptions from the parser and it there was a real noticeable slowdown. That said there will be a @nothrow in the language. And we will probably add a way to group attributes. Like alias @My = @nothrow @nogc @unsafe; or a way to change the default on a per module basis. To turn it on a per module level module mygame.main : function = @nogc @nothrow; int main() { new int[](3); /* error main is per default @nogc */ } that way you could import libraries and use functions from them that does not allocate.

Dang as I write this I don't think we need a flag but instead can just use the per module declaration. This would work as well for final/virtual. module mygame.main : function = final @nogc @nothrow; There could be flags for enforcing that all modules given on the command line has functions set to @nogc and so on.

Please expand on what pragmas you mean?

Yeah AAs could be a library thing, AAs are in the language because we didn't have a good template strategy, I'm not sure if we want to keep them. DAs?

[ParticlePeter]

I was busy myself now, but now its time to comment. Meanwhile I had the chance to dive deeper into jai and must admit that I am heavily biased. The ideas we are discussing here are implemented there already to a much deeper extend I could have imagined. I still love D (and most probably volta as a derivative) but the innovations of Jai are quite unmatched.

Can you expand a bit on exceptions vs implicit function error codes?

I always felt that exceptions are kind of awkward in terms of breaking my code flow at any place. I guess Mr. Jon Blow did thoroughly explain what I feel.

And we will probably add a way to group attributes.

Very smart move, I like both your cases :-) alias @My is still useful as there can be multiple groupings of attributes.

Please expand on what pragmas you mean?

Again I would like to refer to jai. The jai compiler does include a build system and has basically no flags. All build settings are defined in source files. You CAN pass a function name to the compiler and it will run it at compile time to set detailed build options. I really like this approach as I don't like to memorize endless lists of compiler flags (mostly letters) and their meanings. I like human readable settings, or compiler pragmas in d, in source files. So if it were for my all dmd compiler flags should also exist as pragmas.

Yeah AAs could be a library thing, AAs are in the language because we didn't have a good template strategy, I'm not sure if we want to keep them. DAs?

DA = Dynamic Array. I meant something different, when a library implementation of some feature exists it is obviously possible that this feature can also exist as a language built in feature. In the c++ std there is/are implementations of AAs and DAs without relying on the GC, hence it must be possible to have such features built into the language without relying on the GC or RC memory management. Now you could argue that the GC based stuff is safer and more feature-rich, and I would argue that I don't care for the safety and additional features, I don't want to rely on the GC and I don't want it to be a part of the runtime. Can we both be satisfied? Yes we can if all paths (GC, RC, manual via new and delete) are implemented but only ONE of these options can and must be part of the runtime at the same time.

[Wallbraker]

Sorry for taking so long to get back to you.

Yeah I have heard about Jai before, there are some really good idea in it.

Can you expand a bit on exceptions vs implicit function error codes?

I always felt that exceptions are kind of awkward in terms of breaking my code flow at any place. I guess Mr. Jon Blow did thoroughly explain what I feel.

He has some very good points against exceptions, they do have a cost. Now the problem is that you are moving that brain cost into code loc and runtime cost. Its a reason why they are called zero-cost-exceptions (granted they are not really zero cost even when you don't run into them, but its small). I think Volt has a slightly larger target audience then Jai, tho I am a game programmer so I can relate a lot.

Please expand on what pragmas you mean?

Again I would like to refer to jai. The jai compiler does include a build system and has basically no flags....

Ah this feature I actually knew about before you brought it up. When I first saw the feature I went "This is brilliant I must do this for Volt", but as I was thinking about the feature the more its flaws came up. How do you deal with external dependencies, what if the file that has the build directive includes a module that includes a module that is in one of the libraries that needs to be downloaded, but the compiler can only do that if it can compile the build directive, but it can't since it can find all the included modules. So basically you would end up with a build.volt/jai file, that bootstraps the build process for you. And then the code just becomes a really bad DSL. So its a great idea for small projects or large ones where you the developer has all the dependencies installed. For large projects or new developers coming into the project its bound to lead to trouble.

That said, the Volt way would be that you would install battery and it will get all the dependencies, either by looking at the source or in the build.[volt|json|toml] file. Note that dependencies include the compiler it self. For the average code all you need to do to make battery build your project is:

$ battery

By default it will look for source in the current directory or in src/, it will create a executable if the source has a main function, else compile it as a library. It will name it to either the directory name or the only volt file name it can find. So running battery with a file name foo/src/main.volt from the foo directory will name the executable to foo. If main has a unresolved import like import thelib.pack.foo it will search the library repository for thelib, thelib.pack and thelib.pack.foo for it too download. If that library has a system library dependency it will tell you how to install it.

All of these default behavior can be overwritten by the build.json file, or by a user config. And for 99% of the case, you will not have to learn any flags or config file format. That said, learning what the name of the pragma is to do the thing you want is just as hard as learning a flag or a format.

Can we both be satisfied? Yes we can if all paths (GC, RC, manual via new and delete) are implemented but only ONE of these options can and must be part of the runtime at the same time.

Yes they can be satisfied, and I believe they can all live in the runtime but be dormat if you don't need them. For instance thinking about AA a bit more, those can be made to be @rc/malloc based if the key/value types are not gc based. So int[int] or int[rcstring] would not use the GC since it has no reference to hold on, that would be handled transparently by the compiler/runtime. And @nogc functions would not be able to use AAs that are GC based, so you would be guaranteed to not use the GC if you didn't want to.