Take a look at wren language and it's VM

[romix]

I've recently stumbled upon a new language/VM called wren:

https://github.com/munificent/wren

https://news.ycombinator.com/item?id=8826060

It seems to be pretty Lua-like, but with OOP support built-in. And it has its own bytecode-based VM (inspired by Lua VM?). The whole implementation seems to be pretty small, which is nice.

FWIW, Wren may have some interesting design decisions at the language level.

Plus there are some interesting implementation decisions, e.g. fixed class layout (no need to access fields as a hash keys), mark&sweep GC, etc.

[richardhundt]

Thanks again for a great heads up. There are a few gems in there. I really like the fibers and block syntax (although for loops and if statements need parens to disambiguate from map literals). All in all a nice little language. Also props for the computed goto dispatch. I only wish he'd gone with a register based VM.

The main takeaway for me, however, is that wren restricts class structure. I've just sunk in about 60 hours on a complete compiler redesign to support multiple passes needed for type checking, and I'm realising that I'm trying something close to intractable. Even though Shine has classes, it lacks well defined structure (because I can dynamically monkey patch the hell out of anything at any time), so figuring out what members a class has at compile-time is fragile.

So I'm really itching now to take the best from Shine and create a new language from scratch, which will be more structured, support optional or gradual typing - something like a mixture of Dart and wren, but less minimal (i.e. pattern matching, PEGs, and builtin support for defining FFI structs and bindings directly in the language). I no longer think it makes sense to retrofit static analysis to Shine because it'll need to change the character of the language entirely.

[romix]

Hi Richard,

You are welcome! I'm glad you found this useful!

Yes, I also thought that restricting class structure could be very useful both for efficiency and for type analysis, etc.

What I'm not so sure about is: if you map a restricted type structure to LuaJIT it would still use a table for representing it, or (unless you map it to C structs, which you probably don't want to do)? Or do you say that LuaJIT would optimize it in such a good way that it would become equivalent to using C-structs-like fixed layout?

[richardhundt]

Well, it'll still all map to tables underneath (and yep, LuaJIT has some respectable performance with tables). Once the compiler understands the structure though, it can (in some cases) avoid the dynamic dispatch overhead by binding to the method directly so you'd see Foo.__methods__.greet(foo, mesg) being output, instead of foo:greet(mesg). LuaJIT specialises metatables (it assumes that they don't change once assigned), so that particular optimisation is not needed.

More interesting though, is knowing at compile time if the object supports the method I'm calling on it at all, or if it has a does_not_understand or noSuchMethod hook to fall back to.

That's really hard if you can do this sort of thing:

-- create a base class dynamically
local function gimme(str)
   class C
      mesg()
         return str
      end
   end
   return C
end

-- make the method table hard for the compiler to grok
local vtable = { greet = function(self) print("Hi") end }

class Foo extends gimme_base_class("cheese")
   for k, v in pairs(vtable) do
      self.__members__[k] = v
   end
end

If classes didn't let you run arbitrary code in their bodies, inherit from expressions, and get created at runtime in various places, then things get much easier. It'll never be a completely static language, so if you try hard enough you'd be able to break open a class and add something at run-time. All I'm really trying to do at this stage is catch typos and enforce method arity, and with code like the above it gets tricky.

I mentioned structs earlier. I was thinking of adding a struct declaration (in addition to classes), which would make the ffi.cdef and ffi.metatype calls for you. It'll just be sugar. Not sure if that confused the issue for you or not.

[romix]

The main takeaway for me, however, is that wren restricts class structure. I've just sunk in about 60 hours on a complete compiler redesign to support multiple passes needed for type checking, and I'm realising that I'm trying something close to intractable. Even though Shine has classes, it lacks well defined structure (because I can dynamically monkey patch the hell out of anything at any time), so figuring out what members a class has at compile-time is fragile.

It would be still interesting to see your work in progress. Could you create a branch and put your code there, even if it is not (and never will be?) finished?

[romix]

I mentioned structs earlier. I was thinking of adding a struct declaration (in addition to classes), which would make the ffi.cdef and ffi.metatype calls for you. It'll just be sugar. Not sure if that confused the issue for you or not.

Ah. I overlooked this when I read your first message. Yes, that would be nice.

But it would be a one-way ticket, or? I mean structs cannot refer to Lua/Shine objects, or? So, you cannot really mix and match, right? So, ffi-structs would be leaves of Lua/Shine object graphs. It is not bad, but I'd like more flexibility (may be adding GC-like tracing methods so that Lua can iterate/traverse over pointers inside ffi-structs even though it does not know the meaning)?

Overall, it is probably not a problem unless you want to right a very high-performance code, where you want to pack/place your data optimally in memory. For example in Java many people use Unsafe or off-heap memory to achieve this. But this comes at a cost. You have a full control over a memory layout, but you loose a possibility to work with those in-memory structures as if they are usual objects. You need to access them in a special way. It would be cool if it would be possible in Shine to hide this kind of problems behind a facade and have a full control over memory placement, while still working with any such structures as if they are usual objects.

Another related thing: If you want to have an array of structs or structs of structs of structs, it would end up with a table containing a table containing a table in Lua. While this works, it can be pretty memory inefficient and also produce a lot of pressure on GC. So, having more opportunities over object memory layouts would be beneficial here as well. But this is probably more of a question towards LuaJIT (or any other VM which is used by the backend) and not the language itself. For example, Wren has probably more control over a memory layout of its objects as it has a VM which was designed with that in mind.

Yet another thing, which is also related to those remarks about memory: It would be interesting if it would be possible to (explicitly) place local variables on stack, at least in some cases. I guess LuaJIT performs this kind of optimizations already (something like on-stack replacement or scalar replacement of aggregates), but I don't know how far it goes in that direction.

Obviously, all those remarks are more about high-performance kind of apps. It is not so important for scripting or for less demanding applications.

[romix]

BTW, regarding register-based VM vs stack-based VM, the author of wren provides his view on that: https://github.com/munificent/wren/issues/82

[richardhundt]

But it would be a one-way ticket, or? I mean structs cannot refer to Lua/Shine objects, or? So, you cannot really mix and match, right? So, ffi-structs would be leaves of Lua/Shine object graphs. It is not bad, but I'd like more flexibility (may be adding GC-like tracing methods so that Lua can iterate/traverse over pointers inside ffi-structs even though it does not know the meaning)?

Yep, structs will be in C space, so can't safely hold pointers to Lua data. The other way 'round is okay though (obviously). The GC is the issue, of course. Tracing hooks are basically limited to setting up an ffi.gc hook on the root of the struct graph and using that to traverse, but you'd end up building your own GC on top (reference counting probably).

Overall, it is probably not a problem unless you want to right a very high-performance code, where you want to pack/place your data optimally in memory. For example in Java many people use Unsafe or off-heap memory to achieve this. But this comes at a cost. You have a full control over a memory layout, but you loose a possibility to work with those in-memory structures as if they are usual objects. You need to access them in a special way. It would be cool if it would be possible in Shine to hide this kind of problems behind a facade and have a full control over memory placement, while still working with any such structures as if they are usual objects.

Not sure if you'd ever be able to encapsulate CData so that it's completely transparent. What can be made nicer, though, is the ability to add methods and constructors to structs so that they can be used like object instances in many cases. I'm thinking along the following lines:

struct List<T> {
   tail: List[]
   data: T
   self(data: T, tail: List<T>?) {
      self.data = data
      if (tail != nil) {
         self.tail = tail
      }
   }
   cons(item: T) => new List<T>(item, self)
}
let list = new List<char[]>("Hello" as char[])
list = list.cons("World!" as char[])

You'd still need to be aware that you can get segfaults and all the rest, so it'll be no hand holding. The motivation is that you get a consistent way of declaring your structs which lets you decorate plain Lua data (functions, etc.) onto the cdata. Also, since the List above remains a Lua table (it's constructor allocates the underlying cdata), it can have static methods, and can be inherited or mixed in.

Another related thing: If you want to have an array of structs or structs of structs of structs, it would end up with a table containing a table containing a table in Lua. While this works, it can be pretty memory inefficient and also produce a lot of pressure on GC. So, having more opportunities over object memory layouts would be beneficial here as well. But this is probably more of a question towards LuaJIT (or any other VM which is used by the backend) and not the language itself. For example, Wren has probably more control over a memory layout of its objects as it has a VM which was designed with that in mind.

Sure. Once you malloc your cdata, you're off the LuaJIT heap, so you can do whatever you want for fine grained control and efficiency.

Yet another thing, which is also related to those remarks about memory: It would be interesting if it would be possible to (explicitly) place local variables on stack, at least in some cases. I guess LuaJIT performs this kind of optimizations already (something like on-stack replacement or scalar replacement of aggregates), but I don't know how far it goes in that direction.

There's no C stack in Lua. The JIT machinery might implement something similar, but what would you do if JIT was off or you had a fallback to interpreter mode (trace abort)? Locals are in Lua's activation record (the interpreter's equivalent of the C stack), so just by saying local foo you're explicitly putting it on the "stack" (which may be stored in a CPU register by LuaJIT). The thing is that these stack slots are GC roots, so in this sense all values are on the "heap" from C's perspective. I can't see you ever being able to store a value on the C stack from within Lua. You're dealing with two separate abstraction layers.

[romix]

Richard, all your comments make sense. And we are in agreement.

I was speaking about what I'd like to see, if it would be possible. I realize that there are limitations which cannot be ignored. Getting away those limitations would require a new VM or changes to LuaJIT.

But coming back to your examples. Some quick comments:

• The minor one: please, don't use new syntax. Use the name of a class/struct to construct instances, if possible.

Tracing hooks are basically limited to setting up an ffi.gc hook on the root of the struct graph and using that to traverse, but you'd end up building your own GC on top (reference counting probably).

Building your own GC (or even better providing a few different GC implementations as a library) is not such a bad idea, IMHO. The tracing functions (or some kind of GC-roots metadata to be used by tracing functions) could be generated automatically by the compiler, because it knows the structure of your data types. Overall, having some kind of a pluggable GC would cool. One could use ref-counting for some parts and may be other flavours of GC for other parts, depending on requirements.

One could also consider a use-case, where FFI structs are used by the compiler as a pure optimization, e.g. for better memory layout and quicker access to fields. They would not be supposed to be manipulated by any custom C function. Thus, compiler would know the layout and types of all fields and can access them more efficiently, plus GC can trace them without any big problems (as data is only manipulated by Shine/LuaJIT).

There's no C stack in Lua. The JIT machinery might implement something similar, but what would you do if JIT was off or you had a fallback to interpreter mode (trace abort)?

Yes, I realize that, of course. But I'm always assuming JIT when I speak about those features. ( To be frank, I' not sure that a new language on top of Lua is likely to get any traction if it will be used without LuaJIT. But this is another discussion. Let me comment on this later).

In ideal world, I'd like to be able to map almost a complete C to LuaJIT. In particular, I'd like to be able to map memory-related parts of C to LuaJIT, i.e. the ability to allocate objects (e.g. scalars or structs or arrays) on stack. Obviously, it may turn out that this would imply certain limitations when it comes to interaction with Lua (e.g. no pointers to Lua objects inside those objects, etc).

Coming back to my comment regarding getting a traction for a new language. Here are my personal opinions. Please take it with a grain of salt:

There is a bunch of new languages out there, some with a good backing from organizations, corporations, etc. But even they have troubles getting any traction. More over, many of new modern languages are very similar in their semantics and overlap to 80% or more in their feature set.

IMHO, to become more popular, a language needs to have a few unique selling points that are worse switching to it.

• If those unique selling points are only about a syntactic sugar (e.g. PEGs, pattern matching, type system and the like), then it is not very likely that someone would jump on to this new language, because most of these things can be done on top of many popular languages either as a library or as a relatively simple preprocessor, which would map those constructs to the base language. This way people still stay with their popular language, its libraries, its ecosystem, but are able to use new features.
• But if the unique selling point is the implementation of the language, e.g. its ultra-fast execution, its VM, its ability to interface with low-level C-libs via FFI, then this is something that cannot be easily provided by mapping to an existing popular language. Then people have an incentive to switch to this new language, because otherwise it is just impossible to use those features. Of course, the features should be very convincing, otherwise people would not even try.
• IMHO, LuaJIT seems to be such a killer-feature, a unique selling point. It allows you to sketch a language, which is as dynamic as Lua or as expressive as ... you name it, but having a very, very good performance, almost zero-time startup times, small memory footprint, coroutines, and easily embeddable into any C/C++ app. This niche is not filled yet, IMHO. So, there are chances there. But this means that one should actually make use of LuaJIT as much as possible and try to squeeze as much as it goes from it. It may also mean extending LuaJIT (a bit) to add some killer features that currently cannot be expressed by means of LuaJIT or are not efficiently handled by it.

It is actually an interesting question, Richard. How do you position Shine (or its follower)? What are the unique selling points from your perspective if you want to convince someone to start using it? Is it more about expressiveness or is it more about its speed and size? Or may be something else? I think it is worse thinking about it, because it may help to focus on essential features. It would also help potential users to understand what they can expect from the language and how it is different from other languages.

[richardhundt]

The minor one: please, don't use new syntax. Use the name of a class/struct to construct instances, if possible.

I'll need to give this some thought. The case for new currently is that it's a special case to support generics. I hadn't planned to have generic methods or functions initially (you'd need type inference for that), and the compiler can't tell that Foo<T>() isn't a function call. Check out this Dart thread for more: https://groups.google.com/a/dartlang.org/forum/#!topic/misc/uHPi5fM9sLQ

It is actually an interesting question, Richard. How do you position Shine (or its follower)? What are the unique selling points from your perspective if you want to convince someone to start using it? Is it more about expressiveness or is it more about its speed and size? Or may be something else? I think it is worse thinking about it, because it may help to focus on essential features. It would also help potential users to understand what they can expect from the language and how it is different from other languages.

That is an excellent question - yeah, folks say that when the don't have an answer ;-)

These are the things which LuaJIT gives me which are great selling points:

• Blazing performance
• Tail-call optimisation - functional style is cheap and fast
• Coroutines - as generators and a core abstraction for streams, pipelines etc.
• Seamless FFI
• Small footprint

There's no other VM which gives you all of the above in one package. Just the tail-call optimizations are a boon (consider Scala's limitations with the @tailrec annotation because of the JVM), so hell, you've got to be able to leverage all of that in a language not only geared to embedding.

Let's start with the assumption that there's no such thing as a general purpose programming language. So my pitch would be that "language X is a dynamic systems programming language".

In other words, I want:

• to do all my C programming in a higher level, dynamic language.
• the process calculi abstractions of Go and friends (CSP)
• to support a pure functional style
• object oriented
• optional typing

I'd say the closest competitors would be Go and Rust in terms of use cases, just a more dynamic version with more familiar syntax. Shine, was really just for Lua hackers who wanted continue, classes and a few other bells and whistles. There would also need to be excellent documentation.

[romix]

Thanks for your answer! I think that we have a pretty much the same view on the main features and goals of the language.

When it comes to the "dynamic systems programming language", I think that eventually more control over memory allocation, garbage collection and memory layout would be required for the "systems" part. LuaJIT already provides quite a lot, but I could imagine some extensions. I've outlined some of them in previous messages.

I'd say the closest competitors would be Go and Rust in terms of use cases

I think that Apple's Swift could also be named in this category. It is also a compiled language, with generics, with support for functional style, object-oriented and has typing similar to Scala, i.e. in most cases you do not need to explicitly type your variables as their type can be inferred from the context. D is also probably rather related.

The biggest different of all those languages from yours - they really have a compilation step and produce object files. Your language uses JIT. It aims to become more or less a JIT for C/C++ (with high-level extensions and more dynamism).

( BTW, it would be pretty cool to have a JITted C. I.e. if LuaJIT could really support pure C. It would make a very interesting target for language/compiler backends. In fact, it would most likely allow Go, Rust and others to be mapped to LuaJIT.)

Actually, when I think more about it, the "dynamic" part should be worked out more precisely. Without it, the new language becomes (sort of) just a copy of Rust, Go and others. It would be nice to provide a more concrete and clear motivation and examples highlighting the benefits of the "dynamic" part. Some use-cases which would be more difficult or almost impossible to implement using other languages.

[richardhundt]

When it comes to the "dynamic systems programming language", I think that eventually more control over memory allocation, garbage collection and memory layout would be required for the "systems" part. LuaJIT already provides quite a lot, but I could imagine some extensions. I've outlined some of them in previous messages.

I guess I don't really understand what you mean by more control over memory placement and so on. What is it you can do in C, in terms of memory layout, that you can't do with the FFI?. For memory control and layout, you can call brk(), sbrk(), mmap() etc, you can use bit fields, align and pack structs, pun types (I've got a pure Lua + FFI implementation of the Linux kernel's container_of macro, for example, which given a pointer to a nested struct, and a field name, casts to NULL, and uses offsetof to get the negative byte offset of the containing struct, fetches that memory address and casts it to the type you want). You can do reflection on FFI structs (using Peter Cawley's ffi-reflect module or LuaJIT 2.1 ffi.typeinfo call), which is something you cannot do in C. You can choose your own malloc implementation. As far as I can tell, you have complete control. What am I missing?

To me a systems programming language is something I can write a bootloader in, or rootkit my mac with :-) Sure, you'd need a C wrapper with a main to fire up a lua_State and link the LuaJIT VM statically, but most of it can be written in Lua, so transitively, by a language which compiles to Lua.

I think that Apple's Swift could also be named in this category. It is also a compiled language, with generics, with support for functional style, object-oriented and has typing similar to Scala, i.e. in most cases you do not need to explicitly type your variables as their type can be inferred from the context. D is also probably rather related.

Sure, they're also in the "systems" space, however Swift and D lack the concurrency and supporting memory models I'm after. You probably know all this, but I'm highlighting it again because it really matters to me in a language.

The biggest different of all those languages from yours - they really have a compilation step and produce object files. Your language uses JIT. It aims to become more or less a JIT for C/C++ (with high-level extensions and more dynamism).

Well, more accurately, Shine can produce native binaries (it's a LuaJIT feature, of course):

richard$ shinec -n "foo" foo.shn foo.o
richard$ file foo.o
foo.o: Mach-O 64-bit object x86_64
richard$ nm -a foo.o
0000000000000000 D _luaJIT_BC_foo

They just have the bytecode embedded, but can be freely linked with other C static or shared libs to produce shared objects, or executables. You just need the C entry point in the case of a standalone executable, but it's pure boiler plate and can be auto-generated.

So that's kinda nice - you can create an ELF binary of your entire application, with no external libraries which need to be installed (not even LuaJIT needs to be available on the target), and deploy it in one shot. That's what the shinec and shine binaries are, and even with the entire compiler chain, runtime bytecode embedded, and TvmJIT linked in statically, the shine binary is still only ~730k (on my machine).

You can have it both ways, so while you're developing, have it run the usual way, from your checked-out working copy, by feeding sources or bytecode to LuaJIT, with zero impact on performance, and no code changes to make it work. I guess that's another selling point (again inherited from LuaJIT, but could be maximised with a nice build tool which knows how to generate the C wrappers and the whole linker symbol export hand waving).

Actually, when I think more about it, the "dynamic" part should be worked out more precisely. Without it, the new language becomes (sort of) just a copy of Rust, Go and others. It would be nice to provide a more concrete and clear motivation and examples highlighting the benefits of the "dynamic" part. Some use-cases which would be more difficult or almost impossible to implement using other languages.

I'm not sure if this is worth doing. Perhaps. Dynamism is really just a set of language characteristics. The benefits (and pitfalls) would be the same as those of other dynamic languages. Compilation is typically fast. Types are tagged and available at runtime. Dispatch is dynamic. Binding is late. You can do more meta-programming with hooks like noSuchMethod to build dynamic proxies/delegates, without something like IDL compilation steps, and you get a quick development cycle. On the down side, code can be harder to reason about because of runtime monkey patching, eval(), etc.

My main problem with statically strongly typed languages is that they reject programs which are obviously correct, but don't pass the type checker. This sort of thing irritates me:

// Entering paste mode (ctrl-D to finish)

class Mammal { }
class Cow extends Mammal { }

var cow = new Cow
var mammal: Mammal = cow
cow = mammal

// Exiting paste mode, now interpreting.

<console>:12: error: type mismatch;
 found   : Mammal
 required: Cow
       cow = mammal
             ^

I mean, I know, trivially, that mammal holds a Cow, I put it there two lines before. The program is correct. But all this is a debate which is orthogonal to any given language.

[romix]

I guess I don't really understand what you mean by more control over memory placement and so on. What is it you can do in C, in terms of memory layout, that you can't do with the FFI?

Yes, you can do almost everything by means of FFI. But you'd do it sort of "outside" of your language, in the FFI land (i.e. in the C-land). In certain cases, you cannot continue using your objects in the same way as before e.g. if they are placed (as C structs) in your malloc-allocated areas. Imagine an array of C++ classes. Each element is an instance of a class (or pointer to the instance) and each instance has its own vtable for dispatching calls. This vtable-pointer is stored inside the object inside the same memory area, right? And with LuaJIT/Cdata, your object in typically split in two parts: one for holding the fields (the FFI struct) and one for holding the analog of a vtable (i.e. cdata with meta methods), which are stored in different parts of memory: the first is in the memory region you provided, the second one - in Lua heap. Is it correct? Also, if you'd look at the memory layout of multi-level nested structs, etc, where each struct should have its own methods (thus modelling classes), the layout would be pretty different from a contiguous one used by C/C++, because cdata + meta methods need to be stored somewhere on Lua heap, right? Not only the layout is different, the amount of required memory allocations (and deallocations or GC) is eventually rather different. There could be more examples along those lines.

So, what I'm trying to say, is:

• Purely at the data layout level, you can use FFI to describe any memory layout and access it. Which means you can do whatever you want with the memory as you do in C.
• As soon as you try to work with your objects described by means of FFI, you eventually have certain limitations. In particular, if you want "virtual methods" a-la C++, LuaJIT would also involve certain allocations on the Lua heap space for cdata/userdata pointers and meta-tables. And you cannot easily change this part, AFAIK. This would require implementing your own method dispatching mechanism, etc.
• All this is probably not too critical and can be worked around in most cases. It is mostly an inconvenience, rather than a serious impossibility of doing something. But having a "syntax sugar" or mechanisms to hide those inconveniences would be definitely a big plus. Because overall, it would be nice to have a uniform language, where you do not need to use one subset of features to work with regular programs and another one (e.g. FFI C-like syntax) to work with "system"-like programs.

To me a systems programming language is something I can write a bootloader in, or rootkit my mac with :-)

Fair enough ;-) To me, it is also somethng I can write a memory allocator, a driver, a process dispatcher in, i.e. I can write an OS ;-)

Dynamism is really just a set of language characteristics. The benefits (and pitfalls) would be the same as those of other dynamic languages. Compilation is typically fast. Types are tagged and available at runtime. Dispatch is dynamic. Binding is late. You can do more meta-programming with hooks like noSuchMethod to build dynamic proxies/delegates, without something like IDL compilation steps, and you get a quick development cycle. On the down side, code can be harder to reason about because of runtime monkey patching, eval(), etc.

OK. We have a small misunderstanding here. I actually meant something different when I mentioned dynamism.

But let's first start with what you mentioned here. First of all, I sign under each of those features ;-) And yes, this is what most people understand under a "dynamic language". I used a wrong word in my comment (see my explanation later). But even this part should be highlighted/explained to systems people, so that they get it. Many of them have not used dynamic languages extensively or at all. Things, opportunities, programming styles used in dynamic languages are obvious for you and me, but not necessarily to people who never used them. So, they need to see the advantages and possibilities first to get convinced. So, basically everything you mentioned needs to be listed and provided with convincing examples (plus references to other dynamic languages and their docs, because there are a lot of descriptions and explanations already in place).

Coming back to what I had in mind. I used the word "dynamism" to refer to the opportunity to create/replace code including data types at run-time, to "compile" at runtime, to extend the system at run-time and to work with this new code, types and data as if they existed right from the beginning. And all that using the same language and the same mechanisms. More over, due to the fact that we use a (tracing) JIT compiler, all this is possible at virtually no run-time cost for "compiling" the code and even more important, almost at no overhead for executing this new dynamically created/loaded/modified code. This is something that is not supported by any mainstream systems language, AFAIK. All other features mentioned in the previous paragraph are supported by most dynamic languages. But this efficiency of changing/extending the system that we mention here, this is something outstanding. This is probably one of the unique selling points. It potentially opens up a lot of opportunities. It eventually changes the style how you write your systems. The analogy would be Erlang or Lua coroutines: If you don't have Erlang processes or Lua coroutines, you are afraid that creation of threads is very expensive. So, you try to create just a few threads and then use locks and other mechanisms to write your systems. It becomes complex and difficult to understand. With Erlang/Lua you just say "OK, let's start yet another process/coroutine" because you know that you can have millions of them. And suddenly many things start looking much simpler. This is what I mean by "It eventually changes the style how you write your systems.". I think there is a lot of potential there which is probably not covered by existing dynamic languages yet (because most of them are not too efficient).

Yes another thing related to efficiency and dynamic, which you don't see that often (may be only in JVM HotSpot and some JS run-times), is the fact that tracing JITs often produce a better code than any static compilers, simply because they know what data is used at run-time and optimize/specialize the code accordingly. So, not only you are more dynamic, but you are also more efficient at the same time in many cases. This is another great selling point, IMHO.

Thus, when I wrote "Actually, when I think more about it, the "dynamic" part should be worked out more precisely.", I meant that these features should be presented and highlighted more explicitly and with convincing examples. I think they allow for something very powerful that cannot be (easily) expressed by most traditional languages.

[richardhundt]

So, what I'm trying to say, is:

Ah, okay I think I'm with you now... you're after a consistent object layout, regardless of whether it's cdata or table based objects. So for example, one could say that all objects are array-like, have a known length and a header which keeps a link to the vtable and field names. Something like (pretty standard stuff):

 +--------+---------+---------+---------+
 | Header | field 1 |   ...   | field N | 
 +---+----+---------+---------+---------+
     |   
     +--> {
       fields  = { x = 1, y = 2 },
       vtable = { foo = function(...) end }
     }

Then to make cdata work, a pointer to a struct would point to the start of the field data segment - after the header. To get at the at the header you subtract the header size from the pointer. That way you can pass your pointers into C space without leaking the header, but the header will be available in Lua space? The header field sounds a exactly like metatables to me, but the array layout for better space usage and cache locality may be worth doing. The cost is an extra indirection to get the field offset from the field name.

Some things about cdata types are inconsistent with other types, though. You can't overload + and - operators on cdata pointers, and you can't add methods to uint32_t or other primitives (__index isn't overridable). These may be more of an issue to solve.

On the extreme end of the spectrum, you could bypass all of Lua's types, and not use any of the arithmetic, length and indexing operations, and not a single Lua table in sight. What you'd do is to use only conditional/unconditional branches, call and return. You'd keep a C library handle as an upvalue which implements your ops, and everything ends up calling into that. As an example, a for loop from 1 to 10 might look like this:

local i = op.loadint(0)
local _1 = op.loadint(10)
local _2 = op.loadint(1)
::loop_top::
if op.is_false(op.lt(i, _1)) then
   goto loop_end
end
i = op.add(i, _2)
goto loop_top
::loop_end::

This would give you complete control over the data types and operations which can be performed on them. You could go even further and not use the Lua stack for variables defined in the language, so each op call takes an activation record as a first argument. Of course, you'd need to build your own GC (or use an off the shelf one), but you'd divorce yourself entirely from Lua's heap. I'm willing to bet though that the friction would be significant as you're giving up a lot of the JIT potential.

But as I said, that's an extreme. This is what I'd try first to make things consistent:

• Every . operation is a method call - this means that foo.bar compiles to foo:__get_bar() and foo.bar = 42 compiles to foo:__set_bar(42). Then foo.bar(42) either remains foo:bar(42), or becomes foo:bar_1(42) to encode the arity.
• Same applies to foo["x"] and foo["x"]=42 - these compile to foo:__getindex("x") and foo:__setindex("x", 42)
• Then cdata structs would all have a metatable, and the above accessors and mutators would be generated for you by the compiler to wrap field access. This will essentially let you overload __index.

The problem with all this name mangling is how to call into C and vanilla Lua. I can't pretend that C.malloc(size) is C:malloc_1(size) underneath (wrong symbol, and it's not a method on C). So we'd introduce a native keyword, which declares a C function, external for Lua bindings:

native malloc(size: size_t): void[] // means ffi.cdef"void* malloc(size_t);"
external getmetatable(o: any): table

The compiler then groks these and omits the mangling and gets type information as a bonus. Additionally you'd need foo::bar/foo::["x"] which translate to foo.bar and foo["x"] internally, giving you direct field access for low level stuff.

Thus, when I wrote "Actually, when I think more about it, the "dynamic" part should be worked out more precisely.", I meant that these features should be presented and highlighted more explicitly and with convincing examples. I think they allow for something very powerful that cannot be (easily) expressed by most traditional languages.

Okay, I'm reading that as: "we need a language specification". I think that's what it boils down to. To give you an example, we might say that a class defined as dynamic class Foo { } doesn't participate in structural type checking because it is assumed to be extended at run-time, whereas instances of class Bar { } will raise an error if accessed incorrectly. Perhaps static class Baz { } is available at compile time?

I have some more ideas, but I think I need to spend more time hacking than talking about it. Speaking of which, I've set up a project:

https://github.com/richardhundt/blaze

It's obviously early stages - I'm still on the definer to checker passes and working on the supporting meta-model. Don't even bother looking at builder yet, that's still the old Shine code.

[richardhundt]

This silly micro benchmark actually runs faster with boxed int64_t than using Lua numbers (and it includes an allocation during __add:

local op = { }
local ffi = require('ffi')
local int64 = ffi.typeof('struct { int64_t _; }')
local int64_mt = { } 
function int64_mt.__lt(a, b)
   return a._ < b._ 
end
function int64_mt.__add(a, b)
   return int64(a._ + b._) -- allocation!!!
end
function int64_mt.__tostring(a)
   return tostring(a._)
end

ffi.metatype(int64, int64_mt)

function op.loadint(n)
   return int64(n)
end
function op.is_false(b)
   return not b
end
function op.lt(a, b)
   return a < b 
end
function op.add(a, b)
   return a + b 
end

local i = op.loadint(0)
local _1 = op.loadint(1e9)
local _2 = op.loadint(1)
::loop_top::
if op.is_false(op.lt(i, _1)) then
   goto loop_end
end
i = op.add(i, _2) 
goto loop_top
::loop_end::
print(i)

Here's the output (3 GHz Intel Core i7, MacBook Pro 13-inch)... that's 0.3 nanoseconds / iteration.

richard$ time luajit ops.lua 
1000000000LL

real    0m0.314s
user    0m0.306s
sys 0m0.005s

interesting....

[romix]

Regarding the benchmark: LuaJIT is too good at copy propagation & the like. I think it propagates most of your constants and completely eliminates all allocations. Try to rewrite this benchmark in such a way that initial values for i, _1, _2 cannot be derived by LuaJIT from the source. Read them from standard input or pass them as command line arguments. I'm wondering if results will be the same afterwards ;-)

[richardhundt]

Nah, it's allocation sinking. Running it with -O-sink increases the running time to 29 seconds. It makes sense that the GC dominates the running time. I've been holding my breath for the new GC, but not sure if Mike has found a sponsor for it yet.

[romix]

Ah, it is allocation sinking! Interesting...

[romix]

BTW, may be Terra worth a look when thinking about the new language: https://github.com/zdevito/terra ?

It seems to have a few ideas about mixing Lua-layer with low-level C/LLVM-layer, Terra types as Lua values, meta programming tricks (quotes, escapes, etc), support for basic types... May be there is something interesting there.

[romix]

Hi Richard, what's up with Blaze? ;-) Too busy with other things?

[richardhundt]

Yeah, work, family life... I'm poking at it on the train on my way to and from work, hoping to do some coding on Sunday.

On Fri, Jan 30, 2015 at 9:12 PM, romix notifications@github.com wrote:

Hi Richard, what's up with Blaze? ;-) Too busy with other things?

— Reply to this email directly or view it on GitHub https://github.com/richardhundt/shine/issues/71#issuecomment-72263079.

People who think they know everything are a great annoyance to those of us who do.