Some issues with arrays

[romix]

• Some operations on arrays seem to have a strange semantics:

a4 = [ [1,2] , [3,4]]
print a4 

results in 1 2 3 4. So either array is formed in a wrong way or printing of arrays is not quite correct

• Another thing I detected:

nyanga> a1 = [1,2,3,4,5,6,7,8,9]
nyanga> print a1.slice(1,3)
[string "nyanga.core"]:0: attempt to perform arithmetic on global 'i' (a nil value)

• Shouldn't arrays support slices by means of indexing using ranges, just like strings do? a[1,10] could produce a slice of an array
• Would it make sense to have a concatenation operator for arrays? [1,2,3]~[4,5,6]should produce [1,2,3,4,5,6]. And ~= operator should change its left operand accordingly.

P.S. BTW, Richard, have you seen this language called Croc? It is a mixture of Lua, D, Squirrel, etc: http://jfbillingsley.com/croc/wiki/Lang

Overall, it is rather similar to Nyanga. There could be some interesting things there.

• For example, the idea of decorators could be interesting.
• Also meta-programming and reflection is useful. Does Nyanga support metaprogramming or reflection right now? Can I e.g. get information about all (static) methods of a class, all (static) variables of a class, etc?

[richardhundt]

just to clarify, the reason for the previous post is to show how to get hold of variable names for which I think macros are better suited since they're compile time entities.

[romix]

Thanks for the macro-example. Nice!

But it would require to use a custom way to declare things (e.g. using let) if I want them to be intercepted, or? The nice thing about local variables declarations decorators (+ eventually some AOP) is that I can catch some/any of local declarations without explicitly marking each of them in source code by hand...

Think about it like this:

• In some cases, you want to explicitly mark/annotate/decorate certain places in your code, because you know at design-time that something special should happen there. This is what explicit decorators are good for.
• Sometimes you don't know in advance which places are of interest. But later on you (or someone else using your code) decides that more places should be intercepted/manipulated. So, either she needs to change your source code and mark more places (which changes your code and can be a pain if it is a 3rd party code) or she uses some sort of AOP-like approach (which can be both compile-time or run-time based) and simply tells "Now I'm interested in more places, namely in all those satisfying XYZ", where XYZ can be "all local variables declarations", "all variables whose name starts with i", "all fields of classes, which are ..." and so on.

So, in one case you mark interesting places "from inside" and in the other "from outside". Both approaches are complementary, IMHO. I tend to think about it a bit like about set definitions. Sometimes it is easier to enumerate set elements explicitly by hand (if the number of elements is small), but sometimes it is easier to define a set via its complement, i.e. via elements that do not belong to the set, because the size of a complement may be very small, whereas the size of the set cab be huge or even infinite...

[richardhundt]

We can do better!

function let_impl(ctx, ...)
   util = require("shine.lang.util")
   vars = { } 
   vals = { } 
   exps = { ... }
   curr = vars
   for i=1, #exps do
      e = exps[i]
      if e.type == 'BinaryExpression' and e.operator == ':=' then
         ctx.define(e.left.name)
         vars[#vars + 1] = ctx.get(e.left)
         vals[#vals + 1] = ctx.get(e.right)
         curr = vals
      else
         if curr == vars then
            ctx.define(e.name)
         end 
         curr[#curr + 1] = ctx.get(e)
      end 
   end 
   return ctx.op({'!define', ctx.op(vars), ctx.op(vals) })
end

macro let = let_impl

let a, b, c := 1, 2, 3

[romix]

Nice. Syntactically it definitely looks better now. But still at the price of using a custom syntax.

Can you come up with something that can be applied (selectively) to regular declarations (eventually annotated/marked in a special way)?

[richardhundt]

Sorry, out of order reply... the question is how.

So if I've understood you then your argument is that sometimes you might want to intercept local variable declarations - possibly in somebody else's code - without annotations. That simply cannot be done. I hate saying that, but that's the truth.

Shine currently loads 3 kinds of files when you call require or say import: raw bytecode (with debug info possibly stripped), Lua sources and Shine sources. This happens at runtime. There's no linker and there's no hook which the LuaJIT VM gives you to trap a local declaration in pre-compiled bytecode. Variables don't exist. You only have registers, and they get reused fairly aggressively. The same would apply to Lua sources being loaded.

The absolute most you can hope for is that the compiler for only Shine sources gives you some sort of hook. But that requires actually communicating with the compiler, via macros, or decorators or guards.

That's 3 hooks already:

• compile time: macros
• definition time: decorators
• run time: guards

Add the standard Lua debug library and you can do even more.

Sure, we can have declaration names for decorators, but as I said... you really need to think about variables as not existing at run-time. In a dynamic, late bound language, this fact alone is a show stopper (at least for the problem just described).

In a static language this doesn't matter, most of what you care about, where variables are concerned, happens at compile time. The compiler has an extremely complex execution model (especially type safe languages like Scala, which might as well just run your code while proving it, because they're doing everything but).

I'm not saying it's a bad idea, and I really value your input (you've been awesome, frankly), but this one's easy to imagine in a fuzzy way, but really hard for me to nail down and implement. Unless you can tell me how.

[romix]

Richard, thanks for your last comment. This is really insightful. I think I understand now why it is (mostly) impossible. I have to admit I haven't thought about loading the bytecode. And you are right, in this case there are no variables, only registers... It is indeed a show stopper... At best, it could be probably applied only to the source code written in Shine (which is better than nothing ;-) during compilation.

OK. Let's forget about it for now ... ;-(

What's next on your agenda, now that almost all of the features you planned for are implemented already?

[richardhundt]

On 4/5/14 7:35 PM, romix wrote:

Grats on getting your hands dirty with the translator.

Thanks! ;-)

Regarding your example: I know this kind of using annotations, oh, sorry, annotations, very well. Anyone who used Java with JAXB and/or JAX-WS does more or less this kind of things...

But you access these using reflection as:

Method[] methods = someClass.getMethods();
for (Method method : methods) {
  MyAnno anno = method.getAnnotation(MyAnno.class);
  // ...
}

So you already have a reference to the method. In Shine you can do exactly the same thing.

As for your reaction on my proposal: I expected exactly that, really. I totally understand your arguments about consistency. But my point is that while consistent, it limits the application of decorators too much, IMHO.

Look, I agree that I'm throwing away information (i.e. the name of the declaration, the line on which it is defined, any guards, etc.), but they're not any less useful in Shine than they are in Python or Java, for that matter. In fact in Java you can not use annotations for local variables, which makes Shine's more useful than Java's (run-time erasure, basically).

For example, if I only intercept the values of variables being declared, I cannot do too much interesting in the decorator as I don't know what it is being applied to. OK, I can manipulate a value, but blindly. And I'd imagine that when I intercept a variable declaration, I typically what do it for the following reasons:

• debugging (e.g. printing out initial values assigned to variables)

Okay, valid use case.

• manipulating initial values (e.g. wrapping them into something)

Can already be done. Just return a different set of values.

• manipulating the type of a value being assigned and thus manipulating the type of a variable. E.g. I could try to wrap a (scalar) value into an "is"-predicate and thus make a variable (strongly)-typed or attach a constraint to it.

You're mixing run-time and compile time here. For a custom is predicate, you need to use macros. Hmm... macro decorators?

*

• Manipulation of the type could be also partially based on the name of a variable, e.g. like in Hungarian notation. If my var is called "iCounter", I may want to check that its initial value is an integer and attach "is"-predicate to it.

Again, macros.

• Ability to know the name may also allow some AOP tricks I mentioned above. In particular, you don't need to annotate all declarations explicitly by hand, which is rather intrusive.

Hard problem. See my previous rant :)

So to boil it down: I'd either need to come up with a protocol for
decorators which has consistent
meta-data for all of them or leave them as they are.

It would be ideal, if it would be possible and without a (big) run-time hit. But I'm not 100% convinced that consistency in this case is so important or gets so broken.

After all, by definition, in a PL a declaration associates a name with a type and optionally assigns an initial value. So, it is a tuple (name, type, initial value). Since it is a dynamic language, type is not present. And since we often have also names missing in case of anonymous declarations, only the value remains in most situations. But this is incidental. In the ideal world one would expect a tuple above instead of declaring the worst case to become a normal case. So, if variable (and class?) declarations may provide 2 of 3 ingredients we should be glad and not upset about it...

Sure, I actually agree with this. It's not off the cards, but then I'd make all annotations get names.

[richardhundt]

Next on my list? Good question. Spend some time with my family (no jokes, I'm on a 19 day coding streak, it's time to take a breather).

Other than that, I'm building an A/B testing framework which uses ElasticSearch as a backend. Shine needs a real-world application. This will let me push the concurrency stuff a bit, and while I'm doing that I'll be ironing out some creases in the language - bug fixes mainly - and fleshing out the standard libraries as I need them. I'm hoping that the fallout from that will include a web application framework (which will be decidedly unlike Ruby on Rails), for giving Shine a bit of traction.

So it's time to stabilize the syntax and semantics so that others can actually write code in Shine without it breaking every time I push a commit. The only area which might still change is ranges and slices which I'm not 100% happy with.

Then once the dust has settled, I'll bootstrap the compiler and implement Shine in itself. I've learned the hard way that it's best to leave that kind of thing for later, because it means you always need a working version in order to compile the compiler, and it gets really annoying when you overwrite your good copy with a broken one.

And then I'll probably spend the rest of my life trying to get the documentation done. Human languages are just not made for explaining machine languages.

[romix]

Hi Richard,

I simply wanted to check that you are OK. Haven't seen any commits from you since a very long time. I hope you are fine and probably just too busy with your daily job.

Cheers, Leo

[richardhundt]

Hey, I’m still alive. I’ve been working on an application which uses Shine, and it’s a pretty big job, so haven’t pushed it out into the world yet. I’ll update Shine when I’ve flushed this from my stack.

Take care, -R

On 12 Jun 2014, at 12:29, romix notifications@github.com wrote:

Hi Richard,

I simply wanted to check that you are OK. Haven't seen any commits from you since a very long time. I hope you are fine and probably just too busy with your daily job.

Cheers, Leo

— Reply to this email directly or view it on GitHub.

[romix]

Hey, Richard!

Thanks for responding!

Hey, I’m still alive.

I was a bit worried due to lack of any activity on all your Github projects ;-) I'm glad that your are fine and busy with interesting things.

Take care, Leo

[romix]

Hi Richard,

Nice to see recent commits from you.

Hey, I’m still alive. I’ve been working on an application which uses Shine, and it’s a pretty big job, so haven’t pushed it out into the world yet. I’ll update Shine when I’ve flushed this from my stack.

What's up? Are you still work on Shine or use it n your projects? Or may be you are now working on totally new challenges and almost abandoned Shine development? Just curious...

[richardhundt]

Hey, good to hear from you again.

That project was a six month attempt at getting my own data analytics product out the door based on Shine, but the money ran out so I got distracted with becoming gainfully employed again. So yeah Shine's been on a back burner.

However, I'm itching to add optional static typing to the language based on those Lua papers you sent me. This will change the character of the language significantly, and I may need to give up on a few of the current features such as run-time predicate assertions.

Now I just need to free up a bit of time for it...

On Thu, Dec 11, 2014 at 9:41 PM, romix notifications@github.com wrote:

Hi Richard,

Nice to see recent commits from you.

Hey, I’m still alive. I’ve been working on an application which uses Shine, and it’s a pretty big job, so haven’t pushed it out into the world yet. I’ll update Shine when I’ve flushed this from my stack.

What's up? Are you still work on Shine or use it n your projects? Or may be you are now working on totally new challenges and almost abandoned Shine development? Just curious...

— Reply to this email directly or view it on GitHub https://github.com/richardhundt/shine/issues/38#issuecomment-66685037.

People who think they know everything are a great annoyance to those of us who do.

[romix]

That project was a six month attempt at getting my own data analytics product out the door based on Shine

Cool! It would be interesting to hear about your practical experiences of using Shine for a reasonably big project. What features were most useful? What features were almost not used in practice, etc? What is missing? Overall, what was done right and what was overlooked in the initial design of the language? Also, may be you can judge how would it compare to using pure Lua if you would have written your project in Lua?

However, I'm itching to add optional static typing to the language based on those Lua papers you sent me.

Is it because based on your experience of using Shine you think that more (static) typing could be beneficial? Have you run into problems because it is too dynamic currently?

This will change the character of the language significantly, and I may need to give up on a few of the current features such as run-time predicate assertions.

I see your point. Though I'm not sure if you need to give up on certain run-time features. May be it is possible to combine the optional typing with them? Also, the typing is optional as you say. So, it could be to some extent considered as annotations for a type checker, which does not even need to run at run-time. It could be done e.g. by means of a separate tool that is used only during the development cycle.

[richardhundt]

Cool! It would be interesting to hear about your practical experiences of using Shine for a reasonably big project. What features were most useful? What features were almost not used in practice, etc? What is missing? Overall, what was done right and what was overlooked in the initial design of the language? Also, may be you can judge how would it compare to using pure Lua if you would have written your project in Lua?

Long answer; hope I don't bore you with it :P

I find myself not using destructuring assignment much, and I think it's because, for classes, it requires implementing the unapply hook (with Scala case classes you get it for free). One way to fix that would be to have objects store their members in sequential slots, so the underlying structure of an instance is "array-like", instead of "hash-like". Try/catch and given/case are two other constructs which I tend not to use. I think try/catch was a bad idea as the syntax doesn't make it obvious that you're creating a closure in each block which means nesting a try/catch inside a loop pressures the GC by creating and throwing away two (or more) closures on each iteration.

Then there's an ambiguity in the parser wrt bare function calls in class/module bodies, and the only sane way to fix this is to introduce methods with a function keyword instead of just the symbol and parameter list (someone else suggested this a while ago and I didn't like the idea then, but I've changed my mind). This will make it easier for syntax highlighters too. I've already got a branch where I've done that.

On the other hand, one thing I definitely find myself loving is having LPeg as a first class citizen. A lot of what we do in programming is massage and transform data, and the prevalence of regular expressions is a testimony to that. LPeg, however, just blows regexes clean out of the water. As an example of this, I needed to parse command-line arguments, so I looked at python's argparse library for inspiration. Python does it in 2386 lines. Whereas this guy [1] is about 140 lines. Granted, it's not as feature complete, but I think it does most of what the Python version does.

Besides that, I'm liking annotations, and I use the type predicates a lot (i.e. is), especially in function signatures. And of course the scheduled coroutines and I/O runtime. Considering that the event loop and scheduler is written in Shine itself, it's much faster than I expected with the little web server from [2] outperforms node.js by a large margin with a much smaller memory footprint.

Compared to having done it in Lua? I think it's worth noting the tradeoffs Lua makes for its niche in being an embedded language. It can't make assumptions about its global scope, so it can't catch mistyped variable names at compile time, and Shine doing this has been a godsend, and is that's something I would really miss for larger projects. So for me, Shine has succeeded in its goal, which was to take Lua, add some batteries, and make it easier to write stand-alone applications with it, while still leveraging existing Lua libraries.

Having said all this, there are also a couple of thing's I'm undecided about. Zero-based arrays, or having an array type in the language at all. That extends to tables too. You can't make tables monadic without giving them all a meta-table, and I'm thinking it'd be nice to give Shine a more functional set of core libraries (Map, List, Set, etc.), and doing away with the {} and [] syntax for literal tables and arrays. I suppose we could keep both. Not sure.

Another thing is libraries and linking. I've done a bit of Go recently, and I like the idea of having the compiler spit out a single statically linked executable which you can deploy as is, without worrying about dependencies. It's not hard to do, and if you have a look at the current Makefile for the libs, I'm doing most of what I need already to produce the shared object files. The idea would be to just leave the .o files for libraries in a standard location (~.shine/...), statically link them in during compilation along with the VM, and wrap it all in a bit of C template for the main.

[1] https://github.com/richardhundt/shine/blob/master/lib/sys/argparse.shn [2] https://github.com/richardhundt/swarm

However, I'm itching to add optional static typing to the language based on those Lua papers you sent me.

Is it because based on your experience of using Shine you think that more (static) typing could be beneficial? Have you run into problems because it is too dynamic currently?

No problems per se. I've just always liked the idea of solving the whole static-vs-dynamic typing argument by letting the programmer choose. There also aren't many languages which do that, so it'd be another feather in the cap for Shine.

This will change the character of the language significantly, and I may need to give up on a few of the current features such as run-time predicate assertions.

I see your point. Though I'm not sure if you need to give up on certain run-time features. May be it is possible to combine the optional typing with them? Also, the typing is optional as you say. So, it could be to some extent considered as annotations for a type checker, which does not even need to run at run-time. It could be done e.g. by means of a separate tool that is used only during the development cycle.

My concern was with mixing runtime predicate assertions and type annotations. What would it look like? Let's say the run-time type assertion uses is (which it does at the moment), and the type annotation uses :. I think it'd be crazy if you could say: local a: Number is String. So you'd probably want to stick with one syntax and only insert the run-time checks if you can't prove at compile time (this is actually "gradual typing" as opposed to "optional typing").

The tricky bit with all of this is that you need to export descriptors for classes and functions which are defined outside of the current compilation unit, so that the compiler can validate against imported symbols. The compiler would either need to create two files, one with bytecode, the other with the descriptors, or I'd need to invent a file format which packs in both in different segments, where the descriptor segment is read during compilation, and the bytecode segment is loaded at run-time (or statically linked in as described above).

[romix]

Long answer; hope I don't bore you with it :P

Not at all! I enjoyed it very much!

I find myself not using destructuring assignment much, and I think it's because, for classes, it requires implementing the unapply hook (with Scala case classes you get it for free). One way to fix that would be to have objects store their members in sequential slots, so the underlying structure of an instance is "array-like", instead of "hash-like".

But couldn't unapply be generated automatically? E.g. in your class definition you "declare" (by calling a dedicated function) that you have the following set of fields. You already have "like" and "fields" that are used to introduce/describe structural properties. I think this idea could be extended to automatically generate unapply in most cases.

I think try/catch was a bad idea as the syntax doesn't make it obvious that you're creating a closure in each block which means nesting a try/catch inside a loop pressures the GC by creating and throwing away two (or more) closures on each iteration.

Yes, this is an interesting observation. I'm wondering if it is a drawback of the current implementation or if it cannot be done more efficiently in principle without changing the VM.

Then there's an ambiguity in the parser wrt bare function calls in class/module bodies ...

Well, this is a minor issue. It slightly changes the syntactic sugar, but it does not affect the semantics.

On the other hand, one thing I definitely find myself loving is having LPeg as a first class citizen. ... Besides that, I'm liking annotations, and I use the type predicates a lot (i.e. is), especially in function signatures. And of course the scheduled coroutines and I/O runtime.

Absolutely! These features are very powerful. I would second everything you named here.

Having said all this, there are also a couple of thing's I'm undecided about. Zero-based arrays, or having an array type in the language at all. That extends to tables too. You can't make tables monadic without giving them all a meta-table, and I'm thinking it'd be nice to give Shine a more functional set of core libraries (Map, List, Set, etc.), and doing away with the {} and [] syntax for literal tables and arrays. I suppose we could keep both. Not sure.

Yes, I think it would make sense to keep both. More over, {} and [] can be considered as a literal syntax for initializing Maps, Lists, Sets. I.e. it does not automatically mean that their internal representation uses an array or a table internally. It is just a syntax to provide a set of elements to the constructor of a corresponding collection type. The constructor can then decide how to convert it into its desired internal representation.

Regarding your idea of producing statically linked executables: Yes, it is a neat idea. But again, it is more about tooling & deployment, rather than about the language semantics. It is still important and useful, nevertheless.

My concern was with mixing runtime predicate assertions and type annotations. What would it look like? Let's say the run-time type assertion uses is (which it does at the moment), and the type annotation uses :. I think it'd be crazy if you could say: local a: Number is String. So you'd probably want to stick with one syntax and only insert the run-time checks if you can't prove at compile time (this is actually "gradual typing" as opposed to "optional typing").

Ah. You mean it in this sense, OK. Well, I agree with you that using two different syntactical ways for type declarations is strange. But on the other side is can attach any predicate (e.g. any kind of a runtime invariant or contract), not only type predicate. From this perspective it would be still nice to have an opportunity to attach such predicates, if you need them. In most cases providing just a (static) type information would be enough, I guess. And yes, it is a good idea to prove as much as possible during the compilation already and leave only those type-checks that cannot be proven at compile-time. Plus any attached predicates.

BTW local a: Number is String as syntax is probably not the best thing in the world ;-), but it is not too far away from contract based programming a-la Eiffel. It could be nice to split it syntactically into two lines or something like this. One for declaring the type and one for attaching the predicate.

And regarding the possible contradictions, e.g. local a:Number is String, I don't think it is a real problem. You don't need to detect them. They will show-up on their own ;-) And trying to resolve it at compile-time is probably equivalent to finding a solution for a halting problem ;-)

The tricky bit with all of this is that you need to export descriptors for classes and functions which are defined outside of the current compilation unit, so that the compiler can validate against imported symbols. The compiler would either need to create two files, one with bytecode, the other with the descriptors, or I'd need to invent a file format which packs in both in different segments, where the descriptor segment is read during compilation, and the bytecode segment is loaded at run-time (or statically linked in as described above).

Hmm. It goes too much in the direction of separate compilation which we know from strongly typed languages. It would really introduce compiled modules. If you then have only the bytecode of the external module at your disposal, then this is the only way probably.

But if all your external modules are in a source code form, you could just try to parse them on the fly, or? E.g. you'd parse only class definitions and function definitions, without parsing their bodies. Should be still pretty fast I guess. Plus, one could cache/persist the results of such scans, so that you don't need to re-parse later unless the source code was changed.

Anyways, I'm looking forward to see more progress on Shine. I'd be happy to be helpful when you need to discuss certain aspects of it with someone.

[richardhundt]

But couldn't unapply be generated automatically? E.g. in your class definition you "declare" (by calling a dedicated function) that you have the following set of fields. You already have "like" and "fields" that are used to introduce/describe structural properties. I think this idea could be extended to automatically generate unapply in most cases.

The problem is that classes don't have declared non-method members. Any properties are attached to an instance during the constructor (or lazily elsewhere). Here's a sample for reference:

class Point
   self(x, y)
      self.x = x
      self.y = y
   end
end

I see two alternatives. 1) Get rid of the unapply style extractors. 2) Make variable assignments in the class body count as properties, so:

class Point
   x is Number = 0
   y is Number = 0
end

I tried 1) and we talked about that a bit. I guess 2) is reasonably sane anyway. No need for fields then. If we had this, and a # operator for classes (inherited fields would need to be taken into account), then unapply could be made to work out of the box. I'll add that in v0.2.

[try/catch] Yes, this is an interesting observation. I'm wondering if it is a drawback of the current implementation or if it cannot be done more efficiently in principle without changing the VM.

Theoretically the closures could be hoisted out of the loop, but then you need to be really careful that variables which are mutated inside it are hoisted too, and you can't do this if they depend on the loop variables. You'd need alias analysis for that.

What also gets in the way is that return, continue and break statements handled inside the try, catch or finally blocks need to be handled specially, because you want the closures to be transparent. In other words, the outer function or loop needs to return or break, not the function passed internally to pcall. I can't seen an easy way around it. I really want the VM to give me push_error_handler and pop_error_handler so that I can bracket the code with those. That way I don't need to create a closure for the try.

Yes, I think it would make sense to keep both. More over, {} and [] can be considered as a literal syntax for initializing Maps, Lists, Sets. I.e. it does not automatically mean that their internal representation uses an array or a table internally.

This is one of the things I've been struggling with for a while. Arrays were useful because I could add methods to them without polluting the underlying object, which is assumed to be indexed numerically. With hash maps it's a different story. Lua doesn't distinguish between an element in the hash and a property access so t.foo means exactly t['foo']. In Lupa I made the compiler insert a __getitem/__setitem call for the t['foo'] case to distinguish it. This frees up the map itself to contain methods.

In Shine I wanted to keep things as simple as possible, but if this thing is going to grow up, then I'd take this route again.

[...] But if all your external modules are in a source code form, you could just try to parse them on the fly, or? E.g. you'd parse only class definitions and function definitions, without parsing their bodies. Should be still pretty fast I guess. Plus, one could cache/persist the results of such scans, so that you don't need to re-parse later unless the source code was changed.

I've been thinking of a happy middle ground which still lets you distribute your stuff in bytecode form. If we had each compilation unit return a table containing:

{ __head={ <type descriptors> }, __body=function(...) <module body> end }

Then __head would be consulted at compile time without side effects, and __body at run-time. Might need to shim existing Lua libraries for interop, or use some other heuristic (like checking for the existence of a marker in the module table). Sound reasonable?

[richardhundt]

My English parser is broken :-(

I get what you mean now:

But couldn't unapply be generated automatically? E.g. in your class definition you "declare" (by calling a dedicated function) that you have the following set of fields. You already have "like" and "fields" that are used to introduce/describe structural properties. I think this idea could be extended to automatically generate unapply in most cases.

So you're suggesting to use a utility function implemented in the language to generate the __unapply, akin to fields. Okay, that'd be option 3). However, if we're doing some static checks, then I think it'd be nice for the compiler to know the shape of an object, including its properties.

[romix]

Thanks for your comments!

So you're suggesting to use a utility function implemented in the language to generate the __unapply, akin to fields. Okay, that'd be option 3). However, if we're doing some static checks, then I think it'd be > nice for the compiler to know the shape of an object, including its properties.

Yes, you are right. The option (3) would be too dynamic, if we are going to do some static checks. For static checks the compiler needs to know the shape of an object, indeed. So, (3) is more for scenarios where you don't know the shape in advance and want to check it at runt-time. It could be complementary to the static type checks.

I really want the VM to give me push_error_handler and pop_error_handler so that I can bracket the code with those.

This is an interesting comment. Since we are LuaJIT-based, we should not completely exclude the possibility of extending/bending LuaJIT for our needs. Yes, it would be the last resort, probably, and would eventually require much more effort. But it may allow us to implement certain features very efficiently in cases, where it cannot be efficiently expressed in pure Lua.