richardhundt / shine

A Shiny Lua Dialect
Other
231 stars 18 forks source link

Support for guards at more places #49

Closed romix closed 10 years ago

romix commented 10 years ago

HI Richard,

We discussed some of the ideas in the #38 already, but I thought that moving them into a dedicated issue would make it easier to track the progress.

Currently, guards are supported for function parameters. It would be nice to support them at more places, e.g.:

1) Inside the cases of a givenstatement (at the end of the cases, as an additional condition). We discussed it and it should be fairly easy to implement.

2) It would be interesting to support guards for variables as a counterpart of guards for function parameters. I think they can be seen as invariants. It can be type guards, which should check that only values of a certain type are assigned to a variable. Or it can be more generic guards that may check that values of variables satisfy a certain invariant (e.g. >= 10 and <= 100). Obviously, it may be even more complex. I don't know if it is feasible at all in a general case and to which extent. But if it would be possible, it would basically provide an optional typing for Shine, which can be very interesting and useful. The big question if it can be done at all and efficiently. It would be also interesting to see how far LuaJIT can go in optimizing it away and making use of provided information. More over, in some cases, some kind of a local type analysis or inference could probably eliminate a lot of checks introduced by guards, because their results can be computed statically....

I think this feature is probably the most difficult/controversial of all that I mention here.

3) If we have support for parameter guards and/or variable guards, one can think about different way of describing them. Currently, you directly attach them to the declaration of a parameter or variable. But one can think of annotations/decorators. Or even AOP (Aspect Oriented Programming)-like approaches, where you could say "all parameters/variables whose name starts with i should be of type int", "all objects of class Point should be immutable", etc. This approach is very powerful, because with just one such definition you can cover any number of real parameter/variable declarations...

4) And, as we discussed, there could be a compiler/run-time switch to disable/enable guards. This way one could use guards in development mode to ensure that her app is type-correct, etc and disable them for production, if performance is more important.

richardhundt commented 10 years ago

On 3/31/14 9:31 AM, romix wrote:

HI Richard,

We discussed some of the ideas in the #38 https://github.com/richardhundt/shine/issues/38 already, but I thought that moving them into a dedicated issue would make it easier to track the progress.

Currently, guards are supported for function parameters. It would be nice to support them at more places, e.g.:

1) Inside the cases of a |given|statement (at the end of the cases, as an additional condition). We discussed it and it should be fairly easy to implement.

still TODO

2) It would be interesting to support guards for variables as a counterpart of guards for function parameters. I think they can be seen as invariants. It can be type guards, which should check that only values of a certain type are assigned to a variable. Or it can be more generic guards that may check that values of variables satisfy a certain invariant (e.g. >= 10 and <= 100). Obviously, it may be even more complex. I don't know if it is feasible at all in a general case and to which extent. But if it would be possible, it would basically provide an optional typing for Shine, which can be very interesting and useful. The big question if it can be done at all and efficiently. It would be also interesting to see how far LuaJIT can go in optimizing it away and making use of provided information. More over, in some cases, some kind of a local type analysis or inference could probably eliminate a lot of checks introduced by guards, because their results can be computed statica lly....

I think this feature is probably the most difficult/controversial of all that I mention here.

I've pushed a commit which associates guards to variables.

Destructuring works too:

o = { answer = 42 }

-- later

local { answer = x is Number } = o

The way guards work is simple: any object which implements an __is method can be used as the right-hand-side in an is assertion. Classes and modules have the hook already implemented (there's special handling of cdata and primitive types).

Check out util.guards which lets you say things like:

import ge from "util.guards"

local n is ge 0 = 42
n = 69         -- OK
n = -1        -- Error: (ge 0 expected got Number)

On the efficiency side, it depends on what the guard is doing. If the guard has short loops then it's likely to cause a trace abort (i.e. the like guard which does structural checks). Simple checks such as Number or class checks are very fast. I have to crank a micro benchmark up to 100M iterations to notice any difference.

3) If we have support for parameter guards and/or variable guards, one can think about different way of describing them. Currently, you directly attach them to the declaration of a parameter or variable. But one can think of annotations/decorators. Or even AOP (Aspect Oriented Programming)-like approaches, where you could say "all parameters/variables whose name starts with i should be of type int", "all objects of class Point should be immutable", etc. This approach is very powerful, because with just one such definition you can cover any number of real parameter/variable declarations...

I'll need to think about this some more.

4) And, as we discussed, there could be a compiler/run-time switch to disable/enable guards. This way one could use guards in development mode to ensure that her app is type-correct, etc and disable them for production, if performance is more important.

TODO

— Reply to this email directly or view it on GitHub https://github.com/richardhundt/shine/issues/49.

romix commented 10 years ago

I've pushed a commit which associates guards to variables.

Wow! That was fast! Very cool! I'll play with it ASAP and provide some feedback.

Minor comment regarding your current syntax: local n is ge 0 = 42 May be something more typical for PLs could be used? Essentially is-expression comes at the place of a type specifier in many languages. May be local n: ge 0or local n: myType is more natural?

Simple checks such as Number or class checks are very fast. I have to crank a micro benchmark up to 100M iterations to notice any difference.

Nice. I think a typical check in most cases will be that the value being assigned is of a specific type. I.e. it checks that its metatable is the same as of this type. So, it should be very fast then, if I understand you correctly.

romix commented 10 years ago

I found a minor problem. If you define your variable with is-expression, but do not initialize it, it will throw an error, e.g.

local j is Number

When you compile, you get: Number expected got Nil I think Nil should be OK for uninitialized variables....

Another idea: If it is possible, it would be nice to remember the name of the variable. So that when an error message is shown, you can also see the name of a variable. It helps if the line in question contains many expressions.

Question: Is it possible to implement const this way? I.e. a variable that can be assigned only once and keeps its value unchanged? It would be also cool, if one could apply it for complex types like classes or tables...

richardhundt commented 10 years ago

On 3/31/14 1:11 PM, romix wrote:

I found a minor problem. If you define your variable with |is|-expression, but do not initialize it, it will throw an error, e.g.

local j is Number

When you compile, you get: |Number expected got Nil| I think Nil should be OK for uninitialized variables....

That's debatable. It becomes a weak assertion. The following will raise an error, but it's not immediately obvious why:

function foo(a is Number)
   -- do something with a
end

local j is Number
foo(j)

I would rather implement explicit nil-able types:

local j is ?Number

You can already do that with:

import maybe from "util.guards"

?Number = maybe Number
local j is ?Number

j = nil
j = 42
-- etc.

I've extended identifiers to include '?' and '!' btw. I could provide builtin nilable types for primitives, so that you don't need to create your own. However if you define a class Point then you'll need to do the maybe thing. I'm trying to keep as much of this out of the core language as possible (it's basically just an _is hook call at the moment).

Another idea: If it is possible, it would be nice to remember the name of the variable. So that when an error message is shown, you can also see the name of a variable. It helps if the line in question contains many expressions.

This shouldn't be too hard.

Question: Is it possible to implement |const| this way? I.e. a variable that can be assigned only once and keeps its value unchanged? It would be also cool, if one could apply it for complex types like classes or tables...

That's already built into TvmJIT, actually. Was thinking about using final because it's inline with the error message which TvmJIT raises.

— Reply to this email directly or view it on GitHub https://github.com/richardhundt/shine/issues/49#issuecomment-39076870.

romix commented 10 years ago

OK. Thanks for clarifications. Nilable types is a neat idea. Maybe-approach is nice as well. I'm quite OK with this and it is safer than what I was asking for.

That's already built into TvmJIT, actually. Was thinking about using final because it's inline with the error message which TvmJIT raises.

final is OK as well. But I guess it only works for primitive types or top-level types? I.e. it does not have a deep final/ deep const semantics, which is OK in most cases.

It would be just nice to have an easy way to say that something compound has more or less a static type with a predefined structure (i.e. only these three fields), so that any attempt to break it (e.g. by adding new fields, etc) is caught at runtime.

richardhundt commented 10 years ago

On 3/31/14 1:33 PM, romix wrote:

OK. Thanks for clarifications. Nilable types is a neat idea. Maybe-approach is nice as well. I'm quite OK with this and it is safer than what I was asking for.

That's already built into TvmJIT, actually. Was thinking about using
|final| because it's inline with the error message which TvmJIT
raises.

|final| is OK as well. But I guess it only works for primitive types or top-level types? I.e. it does not have a deep final/ deep const semantics, which is OK in most cases.

I think if you wanted immutable structural types, then these needs to types on their own (think Scala). It's not feasible to do it at the compiler level.

It would be just nice to have an easy way to say that something compound has more or less a static type with a predefined structure (i.e. only these three fields), so that any attempt to break it (e.g. by adding new fields, etc) is caught at runtime.

A really cheap way is to do something like this (untested):

local immutable = { } function immutable::__newindex(o, k) error("attempt to modify immutable table",2) end

local t = { answer = 42 } as immutable

I've put a gist[1] up which lets you say:

local o = { } as fields { x = Number, y = Number }

o.x = 42
o.y = 101   -- type checks as a bonus
o.z = 69     -- error

You can even do it for classes (gist needs to be modified for that, but it's not hard):

class Point
   include fields { x = Number, y = Number }
end

That's the kind of thing I was thinking about when I meant flexibility in the language. You can create your own syntax.

[1] https://gist.github.com/richardhundt/9900812

— Reply to this email directly or view it on GitHub https://github.com/richardhundt/shine/issues/49#issuecomment-39078345.

romix commented 10 years ago

Very nice trick with immutable! I'll play with it

BTW, I cannot built trunk after your latest commit:

...
../boot/bin/shnc  -n "sys.unix"      sys/unix.shn      Private/src/git/shine/build/sys_unix.o
tvmjit: ...Private/src/git/shine/boot/src/shine/lang/translator.raw:0: attempt to index a nil value
stack traceback:
    ...Private/src/git/shine/boot/src/shine/lang/translator.raw: in function 'get'
    ...Private/src/git/shine/boot/src/shine/lang/translator.raw: in main chunk
    ...Private/src/git/shine/boot/src/shine/lang/translator.raw: in main chunk
    ...Private/src/git/shine/boot/src/shine/lang/translator.raw: in function 'get'
    ...Private/src/git/shine/boot/src/shine/lang/translator.raw: in main chunk
    ...Private/src/git/shine/boot/src/shine/lang/translator.raw: in function 'get'
    ...Private/src/git/shine/boot/src/shine/lang/translator.raw: in main chunk
    ...Private/src/git/shine/boot/src/shine/lang/translator.raw: in function 'get'
    ...Private/src/git/shine/boot/src/shine/lang/translator.raw: in main chunk
    ...Private/src/git/shine/boot/src/shine/lang/translator.raw: in function 'get'
    ...Private/src/git/shine/boot/src/shine/lang/translator.raw: in function 'translate'
    ...Private/src/git/shine/boot/src/shnc.raw: in function 'start'
    ../boot/bin/shnc:2: in main chunk
    [C]: at 0x01000016a0
make[1]: *** Error 1
richardhundt commented 10 years ago

On 3/31/14 10:22 PM, romix wrote:

Very nice trick with immutable! I'll play with it

BTW, I cannot built trunk after your latest commit:

Thanks, yet again!

romix commented 10 years ago

Trunk works now. Thanks!

I took your gist and experimented with classes + fields. Simply by extending fields my class may have fields of a certain type and only them. And I don't even need to change anything in fields. Not bad.

The include trick does not work out of the box. I guess some changes are needed. It would be actually nice, if one could take an existing class apply something like fields to it and make it behave like "fields" when it comes to the fields of this class. Probably fields could be extended to take a class, enumerate over its fields and thus build the table which it currently takes as a parameter. Or something along these lines.

But now the problem I discovered. If I take your gist and just add a few lines:

-- Your fields definition from gist comes here

-- Start of my change
class Point extends fields { x = Number, y = Number }
end
.
p = Point(1,2)
p.x = 10
--p.z = 100 -- BOOM!
-- End of my change

-- This is your code at the end of gist
o = { } as fields { x = Number, y = Number }
o.x = 42
o.y = 101
o.z = "cheese" -- BOOM!

If you try to execute your gist with my change (marked with "start of my change" / "end of my change") then shine throws no errors. I.e. even the o.z assignment is OK. But if you remove/comment out my change, then this assignment triggers an error. I think it is a bug, or?

richardhundt commented 10 years ago

On 3/31/14 11:20 PM, romix wrote:

Trunk works now. Thanks!

I took your gist and experimented with classes + fields. Simply by extending fields my class may have fields of a certain type and only them. And I don't even need to change anything in fields. Not bad.

The include trick does not work out of the box. I guess some changes are needed. It would be actually nice, if one could take an existing class apply something like fields to it and make it behave like "fields" when it comes to the fields of this class. Probably fields could be extended to take a class, enumerate over its fields and thus build the table which it currently takes as a parameter. Or something along these lines.

But now the problem I discovered. If I take your gist and just add a few lines:

|-- Your fields definition from gist comes here

-- Start of my change class Point extends fields { x = Number, y = Number } end . p = Point(1,2) p.x = 10 --p.z = 100 -- BOOM! -- End of my change

-- This is your code at the end of gist o = { } as fields { x = Number, y = Number } o.x = 42 o.y = 101 o.z = "cheese" -- BOOM! |

If you try to execute your gist with my change (marked with "start of my change" / "end of my change") then shine throws no errors. I.e. even the |o.z| assignment is OK. But if you remove/comment out my change, then this assignment triggers an error. I think it is a bug, or?

Yep. Glad you found this one. Would have been hard to spot.

However, for classes I would do something like:

class Point include fields { x = Number, y = Number } end class Point3D extends Point include fields { z = Number } end

so that fields can be inherited.

I've updated the gist to show the implementation.

— Reply to this email directly or view it on GitHub https://github.com/richardhundt/shine/issues/49#issuecomment-39143627.

romix commented 10 years ago

Yes. This bug is fixed now. And classes work.

But original example does not work anymore with your latest gist:

 o = { } as fields { x = Number, y = Number }
 o.x = 42
 o.y = 101
 o.z = "cheese" -- BOOM!

I guess this is expected? Because now fields returns module and not a class? Or is it possible to make fields work in both contexts? Or even in more general situations?

Of course, a simple workaround is very easy:

function tablefields(schema)
    class tablefields
          include fields schema
    end
    return tablefields
end

o = { } as tablefields { x = Number, y = Number }
o.x = 42
o.y = 101
o.z = "cheese" -- BOOM!

With this trick it works, because tablefields dynamically create a class using fields and everything is fine. The only annoying thing is that one needs to use fields for classes and tablefields for tables, i.e. it is not uniform. But may be it is not bad, because it clearly shows that variable is a class or a table.

Overall, guards seem to be a very nice and useful feature. It allows you to introduce optional typing, invariants and much more. And all that with minimal language changes. Impressive!

richardhundt commented 10 years ago

On 4/1/14 9:05 AM, romix wrote:

Yes. This bug is fixed now. And classes work.

But original example does not work anymore with your latest gist:

o = { } as fields { x = Number, y = Number } o.x = 42 o.y = 101 o.z = "cheese" -- BOOM!

I guess this is expected? Because now |fields| returns module and not a class? Or is it possible to make |fields| work in both contexts? Or even in more general situations?

Yep, it is expected. Modules aren't designed to be instantiated, so whereas saying o = { } as <class> gives you an instance, it doesn't work for modules (as calls setmetatable and modules don't have the required structure to handle __set__ and __get__ in a way that's meaningful when they're used as meta-tables).

If you look at this code path:

o = { } as fields { x = Number, y = Number }

The table you're passing to fields doesn't change. It's constant. So it's kinda crazy allocating these anonymous classes each time that is executed (it's something like 4 tables and 4 closures each time a class is created).

So I'd propose a struct function much like fields but it constructs an anonymous class, instead of a module, and then you'd reuse the definition:

Point = struct { x = Number, y = Number }
function gimme()
   return { } as Point
end

Shiny :)

richardhundt commented 10 years ago

Actually thinking about this some more:

Point = struct { x = Number, y = Number }
function gimme()
   return { } as Point
end

You could make it nicer so that you can say:

Point = struct { x = Number, y = Number }

o = Point { x = 1, y = 2 }

using __apply

function struct(schema is Table)
   -- snip
   class struct
      function self.__apply(init = { })
         o = { } as self
         for k, v in init do
            o[k] = v
         end
         return o
      end
   end
   return struct
end

If you also give it __unapply then you get pattern matching with it (think case classes in Scala).

I've started a tutorial section on the wiki where I'm collecting this stuff. If you feel like it I can add you as a contributor and you can help expand it. Besides myself, you're about the only expert user Shine has :) Only if you want to, so no pressure.

romix commented 10 years ago

I implemented your approach with struct function which creates a class. I also added the __apply method. Works like a charm. Very nice!

Coordinate = struct { x = Number, y = Number }

o = Coordinate { x = 1, y = 2 }
o1 = typeof(o) { x = 10, y = 11}

typeof is also very nice to have, if you want to have two objects of the same type.

I've started a tutorial section on the wiki where I'm collecting this stuff. If you feel like it I can add you as a contributor and you can help expand it. Besides myself, you're about the only expert user Shine has :) Only if you want to, so no pressure.

I'll see what I can do. No promises ;-) So far I'm experimenting a lot. But you add features faster than I'm able to absorb them ;-) I still need to learn more about Shine (and Lua) to write idiomatic code. I probably spent too much time with static, strongly typed languages in my past life. Sometimes I don't see solutions because they are too dynamic (e.g. creating types on the fly, changing types, etc as you go, and so on)... But I'm getting there.

romix commented 10 years ago

BTW, I think variable guards should be mentioned on the home page of Shine and in the Wiki, with examples. The feature is way too powerful to be hidden ;-) Examples should probably show usage of it for both typing, but also for defining invariants/constraints

richardhundt commented 10 years ago

On 4/1/14 10:32 AM, romix wrote:

BTW, I think variable guards should be mentioned on the home page of Shine and in the Wiki, with examples. The feature is way too powerful to be hidden ;-) Examples should probably show usage of it for both typing, but also for defining invariants/constraints

Working on that as we speak.

richardhundt commented 10 years ago

On 4/1/14 10:01 AM, romix wrote:

... But you add features faster than I'm able to absorb them ;-) Speaking of which. I've just added procedural macros (and probably broken something in the process):

macro hello!($, expr)
   util = require("shine.lang.util")
   print util::dump(expr)
   mesg = util::unquote($.get(expr))
   return $.op{'!call', 'print', $.op"Hello %{mesg}"}
end

hello!("World!")

The first argument $ is the compiler context. It basically lets you do everything which is currently done in src/lang/translator.lua. No hand holding.

Macros currently can't be imported or exported. This is actually harder than it seems. I'd need to require modules and pull out symbols at compile-time for that, and since Shine will load Lua sources as well as bytecode, it gets a little tricky.

So I'm not sure how useful macros are right now. I'll have to have a think about how to solve this without too much early binding going on. It'd have to be early linking at the very least. I might keep macro definitions in the registry keyed on the module name, or something.

In the meantime, they can be played with.

romix commented 10 years ago

Cool! I'll look into macros, though I don't see what I want to do with them out-of-the-box yet.

Coming back to guards:

If we have struct { x = Number, y = Number } or the same for fields, how would one express a similar thing that is possible with ìs`-constraints? E.g. for a variable, it is possible to write:

x is ge 2 = 3
x = -1

And it will trigger an error.

How can one specify the same thing for structs/fields? I think it would be useful to be able to express it. It would be also symmetric, i.e. we can do it for any variables: local, fields of classes, keys/values of tables, etc.

More over, such predicates like ge, lt and so on are useful. But even more useful would be if they are composable and you can write: x is ge 3 and le 10 or something like this. This would allow for complex invariants to be expressed in a simple form.

Another convenience issue would be to be able to express type and predicate-like constraints more or less independently. I.e. it would be nice to say x is Integer and ge 3 and lt 100.

I think the last two points are more of a syntactic nature. Since all those conditions are predicates after all and one can easily write a dedicated predicate-class with __is-method for each of them. The question is how to express such conditions in a simple way without writing too much of a boilerplate code.

richardhundt commented 10 years ago

On 4/1/14 5:31 PM, romix wrote:

Cool! I'll look into macros, though I don't see what I want to do with them out-of-the-box yet.

Coming back to guards:

This is where you'd need to overload operators for that. LPeg's approach (which in turn is inspired by SNOBOL) would work using * for conjuction and + for disjunction (because of their precedence).

{ x = Number * ge 0 }

The benefit of this is that even though Number doesn't define __mul, if ge 0 does, then that is chosen.

Another convenience issue would be to be able to express type and predicate-like constraints more or less independently. I.e. it would be nice to say |x is Integer and ge 3 and lt 100|.

Here's a gist[1] which does:

x is Number * ge(3) * lt(100) = 5

[1] https://gist.github.com/richardhundt/9920066

I think the last two points are more of a syntactic nature. Since all those conditions are predicates after all and one can easily write a dedicated predicate-class with |__is|-method for each of them. The question is how to express such conditions in a simple way without writing too much of a boilerplate code.

Well, I think syntactically it could be nicer, however, it already can be done without any changes to the language or runtime. And I'd like to keep that kind of flexibility. Adding dedicated syntax is a step towards a complexity can of worms. Just like Lua, I'd rather have Shine provide mechanisms than solutions.

Regarding the boilerplate; see the conjoin and disjoin functions in the gist. They're really simple and completely generic.

romix commented 10 years ago

First of all, thanks for the gist! Very nice!

Well, I think syntactically it could be nicer, however, it already can be done without any changes to the language or runtime. And I'd like to keep that kind of flexibility. Adding dedicated syntax is a step towards a complexity can of worms. Just like Lua, I'd rather have Shine provide mechanisms than solutions.

I think you misunderstood me. I was not proposing extending the language syntax. On the opposite. I was sure that is possible without changing it, completely in the "user-space" (see the last paragraph of my message). And you have just proven that! :-) My shine-fu is not so strong yet, so I was not sure how to come up with a nice syntax. You tricks with mul and plus are a pretty elegant solution.

I also checked that it works for structs and classes with fields. Wonderful! And all that without builtin special mechanisms... It's getting better and better!

IMHO, the fact that is uses predicates instead of types, makes it way more powerful than usual type systems. It allows you to maintain invariants and take into account any constraints. Expressing it in many other languages is very problematic if not impossible, especially in such a concise form. I haven't seen that many systems with such a flexibility.

I'd suggest to provide what you have shown in your gists as a module (may be with some extensions for the sake of completeness, i.e. all major comparison operators, etc). It could be very useful for many people. And may be used when you need (strict) typing and invariants.

Now, that this feature is in place (and much faster and easier than I expected), I'd suggest that you really give a thought to AOP. BTW, it may be also a bit overlapping with your work on macros... AOP-wise, I think it would be interesting to have something like:

This is just a quick bran dump about AOP. It is a very powerful mechanism. Many powerful abstractions can be built then by users on top of it. And implementation is hopefully not so complex.

richardhundt commented 10 years ago

On 4/1/14 9:26 PM, romix wrote:

First of all, thanks for the gist! Very nice!

Well, I think syntactically it could be nicer, however, it already
/can/be done without any changes
to the language or runtime. And I'd like to keep that kind of
flexibility. Adding dedicated syntax is a
step towards a complexity can of worms. Just like Lua, I'd rather
have Shine provide
mechanisms than solutions.

I think you misunderstood me. I was not proposing extending the language syntax. On the opposite. I was sure that is possible without changing it, completely in the "user-space" (see the last paragraph of my message). And you have just proven that! :-) My shine-fu is not so strong yet, so I was not sure how to come up with a nice syntax. You tricks with mul and plus are a pretty elegant solution. No worries. I didn't want to sound like I was ranting, but yeah, wasn't exactly sure what you meant. Nevertheless, I've had this crazy idea to create a whole suite of user only operators, which have no pre-defined meaning but share precedence with their builtin counterparts. The list looks like this:

local usrop_events = {
   [':!'] = '__ubang',
   [':?'] = '__uques',
   [':='] = '__ueq',
   [':>'] = '__ugt',
   [':<'] = '__ult',
   [':|'] = '__upipe',
   [':^'] = '__ucar',
   [':&'] = '__uamp',
   [':~'] = '__utilde',
   [':+'] = '__uadd',
   [':-'] = '__usub',
   [':*'] = '__umul',
   [':/'] = '__udiv',
   [':%'] = '__umod'
}

This would allow you to use __uamp instead of __mul and __upipe instead of __add to get the following:

x is Number :& ge(3) :& lt(100) = 5

So you basically get smileys all over your code ;-)

I haven't pushed it yet, cuz it's kinda weird. Don't know of a language which does that (Scala comes close, but it's not exactly the same idea).

I also checked that it works for structs and classes with fields. Wonderful! And all that without builtin special mechanisms... It's getting better and better!

IMHO, the fact that |is|uses predicates instead of types, makes it way more powerful than usual type systems. It allows you to maintain invariants and take into account any constraints. Expressing it in many other languages is very problematic if not impossible, especially in such a concise form. I haven't seen that many systems with such a flexibility.

The cost of flexibility, of course, you can still say insane stuff like:

x is gt(10) * lt(0)

... but yeah. Out of scope. Don't do it and expect sane results :-)

I'd suggest to provide what you have shown in your gists as a module (may be with some extensions for the sake of completeness, i.e. all major comparison operators, etc). It could be very useful for many people. And may be used when you need (strict) typing and invariants.

Sure. I was planning on adding a tutorial (actually most of our discussions should end up in a tut), but extending the guards library would be worth doing.

Now, that this feature is in place (and much faster and easier than I expected), I'd suggest that you really give a thought to AOP. BTW, it may be also a bit overlapping with your work on macros... AOP-wise, I think it would be interesting to have something like:

  • Apply certain actions before/after/instead of certain points of execution in your program, i.e. at certain places like start/end of function, statement, assignment, class, variable declaration,etc. or at certain point in time, e.g. when a function with a given name or with a given argument is invoked. But I don't want to go now too deep into AOP, I guess you have a rough or even pretty good understanding about what it is.

I'm actually more on the "rough idea" side. My guess is that around would have the following pattern (before or after would be similar):

class Bank
   withdraw(sum)
      self.balance -= sum
   end
end

function around(meta, name, advice)
   orig = meta.__members__[name]
   meta.__members__[name] = function(...)
      return advice(orig, ...)
   end
end

-- create an 'advice'
around Bank, 'withdraw', (orig, self, sum) =>
   print "intercepted 'withdraw' on %{self}, sum: %{sum}"
   return orig(self, sum)
end

Overriding __index and __newindex would give you even more power, but you get the idea.

It wouldn't be too hard to cook up some nice constructor for it:

Logger = Aspec {
   before 'withdraw', (...) =>
      print 'withdrawing', ...
   end
   after 'self', (...) =>
      print "constructor called with", ...
   end
   -- etc.
}

then say log = Logger(Bank), then log.enter() which it calls wrap on all the items in the table you passed to Aspect().Then when you're done say log.leave() which does the unwrap.

The before implementation would be something like (all of this is untested btw):

class before
   self(name, advice)
      self.name   = name
      self.advice = advice
   end
   wrap(meta)
      self.meta = meta
      self.orig = meta.__members__[self.name]
      meta.__members__[self.name] = function(...)
         self::advice(...)
         return orig(...)
      end
   end
   unwrap()
      if self.orig then
         self.meta.__members__[self.name] = self.orig
         self.orig = nil
      end
   end
end

I'll leave the implementation of Aspect as an exercise for the reader :-)

Unlike AspectJ and friends, you need to get an actual reference to the class you're intercepting, but I don't see that as a major issue. Static languages just have all these extra facades because the compiler needs to know about them to do early binding.

In a dynamic language you can just import all the classes you're interested in creating aspects for into one place and wrap them up there. Do it early enough in your program and you should be good to go (actually, instances don't cache their methods, so you can wrap advice around them after instances are already created and still have it work).

  • Implementation-wise it most likely means that any places in Shine compiler/VM, which are of interest, i.e. could be interesting as interception points, you would allow a developer to provide a callback (or something similar), which should be invoked every time this places is reached. This is the interception part. And of course, there should be an expressive way to say what and where should be intercepted (which variables, which classes, which constructs, which actions, etc).

The way I grok it, you mostly want AOP when you can't (or don't want to) get inside some piece of code, so while I get wrapping methods and property access, I don't see the utility in being able to mark an arbitrary point in a chunk of code. Maybe I just don't get what you're saying exactly.

However, the Lua debug::sethook facility can probably get you that far if you tried hard enough (you can definitely hook into function entry and exit).

Again, no need for compiler extensions :-)

Having said that, I'm no AOP guru, so maybe I just don't get it.

romix commented 10 years ago

The idea with "crazy" operators is not bad, but a bit unusual syntactically, yes. May be first one should allow for overloading of more of already existing operators/keywords, e.g. "or", "and", "|", "&", etc?

Another thing that comes to my mind (but it is not quite formulated yet), is something about string interpolation, where shine-expressions could be inserted with a special syntax. The difference from current interpolation could be that they are probably not evaluated at the stage. Instead their representation as expressions is inserted (a bit similar to macros). Then one could write a function to evaluate such a string. It would interpret the meaning of the content in this string and could at a right time evaluate the interpolated expressions or build more complex expressions out of them and evaluate them. E.g. one could write something like constraint_eval("numbers greater then %[shine-expr] and even") and it would build/evaluate a proper final expression.... As I said it is just a quick idea, haven't thought about it too long.

Thanks for your thoughts on AOP.

You are right that if the only goal is to implement around/before/after you can easily do it even in Lua. Just wrap a method around the old one and substitute it. That's easy.

But there are different applications of AOP. Some of them enhance existing classes, e.g. by adding new methods (i.e. essentially mixing in something) or new data fields (i.e. changing their structure).

Some need to intercept certain kind of things that cannot be captured by other means (e.g. you want to intercept declaration/creation of any variable (may be with a certain name or of a certain type), or you want to intercept any creation of a function and enhance it with additional hidden arguments, you may want to intercept beginning end of then/else parts of a conditional statements or entry/exit from loops, etc). These kind of things are virtually impossible to intercept without compiler/VM support. But I agree that they are probably more interesting for debugging/profiling/testing/etc. And in most (boring :-) real world scenarios before/around/after + mixins would be enough.

richardhundt commented 10 years ago

I've pushed the usrop changes along with case guards, so besides a compiler switch to disable assertions this is done. I'm going to leave the compiler switch for now until some evidence comes in that it actually has a significant performance hit (short loops such as in the like predicate can be easily unrolled using the builtin eval function to construct the function), so I'm closing this issue.

richardhundt commented 10 years ago

Just a comment on overloading & and | (or any of the bitwise operators). This requires doing getmetatable - for the meta-method check - which isn't compiled, AFAIK, whereas directly calling the bit.* functions is. This will slow down the common case for bit operations horribly.

I don't really want to add new operators, but I want to keep getmetatable out of the fast path, so user only overloadable operators seemed like a reasonable compromise.