Take a look at OctaScript

[romix]

There is a yet another language being built on top of Lua/LuaJIT. It is OctaScript: https://github.com/OctaForge/OctaScript

https://github.com/OctaForge/OctaScript/wiki

It seems to use LuaJIT Language Toolkit: https://github.com/franko/luajit-lang-toolkit

Overall, OctaForge seems to have a lot of features overlapping with Shine.

I filed this issue just to attract your attention to OctaScript. May be some ideas or implementation details of OctaScript could be useful for Shine development.

[richardhundt]

Thanks for the heads up. It's pretty cool seeing this. If you look at the LICENSE of luajit-lang-toolkit you'll see that it's derived from Nyanga, which is what Shine was called before, so it's great to see that others have found something useful coming out of the project. That's made my day :)

OctaScript seems to have taken the C-style syntax route, and they have some pretty cool builtin functions. No classes though.

Since we're on the topic, I've got a branch which is maturing which - like OctaScript - also goes the curly-bracket syntax route. It diverges from Shine in other ways too. I'm taking the everything-is-an-expression route, so I can now say some weird things like:

let i = do {
   let v = 0
   print "init i to %{v}"
   v
}
let a = while i < 10 {
   i += 1
   take i
}

let b = for i in a {
   take i, i + i
}

for x, y in a {
   print x, y
}

It's not magic really, loops in expression position just get wrapped in coroutine::wrap, and you get a function which a generic for loop can pump.

I'm thinking along the lines of scalaz.stream, where you can describe a computation separately from evaluating it:

class Stream {
   self(seq) {
      self._ = seq
   }
   run() {
      for _ in self._ { }
   }
   drain(f) {
      for _1 in self._ { f(_1) }
   }
   map(f) {
      Stream for _1 in self._ { take f(_1) }
   }
   filter(f) {
      Stream for _1 in self._ { if f(_1) == true { take _1 } }
   }
}

let s = Stream for i=1, 10 { take i }

s.filter((x) => x % 2 == 0).map((x) => "got %{x}").drain(print)

I've removed arrays and tables are now delimited with [] pairs.

Also, there's better support for composing types using with, with is an infix operator:

module Greeter {
   greet(whom: String) {
      print("%{self.mesg} %{whom}!")
   }
}
module Friendly {
   get mesg() { "Hello" } // implicit return
}
class Foo extends Friendly with Greeter { }

f = Foo()
f.greet("World") // Hello World!

o = Greeter with [ mesg = "Hi" ] // table syntax
o.greet("romix") // Hi romix!

Then there's only a single form of destructuring:

let [mesg = m: String]: Foo = f

Also, try/catch is gone as are macros. I could re-introduce macros, but I've not needed them so far. It's been kind of refreshing starting over with a different syntax to give me a chance to rethink a lot of the decisions made for Shine. I'm porting some of these ideas back to Shine, but figured I'd run it by you to see what you think.

[richardhundt]

meh, responding from my mail client breaks markdown :(

Trying again with the code samples:

let i = do {
   let v = 0
   print "init i to %{v}"
   v
}
let a = while i < 10 {
   i += 1
   take i
}

let b = for i in a {
   take i, i + i
}

for x, y in a {
   print x, y
}

class Stream {
   self(seq) {
      self._ = seq
   }
   run() {
      for _ in self._ { }
   }
   drain(f) {
      for _1 in self._ { f(_1) }
   }
   map(f) {
      Stream for _1 in self._ { take f(_1) }
   }
   filter(f) {
      Stream for _1 in self._ { if f(_1) == true { take _1 } }
   }
}

let s = Stream for i=1, 10 { take i }

s.filter((x) => x % 2 == 0).map((x) => "got %{x}").drain(print)

module Greeter {
   greet(whom: String) {
      print("%{self.mesg} %{whom}!")
   }
}
module Friendly {
   get mesg() { "Hello" } // implicit return
}
class Foo extends Friendly with Greeter { }

f = Foo()
f.greet("World") // Hello World!

o = Greeter with [ mesg = "Hi" ] // table syntax
o.greet("romix") // Hi romix!

[romix]

Hi Richard,

I haven't realized that luajit-lang-toolkit is derived from Nyanga. Very cool!

As for your ideas about changes to Shine:

• also goes the curly-bracket syntax route

  I don't care too much about braces, but yes, it looks like these days you have to use curly braces to become successful ;-)

• everything as expression

  I like the idea. I think it is generally very useful.

• Inspiration from scalaz.stream and the like

  Yes, streams are nice. While we are at it, it would be nice to think about a functional composition in general. Specifically, I've seen in some other not purely functional languages that they have some issues with curried/uncurried functions when it comes to compositions. It should be possible to convert between these two forms easily and so on. Otherwise writing programs in functional style could be difficult.

• I've removed arrays and tables are now delimited with [] pairs.

  Does it mean that arrays are now [1,2,3] and tables are [key1:value1, key2:value2]? If so - yes, it makes sense.

  Optimized 0-indexed arrays could be interesting for numerical computations. But they can be implemented as C libs and used via FFI I guess.

• there's better support for composing types using with, with is an infix operator

  OK. Looks very Scala-like ;-) Looks nice.

  Question: can I create classes at runtime and assigne them to a variable? Or do I simply define them in a local scope and return the resulting class to achieve that? I.e. can I pass/return a class as an object?

• Then there's only a single form of destructuring:

  let [mesg = m: String]: Foo = f

  Hmm. I'm not 100% sure that I understand the semantics. Could you elaborate? I understand that inside the []-brackets you introduce new variables and assign some parts of the RHS to it. But do you always need to provide a type, like you do for m and Foo? I hope it is optional. Can those destructuring patterns be nested? Does it replace switch/case with patterns? Or is it only for destructuring assignment? Overall, I think that pattern-matching should stay pretty powerful. It is one of the most interesting features. BTW, OctaScript has a description on their Wiki about their envisioned destructuring assignment and pattern matching. May be there is something interesting there.

• Removal of macros

  Well, for me they were mostly interesting from a perspective of introducing new features into a language or doing some kind of meta-compilation related things. But I agree that it is pretty rare, especially in a stable language.

• Generic types

  You don't seem to have any support for generics yet. May be it is not required in a dynamic language. On the other hand, if you are thinking in a direction of optional typing, it could be useful and more inline with the idea of typing. Eventually, it can be even implemented similar to fields in Shine, i.e. at the library level instead of at the language level.

• Removal of try/catch

  I can understand that current try/catch implementation is probably not the most efficient one. But I think that having a standard error/exception-reporting mechanism in a language/library is very important if any serious applications are to be written in it. Which form this mechanism may have is another question. Do you have any ideas regarding how you'd like to have it?

And my last question:

Have you seen my last comment on the other thread? https://github.com/richardhundt/shine/issues/38 I've asked a few questions and commented a bit. Would be nice to hear your opinions on that.

[richardhundt]

Just wanted to say, first off, that I'm just toying with ideas to see what kind of mileage I get. So not doing any permanent damage to Shine. Yet :)

Yes, streams are nice. While we are at it, it would be nice to think about a functional composition in general. Specifically, I've seen in some other not purely functional languages that they have some issues with curried/uncurried functions when it comes to compositions. It should be possible to convert between these two forms easily and so on. Otherwise writing programs in functional style could be difficult.

In Shine you can already do curry, uncurry and partial application. The trick is using debug::getinfo(f, 'u') to get the number of parameters. It gets tricky with varargs of course, but you should be able to partially apply those.

This is a quick hack taken from a JavaScript implementation:

function Function::__len(f)
   return debug::getinfo(f, 'u').nparams
end

local curry_helper
function curry(f)
   if type(f) != "function" or #f < 2 then
      return f
   end 
   return curry_helper(f, { })
end

function curry_helper(f, args)
   return function(a)
      args1 = { ...args }
      args1[#args1 + 1] = a 
      if #args1 == #f then
         return f(...args1)
      else
         return curry_helper(f, args1)
      end 
   end 
end

function uncurry(f)
   if type(f) != "function" or #f == 0 then
      return f
   end 
   return function(...)
      args = { ... }
      local r = f 
      for i=1, #args do
         r = r(args[i])
      end 
      return r
   end 
end
function test(a, b, c)
   return a * b * c
end

print(uncurry(curry(test))(3, 4, 5))
print(uncurry(curry([3,5])))

For composition, I've got (not sure if it's in master yet):

function Function.__members__.compose(f1, f2)                                 
   return function(...)
      return f1(f2(...))                                                      
   end                                                                        
end
function Function.__members__.andthen(f1, f2)                                 
   return f2:compose(f1)                                                      
end

Does it mean that arrays are now [1,2,3] and tables are [key1:value1, key2:value2]? If so - yes, it makes sense.

Almost. Tables are [[0]=1, 2, 3, answer = 42] which is {[0]=1, 2, 3, answer = 42} in Lua. So they just have [ and ] as outer delimiters (and a Table metatable). The : is reserved for guards.

Question: can I create classes at runtime and assigne them to a variable? Or do I simply define them in a local scope and return the resulting class to achieve that? I.e. can I pass/return a class as an object?

Next on my list :)

Hmm. I'm not 100% sure that I understand the semantics... [destructuring]

What's gone is Scala style named extractors, i.e. let Foo(a, b) = foo. That now becomes let [a, b]: Foo = foo, however, you don't get the __unapply hook. The Foo is optional. If present, it's used during given/case matching, and for destructuring, you'll get a runtime assertion that the right-hand side is of the given type. As before, nesting and guard annotations can be used:

let t = [ answer = 42, [ "a", "b", "c" ] ]
let [answer = a: Number, [ x: String, y, z ] ] = t

re: OctaScript - definitely some ideas worth borrowing there.

You don't seem to have any support for generics yet.

I'll have to give this one a little think. Given the following:

function foo<T>(a: T) { }

You could push the type T into the arguments list as a first parameter. The trickier bit is the assertion. I could say:

function foo<T>(...) {
   let t = T(...) // construct a T with varargs
}

However, this can't just desugar to something like:

function foo(T, ...)
   if T == nil then error("gimme a T damn you!") end
   local t = T(...)
end

... because you could pass it any ol' value for T, and it wouldn't know.

So we could curry it:

function foo(T)
   return function(...)
      local t = T(...)
   end
end

This at least requires you to call it as foo(T)("yadda"). However, you throw a way a closure each time. Memoization could help, but you'd need a cache per function, which seems a bit heavy. For classes things look a little easier (because parameters aren't ambiguous). Need to think about this some more.

I can understand that current try/catch implementation is probably not the most efficient one. But I think that having a standard error/exception-reporting mechanism in a language/library is very important if any serious applications are to be written in it. Which form this mechanism may have is another question. Do you have any ideas regarding how you'd like to have it?

I'm leaning toward Scala's Try ADT and just doing this:

class Try {
   function self.__apply(body, ...) {
      let ok, rv = pcall(body, ...)
      if !ok {
         Failure(rv)
      }   
      else {
         Success(rv)
      }   
   }   
}
// with appropriate monadic Success and Failure classes

... and used as expected:

let t = Try => {
   play_with_fire() // do something dangerous
}

// map it
v = t.map((r) => { ... })

// or pattern match:
given t {
   case [error = e]: Failure { ... }
   case [result = r]: Success { ... }
}

// or alternatively:

let result, error in t
if error {
   print("d'oh!")
}

You get the idea.

Have you seen my last comment on the other thread? #38

I did read it a couple of times and thanks for that. You raised some points which I'm still chewing on. Your question about automatically generating unapply lead me down this rabbit hole which has resulted in all of the above, so apologies for the delay. As you can see, I'm still processing it all :-) I'll reply soon.

[richardhundt]

Just reread your comment on currying, and I think I get it now. Your concern was that it should work well with composition. It'd be easy to compose partially applied functions, obviously, but not sure about curried functions. You'd need to explicitly compose them with uncurry, so that might get in the way.

[pfalcon]

Glad there's such ticket, so I don't need to open another ;-). Why all these custom language syntaxes? Why not implement syntax of well-known language which many people will be glad to use? Zero-based arrays, differentiation between arrays and maps, generators, exceptions - that all smell Python, so why not just implement its syntax?

[richardhundt]

If language popularity is supposed to motivate me, then I'd be implementing Java, JavaScript or PHP syntax. I don't think any of those three are well designed languages.

This project started because people keep popping up on the Lua mailing list with features that they wish Lua had, so I figured I'd just extend Lua in the way that C++ extended C. I've borrowed ideas from several other languages (destructuring assignment, pattern matching, procedural macros), but I've also added something new, and I think there is still room for innovation in programming languages.

Regarding Python's syntax: it lacks flexibility and I value expressiveness over consistency. Take Scala, for instance. It's a wonderfully expressive language because it lets you create your own syntax. I understand that this doesn't appeal to everyone, but it is important to me, because it gives me a way to create a program which not only does the right thing, it also says the right thing. I believe programming languages should be for communicating with people, not just machines.

[pfalcon]

If language popularity is supposed to motivate me, then I'd be implementing Java, JavaScript or PHP syntax.

That's not what I mean surely. But if the language you design appear to have good resemblance to an existing popular language, why not take it as an affinity, and try to make it more similar to it, to open up to more people?

This project started because people keep popping up on the Lua mailing list with features that they wish Lua had

Yup, I guess there're order(s) of magnitude less people who pop up with "if you accept the fact that Lua is not up to par, why try to patch it, instead of just trying to take its critically acclaimed features (which is only LuaJIT in my list) and apply it to a known good language?". So, I'm taking that role ;-). Don't take it too seriously - I'm sure you know all those arguments, what I'm trying to point is that other people who look at your stuff also know them, and wonder why you doing it that way ;-).

I think there is still room for innovation in programming languages.

Sure, that's why big corporations spend so much resources on that. (Or maybe they just take that as an advertisement - how, Mozilla has their NIH language, how Google can live without their NIH thing?)

Take Scala, for instance.

Ok, so why not Scala, but Shine then? ;-)

because it gives me a way to create a program which not only does the right thing, it also says the right thing.

Or just confuses other people? People look and that's what they say: "Hmm. I'm not 100% sure that I understand the semantics."

I believe programming languages should be for communicating with people, not just machines.

My strong belief too. That's why it makes very little sense to enter room where people speak with dozen popular languages and try to speak some of own invention - people just won't understand you at all. Not it makes much sense to insert funky words of own invention in the speech - while it's definitely kewl to do so, most of the time you'll get response "hmm, what do you mean?".

I settled on Python when I figured that the world can't be cured into accepting Lisp (for me Python is still Lisp for real world). First patch I did to Python was to add curly syntax. I submitted it and got response: "But why?". And I thought - indeed, why? With braces, first war will be how to place them. Next war will be about indentation. With Python, someone should be truly anti-social to use something else but 4-spaces indent. The language is very readable, very writable, very cooperatable (there's much less to disagree about or reinvent, as in many other languages). Many implementations exist, but one based on LuaJIT is missing, hint-hint ;-).

[richardhundt]

Thanks for your comments.

That's not what I mean surely. But if the language you design appear to have good resemblance to an existing popular language, why not take it as an affinity, and try to make it more similar to it, to open up to more people?

It has good resemblance to Lua. Python, much less so. And I'm talking about semantics. Syntax is trivial.

Yup, I guess there're order(s) of magnitude less people who pop up with "if you accept the fact that Lua is not up to par, why try to patch it, instead of just trying to take its critically acclaimed features (which is only LuaJIT in my list) and apply it to a known good language?". So, I'm taking that role ;-). Don't take it too seriously - I'm sure you know all those arguments, what I'm trying to point is that other people who look at your stuff also know them, and wonder why you doing it that way ;-).

I wouldn't say Lua is "not up to par". Lua is an excellent embedded language with more deployments than you realise (World of Warcraft, lots of routers and other devices, Angry Birds, Wikipedia's rendering, etc.).

I've spent a fair bit of time and effort trying to bend the LuaJIT runtime into a language with different semantics:

https://github.com/richardhundt/melody https://github.com/richardhundt/gaia/tree/master/lang/kudu https://github.com/richardhundt/lupa

The language which got the most traction is Shine. What I've learned is that if you want it to be efficient, you've got to use what the VM gives you. This constrains your semantics. As to why I'm doing it the way I am? What in particular do you find strange?

Sure, that's why big corporations spend so much resources on that. (Or maybe they just take that as an advertisement - how, Mozilla has their NIH language, how Google can live without their NIH thing?)

Not just big corporations. Martin Odersky, the Julia crowd, Giancarlo Niccolai (www.falconpl.org), even Rob Pike produced several languages before Go (Newsqueak, Limbo). I'm doing it for fun. I like creating new things :-) In the end, if someone else uses it, then that's a really nice feeling, but if not, it's still worth it for me.

Ok, so why not Scala, but Shine then? ;-)

I use Scala professionally, but I think the world can use a kind of light-weight, dynamic scripting version of Scala, which compiles fast and doesn't run on a monster like the JVM.

Or just confuses other people? People look and that's what they say: "Hmm. I'm not 100% sure that I understand the semantics."

That's a bit out of context as romix has been contributing to Shine's development for ages, so when I'm talking to him, I don't always explain myself fully. If you let me know what in particular you find confusing, then I'm happy to either change it if I agree, or document and explain it if I don't.

[Python] Many implementations exist, but one based on LuaJIT is missing, hint-hint ;-).

Nothing stopping you there. Shine itself is written in Lua, so you can take the code base and start there. Lua is nothing if not simple and easy to learn. Alternatively there's https://github.com/franko/luajit-lang-toolkit as romix mentioned. hint-hint ;-)

[pfalcon]

It has good resemblance to Lua. Python, much less so. And I'm talking about semantics. Syntax is trivial.

Yeah, and people seems to be exploring only easy ways ;-).

I wouldn't say Lua is "not up to par".

Sure, it must the most perfect language which has everything needed. That must be the reason that there're gazillion of Lua forks and dialects - it's just because it spurs creativity, and not because every advanced user finds it lacking, lacking, lacking.

Lua is an excellent embedded language with more deployments than you realise

My hypothesis is that people use Lua due to lack of a better alternative.

I've spent a fair bit of time and effort trying to bend the LuaJIT runtime into a language with different semantics:

Thanks, I will look into them, and that's really productive outcome I can get out of this discussion.

Not just big corporations.

True, every freshman student does that too.

but I think the world can use a kind of light-weight, dynamic scripting version of Scala, which compiles fast and doesn't run on a monster like the JVM.

Yes, sure, that's what I mean - another implementation of Scala, using LuaJIT tech. (It's LuaJIT which interests us in all this, right? - if there weren't LuaJIT, hyped to "run faster than C", there would be much less interest in Lua).

If you let me know what in particular you find confusing, then I'm happy to either change it if I agree, or document and explain it if I don't.

Well, as I hinted, I'm just trying to provide 100% opposite black&white arguments against "I'm designing my own cool language" approach. I surely understand fun, research, etc. value in it. Just exploring the situation "what if, what if it's all wrong, and working on elaborating good existing language is a better approach".

Nothing stopping you there.

Well, it seems that not just me, but a lot of other people find blockers on that way. Mike Pall pretty clearly stated (based on mailing list reading) that LuaJIT is not written for extensibility, and he's not interested in it being so. Looking at its source, one of the first things one sees is:

      case H_(64a9208e,8ce14319): case H_(8e6331b2,95a282af):  /* aligned */

(i.e. magic numbers in the source), and one doens't need to be a big conspiracy theorist to imagine that code is actually written for obfuscation.

Alternatively there's https://github.com/franko/luajit-lang-toolkit as romix mentioned. hint-hint ;-)

Yes, so that's what I've seen first, and relation of that and your project(s) I'm trying to decipher now. My initial impression was that he forked your language, renamed it and develops further. Turns out that actually he's stuck with outdated fork of it. Then shine now uses tvmjit, while franko/nyanga uses luajit-lang-toolkit. How comes? Was franko/nyanga forked before switch to tvmjit, and @franko just keeps digging old code, or was it switched from wanna-be-generic solution back to Lua on purpose (perhaps because writing it in language where typos to variable names cannot be detected and size of array cannot be reliably known make it more fun to develop and make it more clear to other people).

[agladysh]

Please don't say "every". I have over 600KLOC here, and I'm happy with the language.

[pfalcon]

That explains why you're lurking in obscure discussions in Lua fork projects - to say that you don't need all those forks, I presume ;-).

[agladysh]

I did not say that I don't need them, they are one of the ways to look for the innovation in the mainstream language. What I did say is that, while being advanced user of the language, I do not find it lacking at all.

But lets not flame here, you have the right to think otherwise. ;)

[richardhundt]

Yes, sure, that's what I mean - another implementation of Scala, using LuaJIT tech. (It's LuaJIT which interests us in all this, right? - if there weren't LuaJIT, hyped to "run faster than C", there would be much less interest in Lua).

Nope. What drew me to Lua initially was that I was actually looking for an extensible register-based VM which supported coroutines, for a language I was developing. I was writing bytecode compilers for Lua 5.1 a while before LuaJIT2 came out.

My initial impression was that he forked your language, renamed it and develops further. Turns out that actually he's stuck with outdated fork of it. Then shine now uses tvmjit, while franko/nyanga uses luajit-lang-toolkit. How comes? Was franko/nyanga forked before switch to tvmjit, and @franko just keeps digging old code, or was it switched from wanna-be-generic solution back to Lua on purpose (perhaps because writing it in language where typos to variable names cannot be detected and size of array cannot be reliably known make it more fun to develop and make it more clear to other people).

Francesco added a ton of fixes and did some really amazing stuff with it. I switched to tvmjit because maintaining the bytecode generator was distracting me from prototyping my language. He's got the code base in such good shape now, that I've actually been toying with the idea of compiling tvmjit S-expressions using his code generator as a back-end. That way, if you want to run Shine on vanilla LuaJIT, you can.

[shortweekend]

Francesco added a ton of fixes and did some really amazing stuff with it. I switched to tvmjit because maintaining the bytecode generator was distracting me from prototyping my language. He's got the code base in such good shape now, that I've actually been toying with the idea of compiling tvmjit S-expressions using his code generator as a back-end. That way, if you want to run Shine on vanilla LuaJIT, you can.

That would be awesome, Richard! I'm really looking forward to using Shine on vanilla LuaJIT.

BTW Shine's syntax is already very pleasant. Not everyone likes Python's syntax or curly braces ;)

[richardhundt]

That would be awesome, Richard! I'm really looking forward to using Shine on vanilla LuaJIT.

Technically you can already run in on vanilla LuaJIT (same bytecode). It's just not terribly easy given the package.loaders hook which compiles Shine to TP expressions, so require won't just work as you'd expect.

I've got the luajit-lang-toolkit bytecode generator turning a TP expression tree into bytecode. Here's a sample:

local ast = tp.chunk {
   tp.op{'!loop',
      tp.id'i', tp.op(1), tp.op(10), tp.op(1);
      tp.op{'!call', tp.id'print',
         tp.op{'!mconcat',
            tp.op"Hello World",
            tp.op"!",
            tp.id'i'
         }   
      }   
   }   
}

local code = generate(ast, "test")
local func = assert(loadstring(code, "@test"))
func()

However, while doing this I started thinking that if portability was a concern, then producing Lua source code would be ideal. When I started the bytecode generator, I had two problems I couldn't easily solve by compiling to Lua 5.1 code:

• unconditional branching for continue and break in loops
• control over the debug bytecode segment line numbers for error messages.

Since Lua 5.2 has goto, branching is no longer a problem (Shine would depend on LuaJIT 2 being compiled with 5.2 compat turned on).

That leaves the debug segment. Source maps are another way to solve this. Making this work in Lua would probably mean wrapping the main chunk in an xpcall and custom generating the stack trace by consulting a source map.

You'd then run Shine like so:

$ luajit -lshine.run ...

I'll spend some time over the holidays tinkering with it.

[shortweekend]

Great news, Richard. I noticed some time ago that TvmJIT already had Lua 5.2 compatibility turned on: XCFLAGS+= -DLUAJIT_ENABLE_LUA52COMPAT in tvmjit/src/GNUmakefile, and indeed some features (I don't remember which anymore) would not work without that. Thank you and best wishes for the holidays and the year ahead!