Make Ceylon scriptable

[FroMage]

Needs some thinking

[gavinking]

The real basic problem with supporting Ceylon scripts is that since we've just decided that there is no well-defined lexical ordering of toplevel declarations, it doesn't make sense to write procedural code at the top level. So whatever a "script" is, it's not just some statements written at the package scope. i.e. whatever it is, it's some kind of method.

[gavinking]

So a syntactic solution that would work is the following:

void {
    print("Hello!");
}

i.e. this syntax gives you the ability to specify which bits of the file contain procedural code. Declarations can go outside.

void hello() { 
    print("Hello");
}

void bye() { 
    print("Goodbye");
}

void {
    hello();
    bye();
}

This gives us a competing solution to the idea of a module run() method.

[gavinking]

This idea is growing on me. How do you guys feel about this being our helloworld program?

void { print("hello world"); }

[quintesse]

The syntax looks okay to me, but it won't be enough to make Ceylon be seen as "scriptable". I think the following things should be possible before that:

1. Run an uncompiled source file (maybe using a different extension than .ceylon? it will be compiled on-the-fly)
2. Run (as if) from the default module (no extra package or module files necessary)
3. Access other modules (the fact that we can't access other modules from the default module severely limits the usefulness of the default module)

Point 1 should make it possible to use the typical UNIX script hash bang, eg. #!/bin/ceylon run and in that case we could probably make the void { } implicit. Another problem of course is caching, should we keep the compiled code around and if so how do we detect that we already previously compiled the script file and that it is available in the local repository?

Point 3 of course comes with the problem that when using other modules from a script file you would need to provide a module file, but that defeats the purpose a bit of having a script file IMO.

So right now I'm not really sure how to turn this into something useful :(

[gavinking]

I don't think we're going to be able to get this done for Ceylon 1.0. We should address scripting for Ceylon 1.1.

[RossTate]

This could be done pretty snazzily with gradual typing, which was one of the projects we talked about doing for that multiuniversity class I'm a part of.

[gavinking]

Here's what I now think a "script" should look like:

#!/bin/ceylon run --compile

import ceylon.time "1.0.0";
import ceylon.time { today }

"Print the day of the week"
void {
    print("Today is ``today().dayOfWeek``");
}

[quintesse]

1. I'd try to leave out any command line options if possible and have the ceylon command auto-detect it's running a script as much as possible. That way later on we can change how we handle scripts without breaking older ones.
2. how about the idea you had before with combining the 2 imports, like import ceylon.time "1.0.0" { today } ? Needing two just seems a bit weird, doesn't it?

Also, would this syntax be supported outside scripts? (If not, why not? People will ask. Also, think of the default module)

And will we have a special file suffix for scripts? Let's say .ceylonscript or .ceylons or .ceyscript or whatever? (remember that for Linux/Mac the #!/bin/ceylon works but for windows we can only depend on the file suffix).

[gavinking]

how about the idea you had before with combining the 2 imports

Well one a module can contain multiple packages.

[quintesse]

Well one a module can contain multiple packages.

True.

Still, looks weird :)

[HenningB]

How about explicitly declaring the imports, as they're different things:

import module ceylon.time "1.0.0";
import package ceylon.time { today }

Or the other way round:

module import ceylon.time "1.0.0";
package import ceylon.time { today }

First one sounds more "natural", the second one more "computerized".

One thing that could be troublesome for beginners: the module import has a semicolon, the package import doesn't have it.

[gavinking]

@HenningB Well, but then there would be a different syntax in a script than in an ordinary source file or module descriptor.

[HenningB]

Then let's make it an anonymous module

module {
    import ceylon.time "1.0.0";
}
import ceylon.time { today }

That's even closer to the original syntax than

import ceylon.time "1.0.0";
import ceylon.time { today }

Or only allow "default" "unversioned":

module "default" "unversioned" {
    import ceylon.time "1.0.0";
}
import ceylon.time { today }

But I prefer the anonymous module descriptor more.

[quintesse]

Like that idea actually. It's more verbose but does seem to capture the idea better.

[jcllings]

So long as you are not going to use something lame like "main". "main" tells you nothing about what the method does or is. Main? Main what? We have more than 8 characters available but they (Java) could at least have used something like "start" or "begin" or even "go".

[gavinking]

@jcllings the convention in Ceylon today is to use run().

[lucaswerkmeister]

Is the anonymous function still the consensus on where the code should go? Because I really like how similar @HenningB’s anonymous module and @gavinking’s anonymous function are:

module /* implicit name and version */ {
    import ceylon.time "1.0.0";
}

void /* implicit name and parameter list */ {
    print("Today is ``today().dayOfWeek``");
}

Also, that void suggests that you could also put any type there – maybe the script runner could print the return value if the function declares one?

[djadmin]

I would like to contribute to it, using antlr it would quite easy to make it scriptable. Would like to discuss complete workflow before implementing it.

[quintesse]

@djadmin Well as you see in this discussion I think the first issue to solve is not a technical one, but a design issue. Meaning, what should scripting code look like, what syntax do we want/need? We (core team) probably need to take some time and decide how to progress on this one.

[djadmin]

Yeah ! That's true. Please try to notify on this thread once the team decides the code structure and the functionalities to be included, so that I could discuss after that.

[djadmin]

We need a thorough discussions on deciding the scriptable syntax for Ceylon. From the above conversations, I feel the following structure :

#!/bin/ceylon run --compile

import com.myexample { ... } ;
import com.myexample {test, mytest} ;
import com.myexample {Test = mytest} ;
import ceylon.time ; 

void {
    //statements
}

Please correct me if I made any mistake :)

[gavinking]

@djadmin the trouble with that syntax is it doesn't let us declare dependencies to modules. So I think we need a structure more like this:

<module name declaration>
<module imports>
<statements>

That is, something approximately like this:

module myscript "1.0.0" {

    import ceylon.collection "1.1.0";
    import ceylon.file "1.1.0";

    import ceylon.collection { ArrayList, HashMap }
    import ceylon.file { File, Directory }

    class MyClass() { ... }
    MyClass().doSomething();

}

One thing that's never been very clear to me is whether declarations in the body of a "script" are treated like real toplevel declarations, or like local declarations in the body of a method. It seems to me that the second option probably makes a little more sense, meaning that we would basically treat the body of this module declaration as belonging to an explicit run() function. That is, the above would be equivalent to:

module myscript "1.0.0" {

    import ceylon.collection "1.1.0";
    import ceylon.file "1.1.0";

    import ceylon.collection { ArrayList, HashMap }
    import ceylon.file { File, Directory }

}

shared void run() {
    class MyClass() { ... }
    MyClass().doSomething();
}

WDYT?

[lucaswerkmeister]

@gavinking: Why the module myscript "1.0.0" wrapper if you can never use the module anywhere else? I would prefer

import ceylon.collection "1.1.0";
import ceylon.file "1.1.0";

import ceylon.collection { ArrayList, HashMap }
import ceylon.file { File, Directory }

shared void run() {
    class MyClass() { ... }
    MyClass().doSomething();
}

This syntax still looks unambiguous to me.

[lucaswerkmeister]

Another thing I’ve been thinking about is that a Read-Eval-Print Loop (REPL) would be nice to have. That’s a feature that most scripting languages have where if you just type python (or clisp, bash, irb, js, whatever), you get a prompt, and each command you enter (“read”) is run immediately (“eval”), and the result is pretty-printed for you (“print”). It’s a really nice tool for playing around with and exploring a language without the need for an IDE or other heavyweight stuff like that. (And in Ceylon, we could implement awesome tab completion and stuff.)

One important thing about REPLs is that their syntax is identical to the regular syntax of the language: python script.py and python < script.py (where the latter form simulates a REPL run, but with script.py’s lines as input) are identical. If we used a void { ... } or void run() { ... } syntax, the user would need to enter that in the REPL, or ceylon script file.ceylon and ceylon.script < file.ceylon would need to behave differently. I don’t really like any of these variants.

(Of course, we could also have a ceylon repl subcommand.)

[lucaswerkmeister]

If we had a REPL, I would like the following syntax:

import ceylon.collection "1.1.0";
import ceylon.file "1.1.0";

import ceylon.collection { ArrayList, HashMap }
import ceylon.file { File, Directory }

class MyClass() { ... }
MyClass().doSomething();

or, if you just wanted to play around:

print("Hello, ``process.arguments.first else "World"``!");

That is, in a script toplevel declarations and statements would have a defined order. (By the way, @gavinking: I don’t think you explained why you “decided that there is no well-defined lexical ordering of toplevel declarations”; is it because you can’t enforce the order in which the myValue_ classes are loaded by the JVM?)

[lucaswerkmeister]

We would basically treat the body of this module declaration as belonging to an explicit run() function.

^source FTR, this is what the Ceylon Web IDE does, and this causes, for example, ceylon/ceylon-web-ide-backend#58.

[djadmin]

@gavinking Thanks for correcting me, I believe the second one makes sense syntactically but most of the scripting languages do treat it as real top-level declarations afaik which is why It's easier to write needlessly worrying about the blocks as @lucaswerkmeister mentioned.

@lucaswerkmeister I think that idea will be useful, we should show the pretty-printed form of output.

[djadmin]

If we can deal with this syntax, would be great

import ceylon.collection "1.1.0";
import ceylon.file "1.1.0";

import ceylon.collection { ArrayList, HashMap }
import ceylon.file { File, Directory }

class MyClass() { ... }
MyClass().doSomething();

Otherwise putting them into module block is also fine, It makes sense:

module myscript "1.0.0" {
import ceylon.collection "1.1.0";
import ceylon.file "1.1.0";

import ceylon.collection { ArrayList, HashMap }
import ceylon.file { File, Directory }

class MyClass() { ... }
MyClass().doSomething();
}

[gavinking]

@gavinking: Why the module myscript "1.0.0" wrapper if you can never use the module anywhere else?

Well, OK, perhaps you're right, perhaps that's not completely necessary. We could infer the module from the location of the script, or treat it as belonging to the default module, I suppose.

Another thing I’ve been thinking about is that a Read-Eval-Print Loop (REPL) would be nice to have.

To me, a REPL is a way, way less useful feature, and much harder to implement. A REPL is way harder to use than an open editor in the IDE with code in it, and I assume that people like REPLs only because they use languages without decent IDEs.

Anyway, let's not make @djadmin's life harder by introducing new requirements.

[gavinking]

I don’t think you explained why you “decided that there is no well-defined lexical ordering of toplevel declarations”; is it because you can’t enforce the order in which the myValue_ classes are loaded by the JVM?)

That, definitely, and also we let you have multiple source files in a single package, all with toplevel declarations. We don't force all code in a package into a single source file. So where would the order of things declared in different source files come from?

[djadmin]

Anyway, let's not make @djadmin https://github.com/djadmin's life harder by introducing new requirements.

ha ha , not a problem :) If it's really a need I'd be happy to write the code for that.

Dheeraj

[lucaswerkmeister]

I assume that people like REPLs only because they use languages without decent IDEs.

I think the main use is really just for playing around; I can type sudo apt-get install clisp; clisp and just get started immediately without needing an IDE. Of course, for serious work, the IDE’s tooling is invaluable.

Anyway, let's not make @djadmin's life harder by introducing new requirements.

No, of course not :) But if we decide on a REPL-friendly syntax, then a first version of the REPL is really trivial (the “read” and “eval” is both covered by making Ceylon scriptable; we just need to know when a statement is finished and print it). But yeah, it can wait.

That, definitely, and also we let you have multiple source files in a single package

Ah, of course. But this is not a problem in scripts, since scripts are only one file.

[gavinking]

but most of the scripting languages do treat it as real top-level declarations afaik which is why It's easier to write

But the semantics of a scripting language are designed to allow this. If you write this in a scripting language:

 MyClass().doSomething();
 class MyClass() {}

You get a runtime error when the first line is executed.

The semantics of Ceylon are very different, and there is a good reason for the difference between toplevels and nested declarations. In Ceylon, you can't get a runtime error telling you that a class declaration has not yet been executed!

So, since a script is an executable block of code, it makes most sense to treat it as the body of a method.

Now, if we wanted to make declarations toplevel, we could do that, but it would mean requiring that they occur at the start of the script, before any executable statements.

[gavinking]

If it's really a need I'd be happy to write the code for that.

I think you're underestimating just how hard this would be. There's no easy way to redefine a class in the JVM. You would need to do some pretty tricky messing about with classloaders to make it work.

[gavinking]

I think the main use is really just for playing around; I can type sudo apt-get install clisp; clisp and just get started immediately without needing an IDE

Have you ever tried to write a class in a REPL? Or a function with more than one line of code in it? I have. It's painful!

[lucaswerkmeister]

Have you ever tried to write a class in a REPL?

This already goes beyond my understanding of “playing around” :) But I have to admit that I’ve never really used a REPL for more than 5 minutes, so I’m not the best person to talk about them.

[gavinking]

BTW, I want to make explicit a requirement that I'm taking as a given: we do not want to introduce any new kind of language construct or new semantics that would require changing the language specification. Whatever we decide here, it should be definable in terms of a simple syntactic transformation to the language that already exists. That's very important!

[gavinking]

Have you ever tried to write a class in a REPL?

This already goes beyond my understanding of “playing around” :)

If the "playing" is so trivial that you're not even going to write a class or nontrivial function, I don't see what the point of it is. You're not going to learn anything useful.

But I have to admit that I’ve never really used a REPL for more than 5 minutes, so I’m not the best person to talk about them.

Seriously, do try it, REPL's suck ass, especially for OO languages.

[djadmin]

Okay as mentioned it's going to be really tough, let's first focus on making it scriptable (necessary syntaxes first) then we can think about implementing other add-on functionalities. And yeah we should make it as simple as possible.

[lucaswerkmeister]

@gavinking: Well, I use Firefox’ JS Scratchpad (Shift+F4) often-ish, but that already goes beyond a REPL (it has eval+print, but you’re editing a document instead of just appending lines). But that is pretty useful (especially with the built-in Inspector – again, this is way beyond a REPL).

The new requirement rules out a REPL as the Ceylon script interpreter (ceylon script script.ceylon == ceylon script < script.ceylon), so let’s forget that.

[gavinking]

We would basically treat the body of this module declaration as belonging to an explicit run() function.

FTR, this is what the Ceylon Web IDE does, and this causes, for example, ceylon/ceylon-web-ide-backend#58.

Interestingly, if scripts are one-shot things, and you can't run a script twice in the same VM, we could, in theory, allow you to define an enumerated type within the body of the scripts executable code.

[lucaswerkmeister]

Yes, because the objects are still singletons. But that’s new semantics, isn’t it? ;-)

[gavinking]

@gavinking: Well, I use Firefox’ JS Scratchpad (Shift+F4) often-ish, but that already goes beyond a REPL (it has eval+print, but you’re editing a document instead of just appending lines).

That is definitely not a REPL. It has much more in common with creating a Ceylon source file in the IDE and running it from the editor.

[gavinking]

Yes, because the objects are still singletons. But that’s new semantics, isn’t it? ;-)

Strictly speaking yes, it in practice it would mean suppressing a single error check.

[quintesse]

Gavin, I commented some days ago on another thread where enumerated types were discussed but I never got an answer there so I'll repeat it here:

Just as a question : couldn't we make it so that even when [case types] come from different scopes they would still be considered the same?

That way we'd prevent the exhaustion problem while making it possible to use non-toplevel cases, right?

It just seems like a worthwhile thing to have especially because I suspect that in most cases people don't want or need to share case types between scopes.

It's also one of the few things where our scoping doesn't seem regular.

Or are there still other problems I'm not aware of?

Cheers

-Tako

On Mon, May 19, 2014 at 7:28 PM, Gavin King notifications@github.comwrote:

We would basically treat the body of this module declaration as belonging to an explicit run() function.

FTR, this is what the Ceylon Web IDE does, and this causes, for example, ceylon/ceylon-web-ide-backend#58https://github.com/ceylon/ceylon-web-ide-backend/issues/58 .

Interestingly, if scripts are one-shot things, and you can't run a script twice in the same VM, we could, in theory, allow you to define an enumerated type within the body of the scripts executable code.

— Reply to this email directly or view it on GitHubhttps://github.com/ceylon/ceylon-spec/issues/200#issuecomment-43532542 .

[lucaswerkmeister]

@quintesse: So they could be the same without being equals? (I can stick arbitrary, possibly mutable state in them, I can override equals, etc.)

[quintesse]

@lucaswerkmeister Yes, doesn't make much sense, does it? Hmmm I just have the feeling there should be some way to have non-toplevel enumerated types. Anyway, not the place to discuss this really. Shouldn't have mentioned it here, I thought it was just an email thread, not an issue.

[jcllings]

Suggestion: In order to avoid Java's modularity problems while still maintaining some of the benefits of using interfaces, suggest using a version range rather than a specific version. This is what OSGI does and I think it's what is intended in Java 9. Also, what is the reason for quotes? What possible values can a version have? What about values like 1.0.12r.68.A?

Jim C.

On 05/19/2014 10:04 AM, Dheeraj Joshi wrote:

If we can deal with this syntax, would be great

|import ceylon.collection "1.1.0"; import ceylon.file "1.1.0";

import ceylon.collection { ArrayList, HashMap } import ceylon.file { File, Directory }

class MyClass() { ... } MyClass().doSomething(); |

— Reply to this email directly or view it on GitHub https://github.com/ceylon/ceylon-spec/issues/200#issuecomment-43529966.

[lucaswerkmeister]

Module version ranges were discussed in https://groups.google.com/forum/m/#!topic/ceylon-dev/n8a1pbFKrHA (and probably elsewhere); they're not really related to this issue specifically.

Quotes around versions because they're mostly arbitrary strings (OTOH, module names are identifiers, so they have no quotes).

----- Ursprüngliche Nachricht ----- Von: "Jim" notifications@github.com Gesendet: ‎20.‎05.‎2014 09:24 An: "ceylon/ceylon-spec" ceylon-spec@noreply.github.com Cc: "Lucas Werkmeister" mail@lucaswerkmeister.de Betreff: Re: [ceylon-spec] Make Ceylon scriptable (#200)

Suggestion: In order to avoid Java's modularity problems while still maintaining some of the benefits of using interfaces, suggest using a version range rather than a specific version. This is what OSGI does and I think it's what is intended in Java 9. Also, what is the reason for quotes? What possible values can a version have? What about values like 1.0.12r.68.A?

Jim C.

On 05/19/2014 10:04 AM, Dheeraj Joshi wrote:

If we can deal with this syntax, would be great

|import ceylon.collection "1.1.0"; import ceylon.file "1.1.0";

import ceylon.collection { ArrayList, HashMap } import ceylon.file { File, Directory }

class MyClass() { ... } MyClass().doSomething(); |

— Reply to this email directly or view it on GitHub https://github.com/ceylon/ceylon-spec/issues/200#issuecomment-43529966.

— Reply to this email directly or view it on GitHub.

[RossTate]

Maybe a good intermediate goal here would be to Ceylon scriptable except for defining new classes and interfaces. Decent support for aliases, records, and sum types should cover most of the use cases you'd run into in a script. And by employing this restriction we can prevent the need for generating new classes/interfaces at run time.