static methods and attributes

[gavinking]

So, I'm throwing this one out there to see what ya'll think.

A long time ago I decided that Ceylon would not have static methods. The reason for this was that I always hated them in Java: in order to declare a regular function, you were forced to define a rubbish class with no instances, just to stick the function on it, and then to call that function, you had to do something nastily verbose like Integer.parseInt(). These functions had nothing to do with the schema of any type, and they don't have access to the instance-level state of the class, they are just stuck onto the side of a class for no good reason.

Some things happened since that time to make me want to reevaluate this:

1. We had to anyway build almost everything needed to support static methods into the typechecker in order to support Java interop.
2. We added constructors, which sorta look like static methods. In particular, they are invoked on a class name, instead of an instance.
3. People started asking for "factory functions", which are, essentially, a particular sort of static method.
4. I figured out how to resolve the "don't have access to the instance-level state" conundrum: simply require that static members be declared at the beginning of a class with constructors, before any instance-level state is allocated.
5. I started writing lots of code that interoperates Java with Ceylon and found it sorta weird that I was totally comfortable calling Java static methods in Ceylon, but couldn't actually write a static method in Ceylon.
6. I started wanting to define a constant values shared between all instances of a class, without polluting the package namespace.
7. I ran into a (anti-)pattern used in IntelliJ APIs where a service implementation is supposed to define a certain specially-named static member. I couldn't do this in Ceylon, so this class had to be written in Java.
8. I realized that there actually is one very good motivation for static: static members can bypass the visibility restrictions that are imposed on toplevel functions, and access private state of instances of the class. This is unambiguously useful, especially since we don't (yet) have package-private.

Thus, at this point, it would be sooo easy to add static methods and attributes (it basically amounts to one new annotation), that it's hard to see why we shouldn't. It would look like this:

class Integer {

    shared static Integer|SyntaxError parse(String string) => internalParse(string);
    shared static Integer sum({Integer*} integers) => integers.fold(0)((s,x)=>s+x);

    shared new (Integer i) { ... }

    ...

}

Note that I don't see why interfaces couldn't have static members. So this need not just be for classes.

So, pros and cons of doing this include:

Pros:

• it makes it slightly easier to port Java code to Ceylon (paste-Java-as-Ceylon will work a little better)
• it results in the more-expressive Integer.parse(text) and Integer.sum(totals) instead of parseInteger(text) and integerSum(totals)
• it's easier to "discover" Integer.parse() and Integer.sum() in the IDE than it is to discover parseInteger() today
• it provides a nice way to share private state/operations between a class and functions which operate on its instances,
• it provides a way to define class-local constants,
• it probably resolves the perceived need for "factory functions"
• it makes constructors look less "irregular", since static members and constructors share a essentially the same rules
• it slightly eases the cognitive tax of programming in mixed Java/Ceylon projects, since that's one more construct that maps across cleanly (ditto for JavaScript where you have stuff like Math.xxxx)
• it might make some things from Dart and Typescript easier to represent @jvasileff @lucaswerkmeister wdyt?

Cons:

• it might encourage Java-isms of creating rubbish classes just to stick functions on them (but it's easy enough to tell people to use object instead of class for that
• the people (person?) who think(s) that introducing constructors were a terrible mistake that makes the language less regular might perceive this as doubling down on that irregularity rather than resolving it by make it less irregular
• it's an additional language feature that is clearly not a critical thing we have to have
• it might be more difficult to implement in Dart and JS than it is in Java (I doubt this)
• I would have to admit I was wrong about something

Thoughts?

[lucaswerkmeister]

We should probably think about how this plays together with #4005 (most recent proposal in https://github.com/ceylon/ceylon/issues/4005#issuecomment-156656948), to make sure we’re not putting any obstacles in our own way for when we decide to address that issue. What’s the relation between static type members and polymorphic type members?

[gavinking]

@lucaswerkmeister yes, good point.

[gavinking]

@lucaswerkmeister In fact it looks to me like this is just a subset of what I've proposed in #4005 and would be small step towards finally having type classes. So probably belongs as an additional Pro.

[FroMage]

I'm not against this, but I feel it makes classes more complex. It's also "more than one way to do it" because you can just as well scope all statics of Foo into a companion object foo, which perhaps we haven't made an idiom enough. Probably parseInteger and integerSum should have been integer.parse and integer.sum.

On the other hand, given the visibility restrictions and access to non-shared members that static members would get, we'd have to add the concept of friend objects to gain the same with object companions:

friend(`object foo`)
shared class Foo(){
 Integer x = 2;
}
shared object foo {
 shared void accessX(Foo f){
  print(f.x); // I can because I'm a friend
 }
}

Surely this can be useful, especially when we eventually do this for modules (which we'll need to), but at this point static members are much less boilerplate than this.

So all in all, I guess I'm neither for nor against ;)

If we do implement this, we're going to have to do something magic to many static member names, such as main and anything which can end up a static method we use for constructors.

[gavinking]

Personally it seems to me that your "friend objects" (called "companion objects" in Scala and Kotlin) are a more complex / less elegant / inferior approach than just having static methods.

If we do implement this, we're going to have to do something magic to many static member names, such as main and anything which can end up a static method we use for constructors.

I don't see why: constructors and functions share the same namespace so their names can't collide.

[FroMage]

I don't see why: constructors and functions share the same namespace so their names can't collide.

For the JVM compilation I mean, since we generate statics underneath. Especially one called main for classes with no args.

[gavinking]

Oh, right, sure. main would surely need to be escaped. I imagine we already do that escaping for constructors called main, so it should not be a big deal.

[gavinking]

it probably resolves the perceived need for "factory functions"

It properly solves the need for factory functions only if the Java backend also generates a static method for each constructor, along with invocations of that static method instead of newing the class directly. Currently it does not. But that doesn't seem like it would be hard to fix at all.

[lucaswerkmeister]

I don’t understand the factory functions point at all yet. What’s the difference between a static factory function and a named constructor?

[jvasileff]

@lucaswerkmeister a factory function can return 1) a subclass, or 2) a singleton or cached instance, for something like Whole.fromInteger(0).

Edit: and 3) can leak this when performing additional initialization.

[jvasileff]

I believe the "can bypass the visibility restrictions" aspect alone is more than enough justification to do this. But I also agree with @lucaswerkmeister that we need to be very careful to avoid future problems with type classes.

Re: Dart, I don't think there are any specific non-obvious concerns or advantages.

[chochos]

I think to implement static methods in JS all we need to do is to attach the method to the class function instead of the prototype. The problem could be on invocations, the right class needs to be called, but that should be trivial.

What about static attributes? Are we going to have those as well?

[gavinking]

I think to implement static methods in JS all we need to do is to attach the method to the class function instead of the prototype.

Right, should be as simple as that.

The problem could be on invocations, the right class needs to be called, but that should be trivial.

I assume that is trivial or already working, since today when I write this code:

dynamic {
    print(Math.sqrt(2.0));
}

I already get the output 1.4142135623730951.

[gavinking]

What about static attributes? Are we going to have those as well?

Sure, it would be for both methods and attributes. Use cases for static attributes are stuff like Integer.zero, etc. (This becomes even more interesting when you consider type classes.)

What I'm not certain about is static nested classes. If they're easy enough to implement, I guess I would allow them for the exact same reasons that we're proposing static attributes/methods (more regular, less conceptual gap from Java, access to private members, etc).

However, if they're for some reason more difficult to implement on one of the three backends, I certainly don't think they're something we have to have now.

And what about static nested interfaces?

[gavinking]

I have pushed a rough implementation of the typechecker side of this to #6515.

To support this in the JVM backend, I think that pretty much all we would have to do is make the code generator mark members as static if Declaration.isStaticallyImportable() is set. i.e. it seems like it would be a one-liner. Does someone want to point me to where in the code I would need to do that, so we can try this out?

[gavinking]

Hrm, and what about static object? Feels potentially useful...

[gavinking]

By the way, in my list of reasons for supporting static, I forgot to mention something I ran into multiple times when developing the IntelliJ plugin: the desire to define a constant value that is shared between all instances of a class, without polluting the package namespace. With the current semantics of Ceylon, all true constants must be toplevels, or declared to belong to an object. That's actually pretty uncomfortable.

[gavinking]

I have implemented basic support for this in the JVM backend, but I've run into a problem with initialization of static fields. For code like this:

class Stuff {
    static value name = "Trompon";
    shared new() {}
}

What the JVM backend currently tries to do is move the initializer into the constructor of Stuff, which isn't right. @tombentley would you help me fix that?

[gavinking]

However, this raises an interesting point. Consider this code:

class Stuff {
    static value name;
    name = "Trompon"; //currently an error
    shared new() {}
}

One might think that this is perfectly reasonable code. Within the body of a class we can typically split declaration and initialization.

However, the body of a class is something we consider to be "executed" at instantiation time, so the above is not something I think we should allow (we would have to implicitly hoist certain initializer statements into a static initializer, but I just don't think we should go there).

So I think the same restriction should apply to static fields that applies to toplevel fields: that they must have an initializer. However, even this is something one might object to: you can do a whole lot of processing in an expression in Ceylon, and one might object that it's wrong that this processing doesn't occur at the same time as initializer execution. In which case one might argue that static fields should be disallowed completely. I could go along with that reasoning, except for the fact that I just identified constants as a key motivation for wanting static.

[lucaswerkmeister]

You could restrict constant initializers to CaseValues: “string literals, character literals, integer literals, negated integer literals, value references to value constructors, and/or value references to anonymous classes”. That’s an existing notion in the language, it makes sense with the “constants” use case, and it avoids running any code at initialization time.

[gavinking]

@lucaswerkmeister That's an option, yes. But one of the practical usecases for this was to have a constant holding an empty Java array of a certain element type.

[tombentley]

@gavinking I've added support for static attributes.

I found a tc problem wrt assign declarations of static getters: The tc sees them as ending the decl section so the moans if you declare another static member:

class C() {
    shared static String f => "";
    assign f {}
    shared static String g => "";
    assign g {}
    shared new () {}
}

[gavinking]

@tombentley great, thanks!

[gavinking]

So, @tombentley @bjansen this is now all working, it seems, in Eclipse, but in IntelliJ I get errors like this:

Error:(7, 32) ceylon: Ceylon backend error: non-static variable this cannot be referenced from a static context

Which I don't understand at all, after eyeballing the generated code. I suspect the problem might be model-loader related (or mirror-related) but I can't seem to pin it down. The code I'm testing is this:

shared class Hello {
    static String greeting = "Hello";

    static shared void hello(String name)
            => print(greeting + " " + name);

    shared static class Member() {
        shared void hi() => hello("everyone");
    }

    shared new() {
        hello("Gavin");
    }
}

shared void runme() {
    Hello.hello("world");
    Hello();
    Hello.Member member = Hello.Member();
    member.hi();
}

Any ideas?

[bjansen]

The compiler is not using the IntelliJ model loader, so the problem comes from somewhere else.

[gavinking]

Hrm. Weird.

[tombentley]

I can't explain that, it seems very odd.

[tombentley]

I have just fixed a model loader problem with static, so it might be worth trying again @gavinking

[gavinking]

Nope, still same error :-(

[gavinking]

OK, I got more info: a (different) error also occurs with the command line compiler, where I get:

source/deleteme/other/other.ceylon:19: error: Ceylon backend error: an enclosing instance that contains Hello.Member is required
    Hello.Member member = Hello.Member();

The generated Java is:

final .deleteme.other.Hello.Member member = new .deleteme.other.Hello.Member();

which looks totally correct to me. @tombentley would you take a second look, please?

[tombentley]

Still works fine for me on the command line @gavinking, the generated code for Hello.Member() looks like this:

    final .foo.Hello.Member member = new .foo.Hello.Member();

So there must be something different about your compiler than mine. But what? Any thoughts @FroMage ?

[tombentley]

Your generate Member declaration does look like this, right @gavinking

public static class Member implements .com.redhat.ceylon.compiler.java.runtime.model.ReifiedType, .java.io.Serializable {

[bjansen]

@gavinking works fine here, here's what I did:

• checked out the branch 6515 in ceylon/ceylon
• rebuilt ceylon/ceylon
• rebuilt ceylon/ceylon-ide-intellij
• reinstalled the plugin in my main IDE
• pasted and compiled your project

Hello world
Hello Gavin
Hello everyone

Process finished with exit code 0

[bjansen]

If it's not working in you debugged IDE, try doing a Rebuild in your main IDE, to force IntelliJ to copy again the dist in the plugin sandbox. Maybe your test IDE is still using old jars or something like that?

[gavinking]

i.e. the question is: why does javac think that Member is non-static when it is declared static in the generated code?

[gavinking]

Sorry, did not see yall's messages.

[gavinking]

OK, yeah, fine, after nuking my modules directory it worked. Sorry.

[gavinking]

Aaaah, interesting:

Error:(8, 30) ceylon: Ceylon backend error: static interface methods are not supported in -source 1.7
  (use -source 8 or higher to enable static interface methods)

[gavinking]

I have this basically working, though it still needs tests. In the current implementation:

• member class, function, value, and object declarations can be static, but
• static members must belong to a class, not to an interface, since we don't have static interface members in Java 7, and
• a nested interface may not be static, since that leads me down a rabbit hole of $impl classes, and anyway it seems pretty reasonable: a nested interface can't be default, formal, or actual either.

That seems like a reasonable first cut, and once I have some tests, I'm contemplating merging it.

[gavinking]

• static members must belong to a class, not to an interface, since we don't have static interface members in Java 7, and

This an irritating restriction, since static interface members are a requirement for #4005 (type classes). It leads me to the conclusion that either:

1. we can't do type classes until we abandon Java 7 support (which depends on what happens with Android), or
2. we have to move static interface methods to the $impl classes, which is a fair bit more work, and involves doubling-down on the $impls that we already want to get rid of.

Thoughts? @tombentley?

[gavinking]

• a nested interface may not be static, since that leads me down a rabbit hole of $impl classes, and anyway it seems pretty reasonable: a nested interface can't be default, formal, or actual either.

Scratch that restriction, looks like I got it to work. :)

[gavinking]

Alright, @chochos, @jvasileff, time to implement this for the JS and Dart backends. In the current implementation:

• static members must belong to toplevel classes with constructors (and not to interfaces)
• classes, interfaces, functions, values, and objects may be static, even class and interface aliases
• an alias may not be static

[gavinking]

• an alias may not be static

....aaaaand that restriction's gone too. I now allow static alias, just for completeness/regularity.

[jvasileff]

If you have a static member Class.foo(), what happens for the following expressions?

1. Subclass.foo()
2. (instance of Class).foo()
3. (instance of Subclass).foo()
4. foo() from within the body of Subclass

without having spent much time thinking about this, I'm hoping 2 & 3 are disallowed.

[gavinking]

I'm hoping 2 & 3 are disallowed

They're not (yet). The behavior is currently just like Java.

But not that, unlike Java, I can't redefine a shared foo() in Subclass. So the above expressions are all unambiguous.

[gavinking]

I now accept the following code:

abstract class Bool of t|f {
    shared static object t extends Bool() {}
    shared static object f extends Bool() {}
    shared new () {}
}

void fun(Bool b) {
    switch (b)
    case (Bool.t) {
        print(true);
    }
    case (Bool.f) {
        print(false);
    }
}

Since a static object is every bit as unique a a toplevel object.

[gavinking]

OK, @jvasileff has spotted a problem that I'll need help from the backend guys on: named argument invocations of static methods or classes result in backend errors. For example,

Integer.parse { string="124324123"; }

[lucaswerkmeister]

So now we have three options for “enums” (toplevel object, case constructor, static object member)?

[gavinking]

So now we have three options for “enums” (toplevel object, case constructor, static object member)?

Yes.

[gavinking]

One more bug: the generated methods for defaulted parameters don't get a static annotation generated. Need to fix that.