Backend-specific source files

[FroMage]

We need to figure out how to have certain files only compiled for certain backends. It could be as simple as saying .java files for the JVM backend and .js files for the JS backend, but we may also want different Ceylon source files for different backends, as glue or something.

[tombentley]

One way to do it would be to have the source source directory hold pure ceylon, and have separate source-java (which could contain .ceylon and .java source files) and source-js (which could contain .ceylon and .js source files) source directories which are only included when compiling for those backends.

[FroMage]

I side with @tombentley here, except I suppose a more correct name would be ceylon-jvm. It can be implemented trivially. And people need it now.

[quintesse]

But do we want to add --src-jvm and --src-js options to all the commands (and the respective API functions everywhere we have things like getSrcDir())? Being able to separately specify those folders doesn't seem like a really necessary option. Maybe it would be a better to use sub folders and be simply able to deduce them, _jvm and _js for example.

[FroMage]

@quintesse if we do backend-specific in the same source folder using subfolders like you propose then stuff will be in different packages, like com.foo and com.foo._jvm and you won't be able to import them without breaking one backend. Unless imports are also backend-specific.

I really think putting them in separate source folder is cleaner.

I don't think we need --src-jvm because if you define --src it overrides both sources and sources-jvm defaults.

[tombentley]

One aspect of using different directories would be that eclipse would show multiple source folders. That's annoying a lot of the time (I certainly find it annoying that half of ceylon.language is in runtime, and the rest of it is in src), and means you have to form the union of all the compilation units in your head.

[FroMage]

How many modules do you expect with backend-specific stuff, aside from the SDK and a few others?

[luolong]

How many modules do you expect with backend-specific stuff, aside from the SDK and a few others?

For JVM backend -- quite a few, if we're lucky :)

[tombentley]

The truth is that none of us can know that. What we do know is that the easier it is to write modules for multiple backends the more portable modules will be written.

[FroMage]

It's already trivial to write JVM-specific modules because the JVM compiler includes java files by default. It's only marginally harder to write multi-backend modules.

[quintesse]

then stuff will be in different packages, like com.foo and com.foo._jvm

Well the idea was to ignore that folder and treat the code in it as if it were in a directory one level higher. But I agree it might not be the best option either.

But if right now we are able to write:

- sources
  - foo
    - bar.java
    - baz.ceylon

and make it work, then it would be a shame that to make it work for multiple backends we have to do:

- jvm-sources
  - foo
    - baz.ceylon
- js-sources
  - foo
    - baz.ceylon
- sources
  - foo
    - bar.java
    - bar.js

because suddenly we have different Ceylon implementations and are forced to maintain them in different folders away from all the other files.

[quintesse]

Now hopefully with native we can prevent having to write different Ceylon implementations in most cases and we can still go for the first solution and need this only for very specific cases where native won't work.

Another thing I can think of (that doesn't seem very nice either) is:

- sources
  - foo
    - bar.java
    - bar.js
    - baz.jvm.ceylon
    - baz.js.ceylon

But at least all the files would be together.

[luolong]

Nice, I would like that.

To me the most comon story for native annotation is to implement the tiny bit of the glue code needed for interop with the native platform.

And this would serve my purpose perfectly!

[FroMage]

Tako: it would more look like:

- jvm-sources
  - foo
    - bar.java
    - baz.ceylon
- js-sources
  - foo
    - bar.js
    - baz.ceylon
- sources
  - foo
    - barUser.ceylon

And then again you only need the two baz.ceylon if you need a Ceylon glue. If you write JS and Java code that follows the Ceylon compilation mapping (like we do for all native stuff in the language module) you don't need that glue.

[FroMage]

native is orthogonal to this. Its only purpose is to give you a pretend Ceylon schema, which ATM is ignored.

[quintesse]

Stef, the only thing you changed is the location of the native files, which shouldn't matter. You still need to move the files away to a different folder than the code that's actually using it (in your case barUser.ceylon) You're basically using a project structure with "overlays". And I'm wondering how we're going to handle that in the IDE for example (you'll suddenly have 2 of everything defined in those files).

[FroMage]

Well, I find it quite nice that I don't see JS files when I'm working on the Ceylon parts.

[luolong]

well, I think we are mashing together here 2 separate use cases for "native" code n ceylon project.

I am mostly interested in the case where I need to provide native implementation of a platform specific feature, while you are probably just talking about a module that wants to compile to both, JVM and JS backend, so that you could write both server side and client side code in Ceylon.

In my case, the Foo.js source file is actually a partial implementation of the Foo.ceylon class, in which case I really want it to be as close to the .ceylon file as possible. Specially, if I manage to keep the platform specific implementation as simple and lightweight as possible.

[FroMage]

@gavinking we're still waiting for your take on this.

[loicrouchon]

I don't really see why there would be specific ceylon sources for a particular backend.

If this is to avoid to include too many sources in a compilation to java or to js, I would rather use several modules:

• 1 common module
• 1 module per backend For example for a client-server application we would have common (compiled to both java / js), ceylon server (compiled to java only) and ceylon client (compiled to js only)

But maybe I'm missing something

Do you have use cases / examples of were this would be needed?

[FroMage]

Yes, when you write a Ceylon implementation class that interacts with native stuff. Interacting with a Java API in one case, and with dynamic JS stuff in the other case.

[quintesse]

But like @loicrouchon syas, in that case you could use 2 different modules (something we want to implement anyway). So we'd be talking about the case where we want to have 2 different Ceylon implementations, but they are so trivial that we don't want to turn them into separate modules.

So is it worth it to add these backend-specific source folders? (Remember that we'd still be able to have a single Ceylon file interfacing with 2 different native implementations using native)

[FroMage]

Well, it allows you to write trivial implementations that talk to different backend-specific stuff while writing only Ceylon and not a line of Java or JS. That appeals to me a lot.

[quintesse]

You mean without using any native implementation at all, just different behaviour depending on the backend? Hmmm okay, but that seems like something that you could trivially handle within a single source folder with a simple if-statement, which is something that at least won't confuse Eclipse.

[FroMage]

No you can't because you'd have to import backend-specific stuff and when compiling the code each backend compiler would not be able to typecheck the other backend-specific part.

[quintesse]

Import backend specific stuff? Meaning different module.ceylon files? Doesn't that just shout "backend specific modules" to you? Just imagine how that will fuck with the IDE, having to handle alternative module files and imports and such, I'm not even sure how to visualize that. We don't have any of those problems with backend-specific modules. Or am I seeing things to bleak?

[gavinking]

If you have a module that depends on backend-specific modules, it definitely needs backend-specific code to interact with those backend-specific modules.

But have you guys considered just using if for that?

if (process.isJava) {
    //use java-specific module
}
else if (process.isJavaScript) {
    //use javascript-specific module
}

Of course we could do something more typesafe than that, if you think it matters.

But this splitting stuff across multiple source dirs thing feels overengineered to me...

[gavinking]

No you can't because you'd have to import backend-specific stuff and when compiling the code each backend compiler would not be able to typecheck the other backend-specific part.

It's very unclear to me really how much of a problem this is.

[FroMage]

Backend-specific modules is #499. And when we're writing stuff that would take one JDK call or one JS dynamic line, we really want to avoid writing specific modules just for them.

[quintesse]

But this splitting stuff across multiple source dirs thing feels overengineered to me

So do I, especially for the little ROI it seems to give. But I won't mind too much if Stef can ease my fears on the problems I think it will cause for Eclipse.

[FroMage]

@gavinking we can't use backend-specific modules in the same source file as it won't pass compilation since each backend will not be able to see the other backend's modules.

[FroMage]

I don't see a single thing you mentioned which would cause Eclipse to have problems.

[FroMage]

module ceylon.time '1' {
 jvm import java.base '7';
}

in source-jvm/ceylon/time/timeZone.ceylon:

import java.util { TimeZone { getDefault }}

shared Integer timeZoneOffset = getDefault().rawOffset;

in source-js/ceylon/time/timeZone.ceylon:

shared Integer timeZoneOffset {
 dynamic {
  return window.timeZoneOffset;
 }
}

[gavinking]

@gavinking we can't use backend-specific modules in the same source file as it won't pass compilation since each backend will not be able to see the other backend's modules.

I understand your objection. I'm just not convinced it's something you can't work around. I mean, once clear way to solve the problem would be a special language construct:

if (exists module foo.bar.java) {
    //...
}
else if (exists module foo.bar.js) {
    //...
}

But to me it feels like even that should not be necessary...

[quintesse]

You don't see having multiple modules files as being a problem? Modules consisting of files from several overlapping directories, multiple modules with the same name but different dependencies (possibly even versions), what to do when one exists in the "main" source folder as well? And it's the way Eclipse decides what the classpath will look like, showing the Ceylon Module entry in the explorer. Are you sure it'll be easy?

[quintesse]

In your example you need 2 different module files...

[FroMage]

You don't see having multiple modules files as being a problem?

See #499 again and my example, only a single module file.

Modules consisting of files from several overlapping directories

This is already allowed

Multiple modules with the same name but different dependencies (possibly even versions),

Not allowed. Well different backends may have different dependencies, but not different versions.

What to do when one exists in the "main" source folder as well?

Easy: duplicate declaration, like we already do with multiple source folders.

And it's the way Eclipse decides what the classpath will look like, showing the Ceylon Module entry in the explorer

Again, this is already allowed.

[FroMage]

@gavinking: your suggestion seems to imply that there are blocks of code in a single source file which are ignored by the typechecker and backend, depending on the backend compiling it. Also we need backend-specific import statements in the file then. It is a possibility but I'd rather keep backend-specific stuff for the entire file myself.

[gavinking]

@gavinking: your suggestion seems to imply that there are blocks of code in a single source file which are ignored by the typechecker and backend, depending on the backend compiling it. Also we need backend-specific import statements in the file then. It is a possibility but I'd rather keep backend-specific stuff for the entire file myself.

Well, it's not clear to me why that would be necessarily better?

[FroMage]

It's certainly subjective, but I think it makes the code more spaghetti-like.

[luolong]

I can see the point for organizing your source folders into backend specific sources, if that is what you want to do.

I just do not see, how should we treat jvm-source diferently from js-source (or source for that matter).

All source folders should be treated equal. It is the backend specific compiler that needs to sort out, what to do with files in those folders.

The way I see this, I might want to organize my folders like this:

- project/
  - source/
    - my/module/
      - module.ceylon
      - package.ceylon
      - Implementation.ceylon
      - Implementation.js
      - Implementation.java
  - test-source/
    - my/module/
      - test.ceylon

where Implementation.ceylon has some native annotations on methods or top level functions and *.js/*.java sources provide the native implementations for those.

Test folder would then contain test code in ceylon and compiled&run only when testing.

But if I want to, I can go the route proposed by @FroMage and separate out the jvm- and js specific source folders with backend specific implementations in Ceylon.

We could even go a bit further and allow module.ceylon files in the separate folders to augment each other, meaning that with the example above:

With project/source/my/module/module.ceylon defined as:

module my.module '1' {
    import ceylon.time '0.6';
}

and project/test-source/my/module/module.ceylon as:

module my.module '1' {
    import ceylon.test '0.6';
}

would merge into this module declaration after compilation:

module my.module '1' {
    import ceylon.time '0.6';
    import ceylon.test '0.6';
}

[quintesse]

and allow module.ceylon files in the separate folders to augment each other

There's also Stef's suggestion mentioned in #499 where you can specify to which backend a dependency belongs.

I'm not sure which I prefer. On the one hand I like Stef's suggestion because it means having a single module file that prevents contradictions and ambiguities, on the other hand I like the multiple module file way because it means not having to create backend specific annotations (something that might make installing backends on-the-fly more difficult, but that's something for the far future)

[luolong]

I was actually little concerned about the possibility of ambiguities in multiple module files in my proposal as well.

I think the best way to fight this would be to allow slightly different module declaration formats for the "main" module and the augmenting module declarations.

Let me try to illustrate this idea:

Given main module declaration in project/source/foo/module.ceylon:

    module foo '1' {
        import bar '2';
    }

and the augmenting module declaration in project/test-source/module.ceylon:

    augment module foo {
        import baz '3';
    }

This way, the relationship is cleanly one way. Also, the augmentation is compile time only operation. The module declaration that gets into the binary will already be merged.

[ghost]

Sidenote: It might make sense, to have a single ceylon module, but different .java source files per JVM Version. Especially for some core modules, that try to do things as performant as possible for each target. Now this is less important with the JVM because of it's downward compatibilty, but it might be interesting for C# if you ever plan to do that. Hence I would recommend different source foulders like js-source as fallback or default and js-1.8-source for specific stuff.

Need not be implemented asap, but design should allow to add that later.

[FroMage]

It may also make sense to allow backend-specific declarations (not files) that would only be typechecked and compiled by the proper backend. This way the ceylon.json module could have a JS-specific method that translates a Ceylon JSON value into a JS object. Ceylondoc would be able to document that the method in question only exists for the JS backend.

[gavinking]

So I don't know what to do about this one. Thoughts:

• To me it sounds like the eventual solution should be based on my proposal for abstract packages and package inheritance, but that's definitely not a feature we'll have in Ceylon 1.0.
• Any solution which depends upon multiple source dirs (as suggested by Stef) or filename naming conventions or whatever is going to require some significant re-engineering of the IDE, which simply can't handle the notion of the same thing being declared in multiple places.
• My first suggestion to use if is going to cause problems for the commandline compiler which AFAIK can't compile against the source of binary modules.

Is this something we can live without in Ceylon 1.0? Ditto #499? I don't see a simple way solution to this one. Any solution sounds like it would significantly delay the release.

I suppose as a temporary solution we could build support for something like Stef's proposal into just the commandline compiler and not support it in the IDE for now. But in that case I don't see why it's something you can't just deal with in the build tool by copying files around....

[gavinking]

We will solve this and #499 for Ceylon 1.1. Not for Ceylon 1.0. Sorry.

[gavinking]

So I have made a decision on this stuff. Here's how it's going to work:

• A package may be annotated platform("jvm"), platform("node.js"), etc.
• Such a package is ignored by compilers for other backends.
• Declarations in a package annotated platform may not be used by code in packages which are for a different platform, or which are cross-platform.
• There is a function in ceylon.language.meta that returns a platform-specific implementation of a given abstract type, for example: Class<X> xImplClass = platformImplementation(X);
• platformImplementation() uses conventions or annotations to locate the implementation of X in a platform-specific package.

An indicative implementation of platformImplementation() is the following:

shared Class<X> platformImplementation<X>(X x) {
    String packageName = 
            x.declaration.qualifiedName + "." + runtime.name;
    assert (
        exists implPackage=x.declaration.containingModule
                .findPackage(packageName),
        exists impl=implPackage.getClassOrInterface(x.declaration.name),
        is Class<X> implClass=impl.apply<X>(x)
    );
    return implClass;
}

Of course there are some details to be worked out there.

The advantages of this approach are compelling:

1. No changes needed to the IDE.
2. A tiny bit of dependency validation needed in the typechecker at the package level (easy to implement).
3. Backends need to ignore packages (trivial to implement).
4. Does not require multiple modules.

Any other proposal I have seen would have profound impacts upon the typechecker and IDE, and would therefore be impossible to implement for 1.1. This is the only realistically viable approach.

[akberc]

Agreed, will work. platform may be a bit confusing. May I propose enhancing native which we've promised to make available in future language releases? --
http://ceylon-lang.org/documentation/1.0/reference/annotation/native/

[gavinking]

@akberc Well, native is a little bit different since it says that something is implemented in a different language, or partially in a different language. In this case, we're talking about stuff that is actually implemented completely in Ceylon.

[gavinking]

FTR, I'm trying to take credit for this proposal, but indeed it is Akber's musings that inspired it.