search

ceylon / ceylon-spec

DEPRECATED
Apache License 2.0
108 stars 34 forks source link

Backend-specific source files #500

Closed FroMage closed 9 years ago

FroMage commented 11 years ago

We need to figure out how to have certain files only compiled for certain backends. It could be as simple as saying .java files for the JVM backend and .js files for the JS backend, but we may also want different Ceylon source files for different backends, as glue or something.

FroMage commented 10 years ago

This would work, but I have a few remarks:

gavinking commented 10 years ago
  • Looks like the package names have to be the name of the platform?

According to the "indicative" implementation only. In truth I don't like that much and I think it might be better to have something based on annotations.

  • Doesn't look like we can get backend-specific module imports without #499 as well

We already have them. And they already work, except for one little bug in the js runtime (it ignores optional).

  • Can you clarify how we find specific implementations? Name-conventions for declarations seems new in Ceylon.

Agreed, it would be better to have something based on annotations. Like, perhaps:

implementation("node.js", `class JsImpl`)
implementation("jvm", `class JavaImpl`)
interface Api {}

Or whatever.

  • Can we have backend-specific values/methods too?

Backend-specific toplevel functions, sure. Values maybe, I would need to think about it.

  • It's not clear to me how we can make the backends optimise the metamodel part away, when at compile-time we already know the implementations so we should be able to do a lot better.

I don't think it matters much. We can always memoize. It would be nice, but it's not a hard requirement for 1.1.

FroMage commented 10 years ago

We already have them. And they already work

And they don't do what backend-specific imports do.

So let me suggest something else, which we can optimise properly:

interface Api{}

implementation("node.js", `class JsImpl`)
implementation("jvm", `class JavaImpl`)
native Api createApi();

Which would:

This would work for both methods and classes.

akberc commented 10 years ago

@gavinking This is a real breakthrough and the credit is to the team who understand the innards. The ramifications of this decision for scripting, binding browser libraries etc. are immense. I have some spare time these days and would be glad to help out wherever pointed.

quintesse commented 10 years ago

Not sure if you people think it's important but none of these proposals allow for easy after-the-fact extension of platforms, the list of implementations is always somehow hard-coded.

Couldn't we put the annotation on the implementing class?

implementation("node.js", `interface Api`)
class JsImpl() satisfies Api {}

implementation("jvm", `interface Api`)
class JavaImpl() satisfies Api {}

(I agree this starts to look like CDI and therefore might not perform acceptably according to Stef's criteria)

akberc commented 10 years ago

@quintesse The higher the better. Usually, there will be a different developer for each implementation and the inner workings of the package or module may be different for each platform. Also, it is easier for the typechecker/compiler to ignore packages/modules than delve into the semantics of each class.

Multiple implementations will be needed mostly when infrastructure services provided by the VM are needed: storage, database, HTML5 local storage, cookies, networking etc. -- and may be radically different to warrant class-level granularity.

gavinking commented 10 years ago

Not sure if you people think it's important but none of these proposals allow for easy after-the-fact extension of platforms, the list of implementations is always somehow hard-coded.

Well my justification for that is:

Now, OTOH, if we were trying to create a more general "service loader" facility, then yes, we would need extensibility. But I'm not sure if we really want to risk that this grows into a lame version of CDI...

quintesse commented 10 years ago

@akberc Well the recent work on Android suggests that a new platform is not as uncommon as one might expect and in the mean time the creators of Api might not want to have the burden of maintaining/including an unstable platform. At the same time the people working on the new platform might want to easily test additional implementation of Api without bothering anyone else (and without needing to use private forks of Api).

Also this would still need the package/module work to provide the platform specific compilation, I'm just suggesting a change in the way the lookup of the implementation is done.

@gavinking

the number of platforms is small and enumerated

It's the enumerated I wonder about. If "platform" is what we say it is and we define the list (right now "jvm" and "js"... or is it "jvm", "browser" and "node.js"?) then the it is as you say (but like I say above, the people working on a new platform will have to maintain forks of native packages until the time we decide to merge into mainline).

Or platform is a more malleable concept and we allow people to define their own. For example "android" might be a platform with a fallback to "jvm" if a specific implementation for Android wasn't found. Likewise "node.js" and "browser" might be platforms with fallbacks to "js".

This solution avoids scanning, which would be needed for extensibility. But I'm not sure if we really want to risk that this grows into a lame version of CDI

Indeed, not fond of either.

akberc commented 10 years ago

Or platform is a more malleable concept and we allow people to define their own

Malleable, yes, with the compiler/runtime/CMR implementation deciding what to ignore or what to fall back to. The constraint should not be an enumeration, but a compiler/runtime/CMR combination. The cap on proliferation would be the steep effort required to implement compiler rules or compiler/CMR implementations This opens up avenues like script compiler, HTML template compiler. browser 'require' CMR, etc. Maybe I'm getting too enthusiastic :)

thradec commented 10 years ago

I would prefer more general solution, like mentioned "service loader", which is something, what we will need anyway.

thradec commented 10 years ago

This solution avoids scanning, which would be needed for extensibility.

Maybe we can generated some index of provided services, during compile time, to avoid scanning.

gavinking commented 10 years ago

I would prefer more general solution, like mentioned "service loader", which is something, what we will need anyway.

OK, so I can certainly live with that. So I say we split this up into two separate issues:

These are really pretty separate things, though they of course need to work together. The service loader doesn't absolutely have to make it into 1.1.

FroMage commented 10 years ago

I haven't heard any feedback on my proposal to use generated native methods to provide the implementation-specific instances rather than the metamodel. I still think this is the best approach and is otherwise the same as the latest proposal, with many gains and no losses.

gavinking commented 10 years ago

@FroMage Well, it's fine for the limited case but what other people are saying is that they want something more generic that handles more platforms than just { jvm, js }. If that's the case then I don't see how generated methods would solve the more generic problem.

quintesse commented 10 years ago

I haven't heard any feedback on my proposal

I don't like the hard-codedness of it to be honest, which is why I agreed more with @gavinking 's option and even suggested the more CDI-like behaviour.

FroMage commented 10 years ago

But the list of supported platforms is known at compile-time no? The JVM backend supports a fixed set of platforms (currently JVM), and same for the JS compiler. If the list of platforms is known at compile-time and the implementations have to be provided at compile-time (for type checking), since they are in a platform-specific package of the current module, then I don't see how platforms could be added at runtime?

I think pluggability like CDI is something that is way outside the scope of backend-specific implementation for modules, even though we could make it more complex than it requires, but I don't see why we need all this complexity for this task.

quintesse commented 10 years ago

@FroMage To me the list of platforms is not the same as the list of backend we compile to. The latter is indeed pretty fixed and any changes to that list will mean releasing a new distribution.

But how often would someone write backend-specific code that is platform-independent? There is some in ceylon.interop for accessing Java internals like Class. And ceylon.collection I guess? It probably only uses Java classes that are also available on Android. But the part of the Java JRE that Android makes available in their compatibility layer is limited.

So personally I'd rather just have one official public system to handle platform specific code (where the 2 backends could be represented as "generic" jvm and js).

NB: It's just the hard-coding that I don't like. If it would be possible to have some special Xxxx createXxxx() method that could combine "service loading" with method handles to get the best speed possible that would be great. But I see that almost as a generic system we could use for method handles retrieved from a service loader.

akberc commented 10 years ago

Edited for clarity:

Here are the current use cases, most of which we theoretically support but practically do not support yet:

Static

Dynamic

FroMage commented 10 years ago

I'm not sure I understand how we got from backends to platforms. We already have SDK modules like ceylon.math which requires backend-specific code to be made cross-platform.

What do you call a platform then and why can't that work with backend-specific code like what we're trying to fix?

For example, JDK and Android may have different modules, but that would mean that we need certain module imports for JDK, and others for Android then. The problem is that the JDK and Android can't be imported at the same time, since they both provide the same set of packages, so we should create two .cars?

We don't yet understand the issues related to Android modules well enough to reason about Android properly.

Do we have examples of different code for different JS platforms that would require different Ceylon implementations? With dynamic and with the fact that ATM we can't import JS modules (non-Ceylon) in a type-safe way, pretty much any platform-specific code can be implemented by if statements in the same JS backend-specific package.

quintesse commented 10 years ago

As a developer, I would like my JVM-targeted Ceylon code to bind to JS-targeted Ceylon abstractions (and vice-versa) so that a simple compile results in HTML pages, module lookups and loading, and AJAX bindings to server-side code.

Whoah, let's not dive off the deep end! :) I do agree we want this, but I don't think this is part of the discussion on this issue.

gavinking commented 10 years ago

Well, for example, resource loading works differently for a JS program running on node compared to a browser-based client.

quintesse commented 10 years ago

can be implemented by if statements in the same JS backend-specific package.

Yes, which means hard-coding it, which is what at least some of us don't like. Because it means I either convincing others that my platform is worth supporting and it gets added to the official module or I need to maintain a fork. With service loading I could just say: I have an implementation for interface Xxx for platform Y.

akberc commented 10 years ago

and with the fact that ATM we can't import JS modules (non-Ceylon) in a type-safe way

but we can wrap non-Ceylon JS modules in a Ceylon abstraction, .e.g. Twitter bootstrap and have a auto-stitched Ceylon require loader. It is the creation of these abstractions that is a hurdle at the moment.

The problem is that the JDK and Android can't be imported at the same time, since they both provide the same set of packages, so we should create two .cars

Yes, there is a choice to NOT allow multi-executable multi-platform modules and create a Maven-like classifier/extension/type artifact specification within a module. That may be the easiest and consistent way if the alternates are too complicated.

FroMage commented 10 years ago

Well, for example, resource loading works differently for a JS program running on node compared to a browser-based client

OK but in both cases this is done via either the dynamic block which the JS compiler can compile in both cases, using an if statement, or via a ceylon.language API that will just work.

With service loading I could just say: I have an implementation for interface Xxx for platform Y.

So you're talking about multiple modules here, right? This is again more complex than backend-specific code which would indeed be solved with backend-specific-packages. This also means that implementations are either not type-safe or at least not checked during the compilation of the main module.

BTW, I'm sure we can reconcile the two such that for the code generated on behalf of the platform-specific native declaration will properly dispatch for all supported backends (known at compile-time by just being there in backend-specific packages), and typechecked so that the signatures and types match, all with a generated if statement that falls back to whatever pluggable API will use the metamodel to look for alternative unknown (at compile-time) implementations using discover (whatever) if the platform was not supported at compile-time.

This way we get type-safety and speed for backend/platforms we support at compile-time (we solve all the current use-cases for the SDK), and we remain extensible for third-party implementations that we don't know about (no use-case we currently have).

Something like:

interface Api1 {}
abstract class Api2(Integer i){}

platform native Api1 createApi1();
platform native Api2 api2(Integer i);

With implementations in backend/platform-specific packages, and would generate the following code for the JVM:

public Api1 createApi1(){
 if(os.platform == "JVM")
  return Api1JVMImpl();
 else
  return ceylon.language.lookitup(Api1.class);
}

public Api2 api2(long i){
 if(os.platform == "JVM")
  return Api2JVMImpl(i);
 else
  return ceylon.language.lookitup(Api2.class, i);
}

Note that this also works for methods and attributes.

gavinking commented 10 years ago

Not for 1.1.

FroMage commented 9 years ago

So in fact, the biggest limiting factor is the IDE and what it can do, and it looks like it will have to be enhanced to support anything interesting.

Here's a list of requirements:

Here's a list of nice-to-have:

The problem is mostly the IDE because most of the requirements have solutions that would be easier for the CLI. The only way the IDE can load code that is backend-specific ATM is under different modules that live in different projects. That's very restrictive, but is driven by the fact that if the IDE is set to compile for both backends, it will in fact only use the JVM typechecker, and ignore anything that comes from JS until it runs the compiler.

I think the easiest thing for us to do is to put backend-specific code in another module, then define which toplevel native declarations are backend-specific and let the compilers plug those in. The typechecker would then validate that the backend-specific toplevels are indeed provided for all backends. The modules would be cross-dependent (otherwise implementation cannot be validated).

Something like:

module ceylon.math {
 jvm requireImplementation 
 import ceylon.math-jvm;
 js requireImplementation 
 import ceylon.math-js;
}

shared interface Whole {}
shared abstract class Half {}

shared backendSpecific native Integer foo;
shared backendSpecific native Integer bar(Integer i);
shared backendSpecific native Half makeHalf(Integer i);
shared backendSpecific native Whole makeWhole(Integer i);

And then:

module ceylon.math-js {
 implementationProvider
 import ceylon.math;
}

import ceylon.math { Whole, Half, foo, bar }

shared implementation(`value foo`) Integer fooImpl = 2;
shared implementation(`function bar`) Integer barImpl(Integer i) { return 1; }
shared implementation(`function makeWhole`) Whole makeWholeImpl(Integer i) { return WholeImpl(1); shared implementation(`function makeHalf`) Half makeHalfImpl(Integer i) { return HalfImpl(1); }

The IDE will be able to validate that native implementations exist and that their schema is the same. The compiler will just plug them at compile-time differently for JS and Java. This is limited to toplevel values and functions which are easy to plug. It makes no sense for interfaces (though I expect they will be used to define what the methods will return) and is harder to plug for classes.

We can even extend it to suppose dynamic lookups later:

shared backendSpecific dynamicImplementation native Integer foo;

Which would be compiled for the JVM to:

public class foo_ {
 public long getFoo(){
  if(os.platform == "JVM")
   return ceylon.mathJvm.foo_.getFoo();
  else
   return ceylon.language.lookupDynamicProvider(foo_.class).get(); // use metamodel for `value foo` for lookup
 }
}

With a dynamic provider:

module stef.math {
 implementationProvider
 import ceylon.math;
}

import ceylon.math { Whole, Half, foo, bar }

shared dynamicImplementation(`value foo`) Integer fooImpl = 2;

Which would generate:

public class fooImpl_ {
 static{
     ceylon.language.registerDynamicProvider(foo_.class, fooImpl_.class); // use metamodel for `value foo` for lookup
 }
}

The only things the IDE needs to do are:

If it's hard to make the IDE do this, perhaps another solution is to make the IDE run two typecheckers for backend-specific modules, and make each ignore packages reserved to the other. That's a lot less efficient though. This also allows for backend-specific module imports but they would only be visible in backend-specific packages.

akberc commented 9 years ago

Thanks Stef for the detailed breakdown. I have three comments;

Here is a Ceylon-ish take on this: https://gist.github.com/akberc/01c8a492097a4564d79a

gavinking commented 9 years ago

@quintesse has been working on this.

quintesse commented 9 years ago

Closing this, what is implemented right now:

"You'd put your documentation here"
native
shared void test();

native("jvm")
shared void test() { print("A test for the JVM"; }

native("js")
shared void test() { print("A test for JS"; }

shared void run() { test(); }

for toplevel methods (as shown), attributes and classes.

The signatures of the declarations with the same name have to be identical. The declaration that is simply marked native is called the "abstraction" while the ones that have a native annotation with an argument are called "implementations". The only allowed arguments to the native annotation are: jvm and js. A declaration that is shared must have an abstraction, otherwise it's optional (but the implementations still need to have identical signatures). Method and attribute abstractions and members of class abstractions can leave out their implementation (if you specify them they will be ignored by the compilers).

There's no support for imports right now, this is ongoing for #499.

quintesse commented 9 years ago

Btw, this doesn't implement exactly what was proposed in the original text of this issue. But for now this new implementation supersedes that idea.

@gavinking when you fix the error messages please give me a heads-up so I can fix the tests.

gavinking commented 9 years ago

Fantastic, congratulations!

Sent from my iPhone

On 28 Apr 2015, at 7:26 pm, Tako Schotanus notifications@github.com wrote:

Closed #500.

— Reply to this email directly or view it on GitHub.

FroMage commented 9 years ago

Great! So what does this mean for Java or JS declarations then? Can we still have them?

quintesse commented 9 years ago

What are you referring to exactly @FroMage ?

FroMage commented 9 years ago

Suppose I need to write a Java or JS class/function? In a .java or .js file?

gavinking commented 9 years ago

@FroMage In principle, yes, because that's the case of just a native header and no implementation. In practice I'm not sure if the JVM compiler can handle this anywhere outside of the language module. (The JS compiler does support it, I believe.)

quintesse commented 9 years ago

Yes, you can do it like this for example (directly from my test module):

test.ceylon:

shared void run() {
    testNative();
}

native("jvm") void testNative() {
    myprintJvm(NativeCode().test());
}

native("js") void testNative() {
    dynamic {
        myprintJs(nativeCode());
    }
}

and then have:

NativeCode.java:

public class NativeCode {
    public String test() {
        return "This is a native Java class";
    }
}

and:

nativecode.js:

function nativeCode() {
    return "This is a native JavaScript function";
}

(you'll need to have the latest code though, I just fixed a bug related to this)

FroMage commented 9 years ago

Oh, so you can. How does the IDE behave in this case? I expect it not to see nativeCode from js since the JS model loader is not plugged in the IDE, but will it see the NativeCode class in Ceylon code? Will it not allow you to call it even outside native methods?

quintesse commented 9 years ago

The IDE hasn't changed behaviour for what was already there. So you can just call NativeCode outside of native methods, you'll just not be able to enable the JS backend because you'll get errors. The same with using dynamic and trying to enable the JVM backend. But you can enable both backends and then put those calls/dynamic in native methods and everything should work. I say "should" because right now the IDE does have a problem still with JVM code. I'm looking into that.

quintesse commented 9 years ago

Ok, the IDE problem is now fixed.