Friend packages or modules

gavinking commented 11 years ago

Occasionally we encounter a situation where two packages share the same development lifecycle, but are packaged into separate archives for purely practical reasons relating to how the software is distributed. For example, an API implementation org.hibernate.impl, and its tests org.hibernate.test are developed together, but are compiled into separate archives.

In this kind of situation, it makes sense to be able to suppress the usual visibility control checks, for example, allowing test code in org.hibernate.test to access unshared declarations of the unshared package org.hibernate.impl, even though the two packages reside in different modules.

It seems to me that a nice way to address this would be via the concept of friends.

I'm not 100% sure of the right granularity here: package or module? If module level is the best option, this is what the module descriptor might look like:

module org.hibernate.impl {
    allow org.hibernate.test;
}

This approach would work well if all we need to do is grant access to shared declarations of unshared packages.

But I think package granularity makes more sense, like this:

package org.hibernate.impl {
    allow org.hibernate.test;
}

Since I think we probably need to grant access into contents of the package, not just the contents of the module. (Well, I'm not totally sure about this.)

Thoughts?

FroMage commented 11 years ago

Feels to me that it should be at the module level, especially for tests where presumably every one of your packages should be shared, which would require a lot of work if it were on the package level.

gavinking commented 11 years ago

@FroMage That's what's intuitive to me too. But if you need to write tests for unshared declarations? It seems very strange that a module-level declaration would affect the visibility of things inside packages...

jpragey commented 11 years ago

A few remarks:

Is there any other real-life use case for friends than testing? I'm wondering...
In the context of tests, I also feel it should be at the module level (same reason as FroMage). And module-level declaration affecting package visibility isn't that strange if you consider that modules and packages are conceptually different kind of beasts.
I'm a bit dubious about delaying this issue to 1.1. Ceylon 1.0 will need some solid test framework, both for its first applications and its own libraries/frameworks.

RossTate commented 11 years ago

For testing, friend modules makes sense to me. Besides testing, I have had occasion where two packages made sense to be separate but still needed access to each other that no one else should have. Unfortunately I have forgotten the specifics of those occasions, though.

On Thu, Jun 13, 2013 at 11:05 AM, jpragey notifications@github.com wrote:

A few remarks:

Is there any other real-life use case for friends than testing? I'm wondering...

In the context of tests, I also feel it should be at the module level (same reason as FroMage). And module-level declaration affecting package visibility isn't that strange if you consider that modules and packages are conceptually different kind of beasts.

I'm a bit dubious about delaying this issue to 1.1. Ceylon 1.0 will need some solid test framework, both for its first applications and its own libraries/frameworks.

— Reply to this email directly or view it on GitHubhttps://github.com/ceylon/ceylon-spec/issues/668#issuecomment-19397815 .

luolong commented 11 years ago

In Eclipse OSGi they have two kinds of bundles -- the regular ones that each has their own classloader and visibility isolation.

And the other kind are fragments that are basically loaded into the host plugin classloader context thereby becoming an actual part of the host plugin. All the classes and packages in the host plugin were visible to the fragment and vice versa.

The most prominent use of such fragments in Eclipse is SWT. You have the base org.eclipse.swt plugin that contains little more than just declaration of the bundle and the platform specific binaries and java implementations are all put into the platform specific fragments, loaded in the classloader context of the org.eclipse.swt bundle.

They also use fragments for testing purposes - using the same reasoning outlined in this issue - you do not want to "pollute" the dependencies of the module with testing specific dependencies.

gavinking commented 11 years ago

@luolong interesting. Without commenting on the precise semantics, the concept of a "fragment" is certainly appealing. And it would certainly explain how the tests have access to the code they test, if they actually belong to the same package.

I think that's definitely a concept/direction worth exploring.

simonthum commented 11 years ago

I can report we use OSGi fragments for testing too. Other than that, OSGi calls friend modules "buddies", and there are lots of reasons for them. But all I'm aware of boil down to "we need to be compatible with code that has fundamental problems considering real modules".

I definitely recommend favouring the fragment style over friends/buddies, if it is at all possible, to avoid the spillovers.

gavinking commented 11 years ago

@simonthum OK, thanks for the feedback. I like the idea of fragments, I'm going to try to come up with some kind of proposal...

luolong commented 11 years ago

Yes, the whole "buddies" system in OSGi has been a constant minefield of ClassLoader deadlock sources. I dislike it to the point of considering it outright dangerous.

simonthum commented 11 years ago

@luolong Yes, dangerous but sometimes a last resort... I found no other way to deal with serialization libraries or compilers (short of forking them). Any of course you always need to modify the target module to trick the child safety mechanism.

It's not a big problem with OSGi where there are (for us at least) zero workable ways except "roll your own manifest". For a real module system the buddy/friends stuff needs to be independent from modules' deployment units and be safer. IMO.

akberc commented 11 years ago

To keep Ceylon's benefit of compile- and run-time modularity, I would suggest package as the right level as there are differing concerns and modules are different beasts, as mentioned abov.:

restricting package 'sharedness' so that the existing annotation shared is actually shared(*) and shared(packageA, packageB') explicitly allows access to only these packages. This would be very handy for tool or framework APIs where there are inter-package dependencies but the 'internal' packages should not be visible. A package 'a.b.' would be implicitly shared with a 'a.b.test' annotated with @Test in the same module.
an annotation of @Test on the package within the same module can allow a build tool to exclude it
a package would allow access to a package that ends with 'test'. and exists within the same module annotated with @Test

Why?

just as Java tests cannot access private members inside a Java class, Ceylon tests should abide by the language's visibility enforcement and not make exceptions
the Java convention of separate source folder for tests, with the same package name, took advantage of default and protected. Similarly, Ceylon testing conventions will probably evolve new conventions, including some form of inheritance.

loicrouchon commented 11 years ago

As jpragey, I think we should solve this issue for 1.0, not for 1.1

In the current ceylon task engine I'm working on, only one function should be available for public usage. But in order for me to unit-test the module I had to put shared annotation on a lot of functions. The really bad part about this is that those functions appear in documentation generated by ceylon doc.

In my opinion, it would be weird for a 1.0 language not to cover a such use-case.

Concerning fragments, it seems to be quite interesting. If I understood well the concept, fragments are loaded using the same classloader than their parent bundle (module). This would I think solve the visibility problem from the fragment.

A question I’m asking myself is, if we go in that direction, how are we going to declare a fragment.

For security reasons (only allow known fragments), it should be done in the parent module

module mymodule '1.0.0' {
    allow test.mymodule;
}

This option gives the control to the main module, which allow you to decide which modules will be loaded in the same classloader and will have access to your non-shared (top level? more?) elements. But on the other side, I don’t find that it fits the use case of org.eclipse.swt platform specific implementation shown as example by luolong because you will have to declare all supported platforms in the parent module.

An other option could be to declare it at fragment level

module test.mymodule '1.0.0' of mymodule '1.0.0' { }

We could also use the extends keyword (or another one) instead of of

In this case, the parent module owner doesn’t have any control on who is going to use it as parent, but it will, I think, best fit org.eclipse.swt platform specific use case.

gavinking commented 11 years ago

As jpragey, I think we should solve this issue for 1.0, not for 1.1

I would love to be able to do this, but we need to release 1.0 now. This issue is too open-ended to make it in :(

luolong commented 11 years ago

For security reasons (only allow known fragments), it should be done in the parent module

Why would you do that? If we do this, it is no different than declaring friend modules. The whole idea of fragments is that they are applied at the deployment time and the original module should not have to specifically declare the dependency/awareness of the extending module.

Syntactically, I suggest, the most ceylonic declaration would look something like this:

module test.mymodule '1.0' extends mymodule '1.0' { }

Although, I can see how in some cases, the module may want to define a constraint whereby it needs one of the specific extensions/fragments in order to work properly. For example, when a module needs a backend specific implementation or a native/binary module to work with. In this case the following syntax would make sense:

module mymodule '1.0' of mymodule.jvm '1.0' | mymodule.js '1.0 { }

gavinking commented 11 years ago

@luolong you're proposing a syntax that looks a lot like my proposal for abstract packages and package inheritance. (I know I've never written this up anywhere, but that's mostly because I don't want anyone else stealing the idea before I get a chance to implement it!)

But whatever this feature is, I don't think it is that. I don't think it has much to do with inheritance at all, at least not as far as I can tell from the discussion so far.

akberc commented 11 years ago

Think security. Liferay has barely managed to get its marketplace going, and has had to fall back on Java security policies. I am afraid of any means to bypass language visibility going from just a testing tool to a production 'hack'. Modularity at compile time and continuing through build to runtime is a powerful concept and Java 8+ is struggling with it as well - so it is not simple!

Think distribution. What is the targeted granularity of a module? Assume a future Ceylon Android app that is a single package with 10 code units and 20 classes - will it need a code module, a test module and a build module? Then, will these single-package modules be jar-red (not using 'package' to avoid confusion) into a super-module, with its own manifest? So, will modules end up being wrappers for packages?

Testing was a bolt-on to Java. With Ceylon, testing can be baked in, with compile tools and everything else specifically excluding classes and packages, and even methods, annotated 'Test' unless specifically building for Test.

jpragey commented 11 years ago

I understand friends/fragments/whatever are complicated issues, but if releasing 1.0 means it's OK for real-life frameworks, libs and applications, automated testing MUST be possible. As a quick workaround, I would propose that a module sources could be split in 2 directories (say, 'source' and 'testSource'), each with a package.ceylon and a module.ceylon (with test specific dependencies in testSources); then we can have tools ignore them selectively:

eclipse can ignore selectively source files yet; so by ignoring source/.../module.ceylon, it would use the testSource one, which is OK for development. It can't really be used now because of https://github.com/ceylon/ceylon-ide-eclipse/issues/563, which is scheduled for M6.
ant could compile both for test and release if the ceylon-compile ant task could ignore files (currently it can't, or I didn't find how).

So just by solving issue 563 and adding an 'ignore file' option to ant ceylon-compile, we could probably have a workable TDD environment.

gavinking commented 11 years ago

Let's see if there's something simple we can do for 1.0. But it's gotta be something that is no more than a few days work!

akberc commented 11 years ago

I see the assembly keyword in some new commits. If it applies to this issue as well, t is a good simple approach for now and if it allows experimentation (user attributes/annotations) in assembly.ceylon, it should give us real-life feedback for 1.1. :+1:

gavinking commented 11 years ago

Assemblies aren't for Ceylon 1.0. I'm just pre-reserving the keyword for when we do need it.

On Fri, Aug 30, 2013 at 11:10 PM, Akber Choudhry notifications@github.comwrote:

I see the assembly keyword in some new commits. If it applies to this issue as well, t is a good simple approach for now and if it allows experimentation (user attributes/annotations) in assembly.ceylon, it should give us real-life feedback for 1.1. [image: :+1:]

— Reply to this email directly or view it on GitHubhttps://github.com/ceylon/ceylon-spec/issues/668#issuecomment-23589481 .

Gavin King gavin@ceylon-lang.org http://profiles.google.com/gavin.king http://ceylon-lang.org http://hibernate.org http://seamframework.org

jpragey commented 11 years ago

I just realized the ant issue has a simple solution: for tests, copy and merge source and testSource in a temporary directory (ignoring source/**/module.ceylon), and compile from there. From my build.xml:

<target name="compile" depends="ceylon-ant-taskdefs" description="">
    <ceylon-compile encoding="${source.encoding}">
        <module name="org.myapp"/>
    </ceylon-compile>
</target>

<target name="test" depends="ceylon-ant-taskdefs" description="">
    <copy todir="build/testsrc">
        <fileset dir="source">
            <patternset>
                <include name="**/*"/>
                <exclude name="**/module.ceylon"/>
              </patternset>
        </fileset>
    </copy>
    <copy todir="build/testsrc">
        <fileset dir="testSource">
            <include name="**/*"/>
        </fileset>
    </copy>

    <ceylon-compile encoding="${source.encoding}" src="build/testsrc" out="build/testmodules">
        <module name="org.myapp"/>
    </ceylon-compile>

    <ceylon-run run="org.myapp.runTests" module="org.myapp/1.0.0">
        <rep url="build/testmodules"/>
    </ceylon-run>
</target>

"ant clean compile" creates the usual module (without tests) in modules/, and "ant clean test" creates a module (with tests) in build/testmodules/, and run org.myapp.runTests from there.

So the last obstacle to TDD is eclipse 563 issue.

gavinking commented 11 years ago

FTR, ceylon/ceylon-ide-eclipse#563

gavinking commented 11 years ago

Oh, you had already linked it ;)

simonthum commented 11 years ago

FWIW, I thnik this is going into the right direction for a simple and doable solution. IMO the whole point of fragments or what not is to create a useable separation of development vs. deployment issues. Attacking this from the build side is unlikely to create troube later on with a future holistic assembly-based approach.

Condensed: I never really liked to create extra projects for tests. Sounds Great ! ;)

gavinking commented 11 years ago

After playing around a little but with the multi-source-dirs solution in the IDE and Ant, I'm convinced that this is a viable solution for Ceylon 1.0. We can come back to the good ideas of stuff like friends or fragments in a future version of the language.

jpragey commented 11 years ago

The trick of having an empty module.ceylon in source/ seems to work since yesterday (there were backend errors when test code called application code before). Ant can easily build application and test configurations by filtering out test-only imports from module.ceylon on the fly. So we have a workable solution yet...

gavinking commented 11 years ago

The trick of having an empty module.ceylon in source/ seems to work since yesterday (there were backend errors when test code called application code before).

Ugh. So what changed yesterday?

FroMage commented 9 years ago

Jigsaw is going in the direction of specifying which other modules may access unshared types/packages from the current module, much like our friend module feature here. I really think this is the right way to do it.

ceylon / ceylon-spec

Friend packages or modules #668