Downloads mvn:http artefacts on each build/save

[akberc]

Unlike aether, with mvn:http repo, artefacts are stored in `~/.ceylon/cache', which is great, and one place to look - but -

• downloaded on every build - ouch - hammering maven1
• and even if present in cache with the right version -
  • with dodgy network connection, Eclipse hangs
  • with no Internet connection, modules.ceylon error

So,

• probably need to check cache before going to the network
• and need a network timeout

Possibly related to ceylon/ceylon-runtime#45

[quintesse]

Although this simplistic maven support was nice to have before we had "aether" I'm wondering if we can't just remove this functionality right now. It's just more stuff to maintain and more stuff that can have bugs, and for what exactly? Opinions? @gavinking @FroMage @tombentley ?

[akberc]

I counted 4-12 downloads on each build/run - once for the typechecker, once for initialization etc.!

Yes, also see: https://github.com/ceylon/ceylon-runtime/issues/45

Aether leaves some temporary files around and has its own little niggles, but we could focus on one and make it really work.

[gavinking]

It's just more stuff to maintain and more stuff that can have bugs, and for what exactly?

The less time we spend futzing with Maven the better.

Honestly, given the problems we've seen with maven repos having broken metadata I would say just allocate a single classloader to maven and stick all maven-sourced modules on that one classloader. It's likely to be less-broken than anything else we could possibly do.

[akberc]

Yes, that would work. Then we can say 'Maven repository support is enabled, but dependencies have to be individually defined in module.ceylon in this format . . .'.

[akberc]

Without some kind of consistent Maven support within modules, it is pretty hard to interop with Java outside the IDE.

Since aether puts its stuff in .m2, I would go for cached mvn:http so that the dev knows where to find things, or clear things in one place.

[quintesse]

So-called "cached mvn:http" really isn't an option, it's not Maven at all really, you just give a bunch of almost direct URLs to artifacts to download.

And really, Aether works most of the time, we just have this irritating problem that not all Maven artifacts define their dependencies correctly. So to solve that we need a combination of a) a flat classpath like Gavin says and b) a way to override dependencies defined in POMs.

But hopefully most people will never have to deal with that and just adding "--rep aether" will do the trick.

This whole mvn: thing has always been a stop-gap measure, I'm strongly suggesting we remove it before more people try to use it. There's is only one way forward and that's with Aether.

And that Aether can't use our cache is maybe unfortunate for cleaning the cache, but trying to override that will end up with more problems than it solves.

[FroMage]

I also recommend we remove it, but we should improve ease of specifying a non-maven-central maven repo with aether. Without having to define a settings.xml file ;)

[gavinking]

@FroMage as in, something in .ceylon/config?

[akberc]

Great stuff! Roughly, what would aether need to be conffgured to make more use of `~/.ceylon' better?

The end goal is as @quintesse said a few days ago: for a dev to be able to check out a git repo, or unzip a tar ball, together with `.ceylon/config', and voila -- it just works!

[gavinking]

Who is going to review this? @quintesse?

[akberc]

After looking at Shrinkwrap and no way of dynamic configuration, there are two options for aether -

• Since aether is going to create an .m2 in any case, and if we haven't found a settings.xml, we can create a default ~/.m2/settings.xml. The down side is that we would need to merge it with Ceylon config etc.
• Create an aether.xml in ~./ceylon and by default populate it with Maven Central. Only keep it in sync with .ceylon/config aether entries. The up side is that if the user later becomes a Maven fan, we will automatically choose ~/.m2/settings.xml before the aether.xml. The aether.xml would just be a dump of Ceylon config aether lines in settings xml repo format.

In the second way, we never prompt the user to enter a settings.xml - just prompt for a default remote Maven repo in case this is the first aether repo being added and no settings.xml was found -- and pre-populate with Maven Central. With the right defaults and cues, it should be good UI experience and still safe for experienced developers.

[FroMage]

I'm pretty sure we already ship a default settings.xml file

[akberc]

we should improve ease of specifying a non-maven-central maven repo with aether. Without having to define a settings.xml file ;)

@FroMage Sorry, the two options were in reference to this comment.

[quintesse]

@akberc we should really split this into separate issues I think:

• remove mvn: support (make it an alias for aether:)
• I'm pretty sure aether: does proper caching but in case you can show it doesn't it would need an issue as well
• an issue for fixing Eclipse hanging:
  • one sub-issue of this is the progress support for the CMR
  • another one is the possibility to cancel a CMR action in progress
  • and I'm also pretty sure there are already network timeouts, but they might be pretty long so it must also be investigated if we can make Eclipse not hang, networks being what they are there's always the possibility that things will be slow or that they'll fail

[akberc]

Aah, circles and circles :) Specifically,

• OK
• Yes, aether caches in .m2
• In this case, the main cause of hanging and extreme slowness was the mvn not caching and going out remotely for every lookup.

The use cases we are trying to solve are:

• A developer starts working with Ceylon and does not use Maven. He clones a great Ceylon app that uses Java under the covers - for example ceylon.net. Before you say 'the java dependencies for ceylon.net are in Herd', let us agree that they are not all going to be in Herd.
• Or, a developer is excited that she can use java artifacts from JBoss Maven repo. or the corporate Maven repo.
• And then, there are transitive dependencies, which I still have to test with aether.

Without sending a non-Maven user into a crash course on settings.xml, or trying to understand jar installation through the toolset, we should try to make CMR very transparent.

The only options that come to mind are:

• Herd should proxy to Maven Central, JBoss and a couple other major repos, and try to create module.xml files on the fly in its cache. Then we forget about Maven.
• mvn:http goes out to get a maven artefact and installs the jar in the ceylon repo (caching mvn:http and really simple to do as import-jar tool is there
• provide an easy way to configure aether through Ceylon tools - as @Fromage suggested.

IMHO, I don't think we can be ready for 1.0- without addressing this heads-on.

My test case :) : start a blank CI VM, check out a Ceylon project from that relies on Java and with non-default repos and try to build it. What can the said project put in its .ceylon folder in order to build without human intervention?

[quintesse]

IMO if that user has a corporate Maven Repo they will already have it in their system settings.xml, they won't have to do anything. If they don't have it they can ask a colleague that uses Maven how they set up their settings.xml and do the same.

But I agree that being able to easily add an extra Maven repo to the aether lookup would be really nice to have. But I doubt that it will be a big problem if we don't have it before 1.0, there's more than enough artifacts in the standard maven repo and it is possible to use others by adding them to a settings.xml file (which you can distribute with the project so users downloading it don't have to do anything).

So we can add an extra issue for adding for example a [maven] section to the Ceylon config file. Where we can put repository=http://..... items. And we might also extend the aether: (alias mvn:) "protocols" with the same thing, for example mvn:http://url.to.remote.site to add a remote repository.

[akberc]

Yup!

If you can cut out Shrinkwrap, programmatically access aether, use aether-spi extensions to read ~/.ceylon/config and write to .ceylon/cache, and possibly write out module.xml, you have this nailed.

Shrinkwrap is using aether for a very narrow case and does not provide the flexibility that CMR needs.

[quintesse]

No no, let's not do that! Leave Maven alone as it is please! People have set up their Maven environment, have their proper caches, their proxies, their repositories, all neatly set up and then we come along and do it all our way. Not a good idea!

And we only just added Shrinkwrap, so please let's first just focus on 1.0, which is to make sure that things "just work (tm)". If you really want to think about grand refactorings, schedule them for 1.2 ;)

[akberc]

OK, I understand - stability is paramount at this stage.

I solved the immediate problem by figuring out how to bootstrap the excellent https://github.com/loicrouchon/ceylon.build build module and planning a Ceylon custom task to download Maven jars and creating module.xml files -- thereby achieving the goal of a self-contained build with pure Ceylon - no Ant and no Maven in the build.

The timeout and networking issues of CMR remain and exhibit in strange ways. I will update relevant issues on findings.

Thank you for your help and time in discussing this.