search

akhikhl / wuff

Gradle plugin for automating assembly of OSGi/Eclipse bundles and applications
MIT License
152 stars 51 forks source link

Wuff must support p2 repositories as sources of OSGi/Eclipse bundles #9

Open akhikhl opened 10 years ago

akhikhl commented 10 years ago

...but we need to discuss use-cases and come to specification first.

pdeschen commented 10 years ago

Another p2 support use case would be to add the ability to create a p2 eclipse repository.

References:

akhikhl commented 10 years ago

Will the repository being created reside in the file system or http?

pdeschen commented 10 years ago

To be frank I don't fully grasp the p2 repo beast. I think this use case can be broken down into more sub use cases where we would have:

  1. standalone p2 repo creation (where no other repos are involved within the metadata) on FS
  2. p2 repo creation on FS with reference to some other remote repo (http or FS)
  3. publishing (http)

From my perspective, 1 would be a must while 2, 3, would be less so but I guess YMMV.

tschulte commented 10 years ago

From my perspective, the creation of p2 repository is less important than the consumtion of p2 repositories.

If you really need to create a p2 repository, you can as a workaround just use maven tycho in pom-first dependency resolution mode using your libs created by wuff.

Consumption is more important, because the eclipse folks do not deploy their jars to any maven repository, and if you want to use any of the non standard dependencies, you are doomed. We for example use NatTable and Nebula widgets. They are only hosted in p2 repositories. AFAIK Wuff only downloads one of the ready made eclipse distribution-zips and uses the plugins of that distribution.

Also I am not a big fan of having mavenLocal() as repository in my build, because this introduces the risk, that the build works on my machine, but not on CI. Wuff, as far as I understand, mavenizes the plugins after downloading the zip and uploads them to mavenLocal().

Tycho can use both pom-first, and manifest-first dependencies, and you can add p2 repositories to lookup the dependencies. But using mixed pom-first/manifest-first or pom-defined repositories at the same time seems to make problems -- at least at the time you import the project into eclipse. Eclipse does only use the target definition for dependency resolution, and does not consider the definitions made in the pom.

Therefore the best practice seems to be having a .target file in the build, referencing it in the pom.xml, and using this target file also in the IDE. In the target file you define the p2 URL and the ids of the installable units. E.g. 

<location includeAllPlatforms="false" includeConfigurePhase="true" includeMode="planner" includeSource="true" type="InstallableUnit">
    <unit id="org.eclipse.rcp.feature.group" version="4.4.0.v20140522-1614"/>
    <unit id="org.eclipse.emf.feature.group" version="2.10.0.v20140519-0339"/>
<repository location="http://download.eclipse.org/releases/luna"/>

The installable unit defines, which features/plugins in which versions are available. So this is essentially a subset of the repository. It is essentially a filter on both what can be used, and in which versions.

Before importing the project into eclipse, you than first set the target in eclipse (open in eclipse, click "set as target platform").

Currently you don't have to declare dependencies to org.eclipse.* bundles in build.gradle, they are somehow automatically added. If you want to have this feature also for dependencies in a p2 repository, I think you will have to use the .target file approach with defining installable units (a filtered view on the repository). If you can live with defining the dependencies in the build.gradle, you could live with just using the p2 url -- When you import the project in eclipse, gradle will be responsible for dependency resolution, not Eclipse PDE, so both ways might work great.

tschulte commented 10 years ago

I just had another issue with unpuzzle. I wanted to use two sources http://download.eclipse.org/efxclipse/runtime-nightly/site_assembly.zip and http://download.eclipse.org/efxclipse/experimental-nightly/site_assembly.zip. This is not supported, because the name of the zip file is used in the wuff-dir.

This problem would vanish, if gradle would support p2 repositories. But I think, p2 repository support might be best implemented in gradle itself. I don't know if following would be possible when done in a plugin.

repositories {
    p2 {
        url 'http://download.eclipse.org/efxclipse/runtime-nightly/site'
        // optionally define installable units to use
        unit id: 'org.eclipse.e4.ui.workbench' version: '1.1.0.v20140528-1949'
        // more units
    }
    p2 {
        // use target file
        from 'targets/luna.target'
    }
}
tschulte commented 10 years ago

I asked on the gradle mailing list. http://gradle.1045684.n5.nabble.com/Adding-support-for-p2-repositories-td5713110.html. I will now focus on creating some design-docs in my fork of gradle and see if I can implement it in gradle.

pdeschen commented 10 years ago

@tschulte looks like from Adam's answer, that won't be a walk in the park...

@akhikhl Maybe this issue could be split in two: consuming p2 repository + publishing p2 repo ?

akhikhl commented 10 years ago

Building p2 repos is easier than consuming, so I will implement it soon. Consuming p2's - I'll need to read relevant API docs and see how p2 dependencies could be mapped to maven/gradle "on the fly". Eclipse Tycho does it, that means - it is doable.

tschulte commented 10 years ago

In http://stackoverflow.com/questions/12601526/where-can-i-find-the-p2-repository-specification the accepted answer says, there is no specification, only the implementation. And the other answer says it is practically unusable without OSGI.

akhikhl commented 10 years ago

I'm looking at PDE-build right now. It has no API and is driven by some ant scripts. Sigh.

akhikhl commented 10 years ago

...I've seen this phenomenon sometimes. Some poor guy develops piece of software, thinking it is a prototype. Then comes jolly product manager and says: "Prototype completed? Wonderful, now lets call it a product and start marketing".

pdeschen commented 10 years ago

@akhikhl pde-build is a mess. There's also buckminster if you're curious.

aalmiray commented 10 years ago

perhaps looking at the code from https://github.com/reficio/p2-maven-plugin can help? I know the guy behind the project or you can ping him directly (@reficio). I'm also interested in seeing p2 support in gradle, alas gradleware is currently focusing in other areas. I can spare some time to help pushing wuff/unpuzzle forward.

akhikhl commented 10 years ago

p2-maven-plugin - wow, that's a nice project, worth close look at the implementation and learning from it. Pushing wuff/unpuzzle - yes, please! I am currently busy implementing generation of Eclipse features and repositories in Wuff. I can imagine, for many people this would be a "tip point", after which they would decide to switch to Wuff or invest their time into Wuff.

akhikhl commented 10 years ago

Dear colleagues, You are heartily welcome to Wuff discussion group: https://groups.google.com/forum/?hl=en#!forum/wuff-dev I invite you to discuss design and implementation of Wuff. Our goal is to develop the much needed instrument making Eclipse RCP/IDE development simpler and faster. Suggest your topics: wishes, ideas, experiences.

akhikhl commented 10 years ago

Early preview of Eclipse Feature and Repository generation is now available in Wuff 0.0.13-SNAPSHOT. I created set of example projects, showing this feature at work, at: https://github.com/akhikhl/wuff-sandbox Your critique, comments and test results are very much welcome.

kdoteu commented 9 years ago

I think we should add an extra feature request for p2 consuming. So this one is only for generation of a Eclipse Repository?

petarov commented 9 years ago

What @kdoteu said would be awesome to have! I'm also trying to find a way to support p2 repositories.

akhikhl commented 9 years ago

Developing native P2 support right now.

petarov commented 9 years ago

@akhikhl Amazing! Thanks for taking it into consideration. My only option atm is to convert P2 repo to Maven repo, which doesn't seem so easy so far.

igor-ryabchikov commented 9 years ago

@akhikhl tell please the approximate date when p2 repositories support will be available?

golauty commented 9 years ago

@akhikhl Did you make any progress in that topic? @petarov What approach do you currently use to transfer a p2 repo to maven?

petarov commented 9 years ago

@golauty I'm not sure what your use case is. Mine was that I already had plugins and features folders and I wanted to use them at compile time, otherwise building failed. Here is what I tried:

  1. (This worked) I converted my plugins and features folders to a P2 repo with meta data then I uploaded only the plugins (10 or more) I was interested in, in a Nexus Maven2 repository. This is a slow and burdensome process.
  2. Tried to write a mavenizing script, but then I saw that @akhikhl already wrote something like that here. I haven't tried it though. If you do, please share some info.
  3. (Solved my problem) I actually got an idea from this bug here. I extended the gradle wuff task and specified my own source parameter that points to a P2 zipped plugins directory. Worked like a charm.

Hope this helps.

golauty commented 9 years ago

@petarov I used method 2 to mavenize my bundles. I had a large folder with multiple jars and managed to use a local zip file. I used following script:

apply from: 'https://raw.github.com/akhikhl/unpuzzle/master/pluginScripts/unpuzzle.plugin'

unpuzzle {

  localMavenRepositoryDir = new File(System.getProperty('user.home'), 'Documents/Java Libs/P2Repository/Repo')

  unpuzzleDir = new File(System.getProperty('user.home'), '.unpuzzle')

  selectedEclipseVersion = '4.4.2'

  eclipseVersion('4.4.2') {

    eclipseMavenGroup = 'eclipse-luna-sr2'

    eclipseMirror = 'http://mirror.netcologne.de'

    eclipseArchiveMirror = 'http://archive.eclipse.org'

    sources {

      source "file:///C:/Users/userXY/Documents/Java Libs/P2Repository/LargeZipFile.zip"

    }
  }
  }

Actually that worked, but the unpacked zip File in the folder .unpuzzle\unpacked was not correct. He did not use the root folder, but after manually transfering it to a plugins folder everything worked.

Did you use the nexus p2 repository plugin or something else? I would be glad if you can share some information about solution one. I will try this next.

petarov commented 9 years ago

@golauty 10x for the example! I guess I use something pretty similar with the wuff plugin, but I haven't used unpuzzle so far.

wuff {
    localMavenRepositoryDir = new File(System.getProperty('user.home'), '.wuff/m2_repository')

    wuffDir = new File(System.getProperty('user.home'), '.wuff')

    selectedEclipseVersion = '4.4.2.1'

    eclipseVersion('4.4.2.1') {
        extendsFrom '4.4.2'
        eclipseMavenGroup = 'eclipse-luna-sr2'

        sources {
            source "http://nexu-repository/path/p2-folder.zip"
        }
    }
}

4.4.2.1 isn't a real Eclipse version. I just use it, so I can extend from the already defined 4.4.2 version. The p2-folder.zip file contains a standard P2 layout - plugins, features, etc. No, I don't use the Nexus P2 Repository Plugin in this case. It's just ZIP file in a Maven repository. You may also not use Nexus at all in this case.