Please publish jMeter artifacts on Maven central repository

asfimport commented 14 years ago

Nicolas De loof (Bug 49753): Building tools around jMeter (maven plugin, Hudson plugin) requires to have the artifacts available on public repository. Please upload jMeter one's at http://repository.apache.org/

Created attachment mavenPOMs.zip: proposal Maven POMs

Votes in Bugzilla: 5 OS: Windows XP

asfimport commented 14 years ago

Sebb (migrated from Bugzilla): Does not work for me:

[INFO] snapshot org.htmlparser:htmlparser:2.0-SNAPSHOT: checking for updates from htmlparser [INFO] snapshot org.htmlparser:htmlparser:2.0-SNAPSHOT: checking for updates from apache.snapshots Downloading: http://htmlparser.sourceforge.net/m2-snapshot-repo/org/htmlparser/htmlparser/2.0-SNAPSHOT/htmlparser-2.0-20 060923.183906-2.pom

[INFO] ------------------------------------------------------------------------ [ERROR] BUILD ERROR [INFO] ------------------------------------------------------------------------ [INFO] Failed to resolve dependencies for one or more projects in the reactor. Reason: Unable to get dependency informat ion: Unable to read the metadata file for artifact 'org.htmlparser:htmlparser:jar': Parent: org.htmlparser:htmlparser:ja r:2.0-20060923.183906-2 of project: org.htmlparser:htmlparser has wrong packaging: jar. Must be 'pom'. for project org.h tmlparser:htmlparser org.htmlparser:htmlparser:jar:2.0-SNAPSHOT

from the specified remote repositories: apache.snapshots (http://repository.apache.org/snapshots), central (http://repo1.maven.org/maven2), htmlparser (http://htmlparser.sourceforge.net/m2-snapshot-repo)

Path to dependency: 1) org.apache.jmeter:ApacheJMeter_http:jar:2.4

Also, there are no source jars. I don't know if this is acceptable anymore.

asfimport commented 14 years ago

Nicolas De loof (migrated from Bugzilla): htmlparser has an issue in its pom : it set <parent> <artifactId>HTMLParser</artifactId> <groupId>org.htmlparser</groupId> <version>2.0-SNAPSHOT</version> </parent>

where parent is HTMLParserProject

I can reproduce you issue with maven 2.2.1. Using maven 3.0-beta I only get a warning : [WARNING] Invalid POM for org.htmlparser:htmlparser:jar:2.0-SNAPSHOT, transitive dependencies (if any) will not be available, enable debug logging for more deta

I4m in contact with htmlparser devs to improve the maven build and release 2.0, I'll report this issue. Please check with a recent maven 3 build

The attached POM are only targetted to manually create the upload bundles, not to create the bundles from maven with attached sources/javadocs, that would require the project to be allready mavenized.

asfimport commented 14 years ago

Nicolas De loof (migrated from Bugzilla): htmlparser metadata just have been fixed on snapshot repo You may need to call maven with -U to force update

asfimport commented 13 years ago

Yoann Ciabaud (migrated from Bugzilla): Is this feature planned to be integrated to release process?

asfimport commented 13 years ago

Yoann Ciabaud (migrated from Bugzilla): The htmlparser snapshot repository seems to be down, if so you can replace in src/protocol/http/pom.xml : http://htmlparser.sourceforge.net/m2-snapshot-repo

By a mirror located on: http://acris.googlecode.com/svn/snapshot-repository/

I submit an updated archive with the modification.

asfimport commented 13 years ago

Yoann Ciabaud (migrated from Bugzilla): As the htmlparser repository is down, you will find in this archive a fully working maven integration.

Created attachment mavenPOMs_mirror.zip: Maven poms with a mirrored repository for htmlparser

asfimport commented 12 years ago

mark (migrated from Bugzilla): I've taken the POM's originally supplied in this BUG and tweaked them a bit to work with 2.5.1.

It's all checked into github and there is a pull request to include it here:

https://github.com/apache/jmeter/pull/2

asfimport commented 12 years ago

mark (migrated from Bugzilla): This is a patch file containing the changes made in this pull request:

https://github.com/apache/jmeter/pull/2

Created attachment bug-49753.patch: Patch file of the change linked to in previous comment.

asfimport commented 12 years ago

Sebb (migrated from Bugzilla): (In reply to comment 8)

Created attachment 28133 [details] Patch file of the change linked to in previous comment.

This is a patch file containing the changes made in this pull request:

https://github.com/apache/jmeter/pull/2

Thanks, but unfortunately the patch files do not apply at all.

We need unified diff files, or alternatively, since they are all new files, could you provide them as plain files in the correct relative directory structure?

asfimport commented 12 years ago

mark (migrated from Bugzilla): I'll try again, here they are in a rar.

Created attachment jmeter.rar: Added files zipped up in correct directory structure

asfimport commented 12 years ago

mark (migrated from Bugzilla): Updated the archive to add in the following modules:

junit monitor examples

Created attachment bug-49753.rar: Updated Archive

asfimport commented 12 years ago

Sebb (migrated from Bugzilla): Finally managed to try the patch.

It failed when compiling core, because the xalan dependency is missing. Fixed that, and was able to install the JMeter jars locally.

However, the poms don't actually use jars create by the Ant, so the Maven jars are not the same as the non-Maven jars. The build process creates a lot of additional target directories, which will have to be added to SVN ignore.

Also, there 18 poms, AFAICT each of which has to be updated when changing release version.

It's going to be very time-consuming maintaining this parallel but separate Maven build, so I'm not keen to include it.

Would it be possible to create a single pom to create the Maven artifacts from the dist area?

Or even better, an Ant script to do the same?

The existing Ant build already creates all the required jars, so it should not be too difficult to package these up for Maven.

I think the Tomcat project does something similar.

asfimport commented 12 years ago

mark (migrated from Bugzilla): Generally it's good practise in maven to define dependancies for each module and not have them all declared in one place that all the modules pick the dependancies up from, hence why it is done this way.

That being said it is possible to have all the project dependancies in one place.

I take it that JMeter is developed as one big project with libraries being updated across all components when a new feature is needed.

I'll have a look at simplifiying it and making it more maintainable and then resubmit.

asfimport commented 12 years ago

Sebb (migrated from Bugzilla): The Poms are still going to have to be generated somehow, so there is potential for defining customised unversioned dependencies for each jar, with the full list of versions in the generated parent pom.

Ideally the information should be taken from build.properties, so we don't have to maintain lots of files.

asfimport commented 12 years ago

mark (migrated from Bugzilla): Rejigged the POM files a bit.

All dependencies are now declared in the parent POM with the various modules inheriting them. Multiple target folders are history, it now builds all projects in <jmeter_home>/build/maven/<module>. Default version has been set to "SNAPSHOT".

To update the version of all POM's is a one liner, in the root jmeter folder type:

mvn versions:set -DnewVersion=2.5.1

The component POM's should only ever need to be updated in two instances

One of the components needs to depend upon one of the other components
You add a properties file to a directory that does not currently have any properties files in (This will require a new <resource> definition)

I have had a look at reading in the dependancies from build.properties but whilst maven can read in external properties, it can't do this before the dependecies are checked as valid so the dependencies need to be defined inside the POM file.

One possible option for a single source of truth would be to get ANT to use the pom.xml in the JMeter root folder instead of the build.properties, it looks like you could parse the pom file with ant and pull out dependency information. I'll investigate this next to see if it is a realistic option.

If you implemented the files as they currently are you would need to ensure that any dependency changes are made in both the build.properties and the root pom.xml

Created attachment bug-49753-r2.rar: Updated POM files

asfimport commented 12 years ago

@pmouawad (migrated from Bugzilla): Another option would be to make ANT run Maven, I already tried this approach and it works rather well.

But this means we switch to Maven build, this has been voted against previously.

asfimport commented 12 years ago

@pmouawad (migrated from Bugzilla): Regarding : mvn versions:set -DnewVersion=2.5.1

This tasks updates all pom.xml with current version, but once modified and a version released, we need to put back SNAPSHOT, how would we do this ? I tried running : mvn versions:revert and it failed.

asfimport commented 12 years ago

@pmouawad (migrated from Bugzilla): Date: Thu Jan 12 22:17:30 2012 New Revision: 1230815

URL: http://svn.apache.org/viewvc?rev=1230815&view=rev Log: https://github.com/apache/jmeter/issues/2392 - Please publish jMeter artifacts on Maven central repository Modified js version to 1.7R2

Added: jmeter/trunk/pom.xml (with props) jmeter/trunk/src/components/pom.xml (with props) jmeter/trunk/src/core/pom.xml (with props) jmeter/trunk/src/examples/pom.xml (with props) jmeter/trunk/src/functions/pom.xml (with props) jmeter/trunk/src/jorphan/pom.xml (with props) jmeter/trunk/src/junit/pom.xml (with props) jmeter/trunk/src/protocol/ftp/pom.xml (with props) jmeter/trunk/src/protocol/http/pom.xml (with props) jmeter/trunk/src/protocol/java/pom.xml (with props) jmeter/trunk/src/protocol/jdbc/pom.xml (with props) jmeter/trunk/src/protocol/jms/pom.xml (with props) jmeter/trunk/src/protocol/ldap/pom.xml (with props) jmeter/trunk/src/protocol/mail/pom.xml (with props) jmeter/trunk/src/protocol/tcp/pom.xml (with props) jmeter/trunk/src/reports/pom.xml (with props)

asfimport commented 12 years ago

@pmouawad (migrated from Bugzilla): I commited pom.xml with a change on js version (1.7R2 for trunk). I don't mark issue as fixed as there is still some work, but that way we can update pom.xml until we find the good solution

Thank you.

asfimport commented 12 years ago

mark (migrated from Bugzilla): Looks like it's very easy to get ant to use the POM file to download dependencies, you just need the to use maven-ant-tasks:

http://maven.apache.org/ant-tasks/

I created a directory called scratch in the root jmeter folder and then created a build.xml with the following in it (This is assuming you have the latest set of POM files I have provided):

<?xml version="1.0"?>

&lt;path id="maven-ant-tasks.classpath" path="&lt;path.to&gt;/maven-ant-tasks-2.1.3.jar"/&gt;
&lt;typedef resource="org/apache/maven/artifact/ant/antlib.xml" uri="antlib:org.apache.maven.artifact.ant" classpathref="maven-ant-tasks.classpath"/&gt;

&lt;target name="download_packages" description="Run the stand-alone Findbugs detector"&gt;
    &lt;artifact:pom id="jmeter.pom" file="../pom.xml"/&gt;
    &lt;artifact:dependencies filesetId="jmeter.dependencies" pomRefId="jmeter.pom"/&gt;
    &lt;copy todir="lib"&gt;
        &lt;fileset refid="jmeter.dependencies"/&gt;
        &lt;!-- This mapper strips off all leading directory information --&gt;
        &lt;mapper type="flatten"/&gt;
    &lt;/copy&gt;
&lt;/target&gt;

</project>

If you run this is steps up a directory to the POM file, reads the dependency list and downloads the dependencies and copies them into the lib dir. very quick, very easy and would make the jar download sections of your build.xml obselete.

I've attached a rar with the scratch implementation so that you can give it a quick try. All you need to do is download maven-ant-tasks-2.1.3.jar from here:

http://www.apache.org/dyn/closer.cgi/maven/binaries/maven-ant-tasks-2.1.3.jar

and stick it in the scratch folder.

Is this worth pursuing? If so I'll start tweaking the build.xml

Created attachment alternate_dep_management.rar: using maven-ant-tasks to download dependencies

asfimport commented 12 years ago

mark (migrated from Bugzilla): Philippe, you should be able to just do a

mvn versions:set -DnewVersion=SNAPSHOT

to change all the versions again (although revert should work)

asfimport commented 12 years ago

@pmouawad (migrated from Bugzilla): I had tried this it failed. Regarding ant tasks, I am ok with it , but let's wait for Sebb opinion.

asfimport commented 12 years ago

mark (migrated from Bugzilla): I've updated the parent POM file to add in the required info for it to be added to the central maven repo.

I have taken the info for the developer section from http://wiki.apache.org/jmeter/JMeterCommitters so I'm assuming that is up to date. It probably needs checking anyway to be sure.

I haven't added a contributor list as it isn't required, but I can do if it is wanted.

I guess what I need now is a list of chnages that are required to what I have submitted so far. I currently have the following todo list:

Update build.xml to use POM instead of build.properties (Awaiting confirmation from Sebb that this is the way to go)
Investigate getting jChart 0.7.5 into central maven repo
Investigate getting bsh 2.0b5 into central maven repo
Removes the repositories block so that it can be uploaded to central
Double check the contants of all jars created by maven and normal build process and see if there are any differences that should not be there

The final question is how to get this added to maven central. The standard route for OSS projects seems to be via sonatype (as a result I have added in the sonatype repos in the dependency management section).

Is there somebody who already deals with this? Is there anything I can do to help with it?

Created attachment pom.xml: Main POM Update

asfimport commented 12 years ago

mark (migrated from Bugzilla): Philippe, I have attached the console log showing the mvn versions:set commands I used and what they did.

Does this match your usage?

Created attachment versions.txt: Output of versions:set command

asfimport commented 12 years ago

Sebb (migrated from Bugzilla): I don't like the idea of rebuilding the sources in order to create the Maven jars.

The Ant build creates all the binary jars required for Maven; these should be used in the uploads, rather than rebuilding them using Maven.

We are not trying to convert the build to use Maven here, only to automate the deployment of Maven artifacts.

I need to find out how Tomcat do this, because they also use Ant to build; as far as I know they also use Ant or a shell script to deploy to Maven.

asfimport commented 12 years ago

mark (migrated from Bugzilla): I guess I need to do some more research, i'm aware of a way to upload existing jars to a maven repo, i'm not aware of a way to upload existing jars and magically work out the dependecies of the jar (which is why i'm making maven build in paralell with ant).

The intention is not to replace the ant build in any way, only to enable the deployment of maven artifacts.

Off to look at Tomcat I guess...

asfimport commented 12 years ago

Richard Calmbach (migrated from Bugzilla): In case you haven't seen this, look at [1]. It has instructions for uploading artifacts to the Maven Central snapshot and staging repos with Ant. The parallel Maven build would then only serve the purpose of validating that the pom.xml lists all the dependencies, but the uploaded artifacts would be those built by Ant. That wiki page also describes other ways to deploy to Central.

Disclaimer: I haven't done this myself, but I previously researched this while trying to convince the Weka project to upload their artifacts to Central.

Ref.: [1] https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide#SonatypeOSSMavenRepositoryUsageGuide-7c.DeploySnapshotsandStageReleaseswithAnt [2] http://www.sonatype.com/people/2009/06/publishing-your-artifacts-to-the-central-maven-repository/

asfimport commented 12 years ago

Sebb (migrated from Bugzilla): The Tomcat project only use Ant, see:

http://svn.apache.org/repos/asf/tomcat/tc7.0.x/trunk/res/maven/

We'll also need to set up Nexus for JMeter. I can start on that next week.

asfimport commented 12 years ago

Sebb (migrated from Bugzilla): I raised the JIRA issue to set up Nexus:

https://issues.apache.org/jira/browse/INFRA-4332

I'm also looking at the Tomcat Ant script with a view to using it in JMeter.

asfimport commented 12 years ago

mark (migrated from Bugzilla): OK, I've gone through the Tomcat implementation and they seem to be basically using ant to mirror the functionality of the mvn deploy:deploy-file command.

The downside of this (from what I can see) would be that either the dependencies are not specified in the POM files, or all dependecies are specified in all POM files (It would seem all dependencies are in all POM files unless i'm reading it wrong) unless you start writing something to apply specific dependencies to specific POM files.

Based upon this I have created the attached.

This will use ant to deploy the locally build ant files by executing the command line mvn deploy:deploy-file command from ant. I have so far got it building all of the componenets, I need to get it to create the parent POM files which seems to be a little more tricky.

Adding javadoc and sorces should be as simple as adding an extra attribute to the exec command to specify the javadoc/source jars. To make the whole thing cleaner you could probably plug in the base project build.xml and make the deployment depend upon various jars being built.

The final piece of the puzzle would be to create the mainpom file which has the dependencies and make all of the componenets depend upon it which I am working on at the moment.

Is this implementation what you are looking for?

Created attachment ant_deploy.rar: ant deploy implementation

asfimport commented 12 years ago

Sebb (migrated from Bugzilla): (In reply to comment 30)

Created attachment 28162 [details] ant deploy implementation

OK, I've gone through the Tomcat implementation and they seem to be basically using ant to mirror the functionality of the mvn deploy:deploy-file command.

The downside of this (from what I can see) would be that either the dependencies are not specified in the POM files, or all dependecies are specified in all POM files (It would seem all dependencies are in all POM files unless i'm reading it wrong) unless you start writing something to apply specific dependencies to specific POM files.

AIUI they use individual pom files which include the required dependencies.

Based upon this I have created the attached.

This will use ant to deploy the locally build ant files by executing the command line mvn deploy:deploy-file command from ant.

I was hoping to avoid needing to install Maven, which is what the Tomcat version achieves.

I have so far got it building all of the componenets, I need to get it to create the parent POM files which seems to be a little more tricky.

It should not build any components, only upload them.

Adding javadoc and sorces should be as simple as adding an extra attribute to the exec command to specify the javadoc/source jars. To make the whole thing cleaner you could probably plug in the base project build.xml and make the deployment depend upon various jars being built.

The final piece of the puzzle would be to create the mainpom file which has the dependencies and make all of the componenets depend upon it which I am working on at the moment.

Why do you need the parent POM if all the jars are uploaded?

Is this implementation what you are looking for?

Not exactly.

asfimport commented 12 years ago

mark (migrated from Bugzilla): The individual pom files in this implementation have the jmeter dependencies set, however they do not have the external library dependencies set. This is why a parent pom would be required to to set the external dependencies that all components can depend upon.

Build was a bad choice of words, what I meant to say was deploy. This will take the existing jars built by ant and deploy them to a maven repository.

As for not having maven installed at all, you could do that, but then you are rewriting mavens deploy mechanism using ant and if maven ever changes anything you will need to modify your ant implementation to match the changes.

Is it really worth all the extra hassle? That sounds like a much bigger maintenance nightmare than previous suggestions.

asfimport commented 12 years ago

Sebb (migrated from Bugzilla): (In reply to comment 32)

The individual pom files in this implementation have the jmeter dependencies set, however they do not have the external library dependencies set. This is why a parent pom would be required to to set the external dependencies that all components can depend upon.

I see.

So you are not proposing using different library dependencies for different JMeter jars?

Thee will need to be some way to synchronise the version settings in build.properties and the parent POM - or at least there needs to be a simple way to detect if the settings are different.

Perhaps use properties in the pom to define the versions; it should then be fairly easy to extract the versions from build.properties and the pom and compare the lists.

Build was a bad choice of words, what I meant to say was deploy. This will take the existing jars built by ant and deploy them to a maven repository.

OK.

As for not having maven installed at all, you could do that, but then you are rewriting mavens deploy mechanism using ant and if maven ever changes anything you will need to modify your ant implementation to match the changes.

It's only copying files to a URL. Tomcat already does it.

Is it really worth all the extra hassle? That sounds like a much bigger maintenance nightmare than previous suggestions.

AFAICT no extra maintenance.

However it is harder to set up, so perhaps start with the Maven CLI. If that causes problems for JMeter devs. then rework to use Ant only.

asfimport commented 12 years ago

Ian Brandt (migrated from Bugzilla): Greetings,

I've been following this bug as an interested party. I help maintain the Tomcat Embedded container for Arquillian, where Tomcat is used as a pure Maven dependency. I've also taken to publishing to Maven Central the Eclipse JDT Core Batch Compiler, a Tomcat dependency that was otherwise missing. Also at my company we produce an internal Tomcat distribution as a Maven assembly using only the Tomcat artifacts published to Maven Central. I was hoping to do something similar with JMeter, which is how I found my way to this bug.

In my experience while the Tomcat approach to publishing artifacts with the Maven Ant Tasks is okay, it suffers from the fact that they don't regularly build and test under the dependencies as expressed by the published POMs. Also while their approach does remove the need to install Maven along with Ant, there is still duplication involved in the build code, and the usual "Don't Repeat Yourself" issues inevitably come to bear. I can offer up current and past examples if anyone is interested.

I'm guessing that an outright switch to Maven for JMeter is not on the table. If it is let me know and I'd be glad to help with the port. You'd gain the use of the Maven Dependency Plugin (the dependency:analyze and dependency:tree reports are very helpful when it comes to understanding project dependencies and producing correct POMs). You'd no longer have to maintain eclipse.classpath as Eclipse (and other IDEs) can work straight off the POMs. Having skimmed the JMeter build.xml I'd guess the resulting set of POMs would be about 1/3 or less the lines of build code to maintain. I don't see much that wouldn't be trivial to convert. FindBugs and Checkstyle should all port over just fine. I believe RAT and the xdocs would port as well, but otherwise the Maven AntRun Plugin could still be used. I don't have any experience working with Gump, so I can't speak to the level of effort for that part of a switch.

Otherwise there is another approach that I haven't seen mentioned. Dependency management could be completely pushed down to the Maven Ant Tasks or Apache Ivy. Either could be bootstrap installed with the Ant Get Task, and could be used from Ant to specify dependencies, download them, provide corresponding compile/test/runtime classpaths, and generate and publish suitable POMs along with the JMeter artifacts. There would be no duplication in dependency declarations, and no maintaining parallel POMs. You'd still cut down on your build code a bit I believe, and with Ivy you could use IvyDE to still get away from eclipse.classpath as well.

Regards,

Ian

asfimport commented 12 years ago

Sebb (migrated from Bugzilla): (In reply to comment 34)

I'm guessing that an outright switch to Maven for JMeter is not on the table.

This has already been discussed on the JMeter dev list. See also https://github.com/apache/jmeter/issues/2429.

I don't see much that wouldn't be trivial to convert.

I agree that source compilation, findbugs reports etc. would likely be easy to do.

However, there are several other non-standard elements in the JMeter build, e.g. the site docs and help files are built from the same xml files.

asfimport commented 12 years ago

Ian Brandt (migrated from Bugzilla): (In reply to comment 35)

This has already been discussed on the JMeter dev list. See also https://github.com/apache/jmeter/issues/2429.

Thanks for the bug and mailing list references. I've read through it all and it was very helpful.

Accordingly I've moved discussion of the Maven Ant Tasks or Apache Ivy approach to the dev list: http://mail-archives.apache.org/mod_mbox/jmeter-dev/201201.mbox/%3C30C6A9A8-08C0-461F-9A1B-362BCA69A515%40ianbrandt.com%3E.

asfimport commented 12 years ago

mark (migrated from Bugzilla): I've added where I currently am with an ant only based maven deploy.

Dependencies are:

Maven must be installed on your system (so that you can use the "mvn" command in the terminal/command prompt)
You must have ant-contrib in your ant lib directory (you can download it from here http://sourceforge.net/projects/ant-contrib/files/)

Currently it only deploys the source files, it should be trivial to add sources and javadoc jars if they are created by the standard ant build.

You set the repo to deploy to by modifying mvn.properties (the url property). This works fine for a deploy to a local file system in windows and Linux, you may need to add some login/key information to deploy to a remote host.

Created attachment maven.rar: Current maven deploy via ant

asfimport commented 12 years ago

mark (migrated from Bugzilla): Added in my final version of maven deploy.

This one will automatically download ant-contrib so you don't need to worry about having it in your classpath in advance.

It still requires you to have mvn installed.

I've tweaked the POM files a bit to fix some transitive dependency problems I was seeing with the built artifacts. I have a copy of the artifacts that have been built using this code here:

http://ardesco.github.com/jmeter-maven-plugin/repository/org/apache/jmeter/

All the dependencies are defined in ApacheJMeter_parent this means that there are now two places where dependencies are declared. Here for the mvn deployment and in build.properties for the standard ant build process.

Created attachment maven_deploy.rar: Maven deploy final

asfimport commented 12 years ago

Sebb (migrated from Bugzilla): As far as I can tell, the poms and associated Ant build.xml deployment script in the JMeter res/maven directory are now fully up-to-date and working.

Source files and javadoc are not included, but they are not necessary merely for running JMeter from Maven.

If there are any fixes needed, please re-open with a patch against JMeter SVN trunkc.

apache / jmeter

Please publish jMeter artifacts on Maven central repository #2392