search

eclipse-birt / birt

Eclipse BIRT™ The open source reporting and data visualization project.
http://www.eclipse.org/birt
Eclipse Public License 2.0
441 stars 386 forks source link

Publish Maven Build To Central #625

Open pipebaum opened 3 years ago

pipebaum commented 3 years ago

Currently, there is no automated feature to publish the BIRT-Runtime to Maven. Wouldn't it be nice to fix this.

JacquesLeRoux commented 2 years ago

Hi @SteveSchafer-Innovent,

We are ready to create a new release of Apache OFBiz. We just miss this new publish on Maven to fix a CVE due to Birt used as a plugin. We can avoid to use the Birt plugin for this release but would, of course, prefer not. How many time do you think it will need to publish on Maven?

TIA

SteveSchafer-Innovent commented 2 years ago

I'm having problems getting the jenkins job to build. I created the branch BIRT_4.9.0_maven and created a jenkins job that is basically the same as birt-master except it refers to that branch, specifies the gpg keyring file, and invokes the release profile that I added to the pom.xml file. It fails with this error:

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-p2-plugin:2.5.0:p2-metadata (attached-p2-metadata) on project org.apache.batik.dom.birt.ext: Execution attached-p2-metadata of goal org.eclipse.tycho:tycho-p2-plugin:2.5.0:p2-metadata failed. IllegalArgumentException

which seems strange because org.apache.batik.dom.birt.ext doesn't invoke the tycho-p2-plugin plugin. I get this error whether I invoke the build-server profile or not.

More info:

If I remove the "release" profile it should, theoretically, build as it originally did, however instead it gets a test failure:

Tests run: 241, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 46.863 s <<< FAILURE! - in org.eclipse.birt.chart.tests.AllTests 2dline(org.eclipse.birt.chart.tests.device.render.ImageOutputBaseTest) Time elapsed: 5.815 s <<< FAILURE! junit.framework.AssertionFailedError at org.eclipse.birt.chart.tests.device.render.ImageOutputBaseTest.runTest(ImageOutputBaseTest.java:113)

Perhaps this is related to a change I had to make to org.eclipse.birt.target.target. The repository https://download.eclipse.org/eclipse.updates/4.23-I-builds no longer exists, so I changed it to 4.23.

SteveSchafer-Innovent commented 2 years ago

I'm getting 504 errors on ci.eclipse.org/birt. I think I broke it.

wimjongman commented 2 years ago

haha. Well done. It works fine for me: https://ci.eclipse.org/birt/

SteveSchafer-Innovent commented 2 years ago

It's working for me now too. I deleted a project and then saw that it wasn't deleted but just disabled. When I tried to re-enable it, it went out to lunch. I'm going to try it again. Update: deleting the project worked this time.

SteveSchafer-Innovent commented 2 years ago

Can I push a change to BIRT_4.9.0 branch to change https://download.eclipse.org/eclipse/updates/4.23-I-builds to https://download.eclipse.org/eclipse/updates/4.23 in org.eclipse.birt.target.target because the former no longer exists and is keeping it from building in jenkins?

wimjongman commented 2 years ago

Yes, that is fine. It should be that before we released it.

SteveSchafer-Innovent commented 2 years ago

I actually got the birt_4.9.0_maven branch to build by re-creating the project and removing the "release" profile. When I added the release profile back, it got the tycho-p2-plugin error I mentioned earlier. It's a clue! But what does it mean?

wimjongman commented 2 years ago

I see that it complains about a source that is already attached. In your profile I see this line: attach-sources

Maybe this is not needed.

wimjongman commented 2 years ago

The batik project contains the jar-signer in its own pom.xml https://github.com/eclipse/birt/tree/BIRT_4.9.0_maven/common/org.apache.batik.dom.birt.ext

Do you think that this can cause it?

SteveSchafer-Innovent commented 2 years ago

Removing the jarsigner from org.apache.batik.dom.birt.ext didn't fix the error.

SteveSchafer-Innovent commented 2 years ago

@wimjongman thanks for adding the tracebacks to the jenkins log. I see the illegal argument exception is coming from P2GeneratorImpl.getCanonicalArtifact and you can see the code here, but it doesn't mean anything to me at this point. I will try digging further.

AlexSchuetz commented 2 years ago

I am new to BIRT and I also don't know much about tycho and eclipse plugins. But I see one differeence in the poms between the two projects that work (java.xml.rpc, org.apache.axis) and org.apache.batik.dom.birt:

There is no groupid in the working projects.

I don't know if this is of any help, but maybe you should give it a try as the groupid is already in the parent pom and is redundant.

AlexSchuetz commented 2 years ago

The build.properties look also pretty different. Mayby the trailing \ and following newlines produce the error.

Since there is no hint on the internet on this IllegalArgumentException my experience tells me this is no general issue, but more like a uncommon input, to cause this.

wimjongman commented 2 years ago

Thanks, Alex! Those kinds of things could just be it.

SteveSchafer-Innovent commented 2 years ago

I removed the groupId from org.apache.batik.dom.birt.ext/pom.xml and changed build.properties a bit but it still got the error. I don't think there's anything wrong with build.properties. On the next line after the trailing backslash there's a dot so I think that's not an error. Anyway, more importantly, the contents of org.apache.batik.dom.birt.ext are identical to the 4.10 version so it seems unlikely that anything within that project is causing the error. Maybe something about p2 configuration?

claesrosell commented 2 years ago

I am not very good with Maven or Tycho, but I have tinkered with it a little in the past.

I think that the: [ERROR] Failed to execute goal org.eclipse.tycho:tycho-p2-plugin:2.5.0:p2-metadata (attached-p2-metadata) on project org.apache.batik.dom.birt.ext: Execution attached-p2-metadata of goal org.eclipse.tycho:tycho-p2-plugin:2.5.0:p2-metadata failed. IllegalArgumentException -> [Help 1] error happens because of missing metadata in org.apache.batik.dom.birt.ext-1.14.0-SNAPSHOT-sources.jar this in turn happens because we, since the introduction of the "maven-source-plugin" in our parent pom.xml, generate the source bundles twice. First with we generate the sources with "tycho-source-plugin" and then that generated source bundle is overwritten when we generate the sources again with the "maven-source-plugin".

What happens if we remove the "maven-source-plugin" from the release profile? I think that we can remove the "maven-javadoc-plugin" as well since we generate the javadocs elsewhere (with Ant).

wimjongman commented 2 years ago

Maybe it is Tycho and we try bumping the Tycho version to 2.7.1

SteveSchafer-Innovent commented 2 years ago

When I remove maven-source-plugin and maven-javadoc-plugin, it gets this error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.0.0:run (javadoc) on project org.eclipse.birt.doc.isv: An Ant BuildException has occured: The following error occurred while executing this line: [ERROR] /home/jenkins/agent/workspace/birt_4.9.0_maven/docs/org.eclipse.birt.doc.isv/BuildISVDoc.xml:201: Replace: source file /home/jenkins/agent/workspace/birt_4.9.0_maven/docs/org.eclipse.birt.doc.isv/engine/api/index.html doesn't exist [ERROR] around Ant part ...... @ 7:130 in /home/jenkins/agent/workspace/birt_4.9.0_maven/docs/org.eclipse.birt.doc.isv/target/antrun/build-main.xml [ERROR] -> [Help 1]

I will try Wim's suggestion also.

wimjongman commented 2 years ago

BuildISVDoc.xml is an ant file that depends on the javadoc being generated by the maven-javadoc-plugin.

claesrosell commented 2 years ago

@SteveSchafer-Innovent, this is actually progress. This is the same problem that, for unknown reasons, happend in master a couple of weeks ago. And was temporarily fixed with this bug: #917. @wimjongman : It is my understanding that the javadocs that BuildISVDoc.xml is dependent on is generated in BuildISVDoc.xml itself, it is not dependent on : maven-javadoc-plugin.

wimjongman commented 2 years ago

@claesrosell that could be, I just derived from what I saw. Apologies for the confusion.

SteveSchafer-Innovent commented 2 years ago

I copied the changes from pr 920. Now we have this:

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-packaging-plugin:2.7.1:package-feature (default-package-feature) on project org.eclipse.birt.chart: Execution default-package-feature of goal org.eclipse.tycho:tycho-packaging-plugin:2.7.1:package-feature failed: Duplicate key org.eclipse.swt (attempted merging values org.eclipse.tycho.model.Feature$ImportRef@d05993c2 and org.eclipse.tycho.model.Feature$ImportRef@387574f4) -> [Help 1]

Note: we are currently using Tycho 2.7.1. I should probably switch it back to 2.5.0 to eliminate that difference from 4.10.

SteveSchafer-Innovent commented 2 years ago

Amazing! After I switched back to Tycho 2.5.0 the build worked. Now to see what happened on Maven Central...

SteveSchafer-Innovent commented 2 years ago

I'm not seeing org.eclipse.birt on sonatype. I've left a question on the github issue.

claesrosell commented 2 years ago

From what I can see in the build log (https://ci.eclipse.org/birt/job/birt_4.9.0_maven/16/) we are still calling: mvn clean verify I think that one need to call mvn clean deploy To deploy to a external repository. I might be wrong though, I have never done this myself.

wimjongman commented 2 years ago

Yes, that is correct. Good catch.

wimjongman commented 2 years ago

I changed that and started the build again.

wimjongman commented 2 years ago

We are getting very close:

[INFO] Deploying remotely...
[INFO] Bulk deploying locally gathered artifacts from directory: 
[INFO]  * Bulk deploying locally gathered snapshot artifacts
[WARNING] Failed to upload checksum to org/eclipse/birt/birt-runtime/4.9.0-SNAPSHOT/birt-runtime-4.9.0-20220427.100452-1.zip.sha1
org.apache.maven.wagon.TransferFailedException: transfer failed for https://oss.sonatype.org/content/repositories/snapshots/org/eclipse/birt/birt-runtime/4.9.0-SNAPSHOT/birt-runtime-4.9.0-20220427.100452-1.zip.sha1, status: 502 Bad Gateway
    at org.apache.maven.wagon.providers.http.wagon.shared.AbstractHttpClientWagon.put (AbstractHttpClientWagon.java:835)
    at org.apache.maven.wagon.providers.http.wagon.shared.AbstractHttpClientWagon.put (AbstractHttpClientWagon.java:750)
SteveSchafer-Innovent commented 2 years ago

I saw that everything was deployed to snapshots so I removed "-SNAPSHOT" from all the version numbers and pushed. I did that before noticing wim's last comment and now I see that it's building wim's changes. Is there a way to cancel my build?

wimjongman commented 2 years ago

Why did you remove SNAPSHOT and where did you remove it from?

You can cancel a build by pressing the red icon next to it (you must be logged in to ci)

SteveSchafer-Innovent commented 2 years ago

Oh that explains it. I didn't notice I wasn't logged in.

I changed "4.9.0-SNAPSHOT" to "4.9.0" everywhere. All in pom.xml files. That's what you should do to release to maven central according to this:

https://central.sonatype.org/publish/publish-maven/#nexus-staging-maven-plugin-for-deployment-and-release

SteveSchafer-Innovent commented 2 years ago

If you feel we are not ready to release to maven central, I can revert my changes. Let me know.

wimjongman commented 2 years ago

Don't remove it from the pom.xml:

https://stackoverflow.com/questions/23501119/remove-snapshot-from-project-version-in-pom

We should go ahead and release it.

SteveSchafer-Innovent commented 2 years ago

I reverted the changes and added versions:set -DremoveSnapshot to the maven goals in the jenkins configuration.

SteveSchafer-Innovent commented 2 years ago

That doesn't seem to work. I'll leave this here to see if you have any ideas how to accomplish this other than changing the pom's.

SteveSchafer-Innovent commented 2 years ago

I've been conversing with Frederick Gurr at gitlab.eclipse.org and he says that it has to be manually released the first time and that I should be able to get permission for sschafer@innoventsolutions.com to release org.eclipse.birt. I'm trying to get that permission now.

SteveSchafer-Innovent commented 2 years ago

Okay, I think the step to take now is to build with out "-SNAPSHOT" in the versions. Sorry but this whole process is a bit confusing. You can see my conversation with Frederick Gurr at https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/1077.

@wimjongman, Adding versions:set -DremoveSnapshot to the maven goals didn't work. At this point I don't know what to do other than changing all the pom.xml's. Do you have another suggestion?

wimjongman commented 2 years ago

I think you have to use the maven versions plugin before it will work.

Modifying all poms before a release does not seem right.

However, if it is really needed, we can do it in the build. Use grep or something to mass replace snapshot instead of commiting it.

Best regards,

Wim Jongman

From: Steve Schafer @.> Sent: Thursday, April 28, 2022 8:47:05 PM To: eclipse/birt @.> Cc: Wim Jongman @.>; Mention @.> Subject: Re: [eclipse/birt] Publish Maven Build To Central (#625)

Okay, I think the step to take now is to build with out "-SNAPSHOT" in the versions. Sorry but this whole process is a bit confusing. You can see my conversation with Frederick Gurr at https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/1077.

@wimjongmanhttps://github.com/wimjongman, Adding versions:set -DremoveSnapshot to the maven goals didn't work. At this point I don't know what to do other than changing all the pom.xml's. Do you have another suggestion?

— Reply to this email directly, view it on GitHubhttps://github.com/eclipse/birt/issues/625#issuecomment-1112544930, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AABMF2JSTALP6CZER4EBDH3VHLMKTANCNFSM4ZYBQNVA. You are receiving this because you were mentioned.Message ID: @.***>

SteveSchafer-Innovent commented 2 years ago

I think that using versions:set is using the versions plugin, but now that I look at the documentation I see it's a very complicated plugin and I may not be using it correctly. I'll experiment.

SteveSchafer-Innovent commented 2 years ago

"versions:set -Dremovenapshot" doesn't appear to be working because it's still trying to build everything with "-SNAPSHOT" and it's also getting an error because of not finding dependencies with "-SNAPSHOT", so it's doing something but not the right thing. I may need to try using sed in a script to edit the pom.xml files.

claesrosell commented 2 years ago

Since we are building with Tycho, and the bundle versions and POM.xml versions most much, I think that we need to use the tycho-versions-plugin. I have never used it my self though. Links

It does not seem to have the "remove snapshot", one need to specify the complete version.

Hopefully there are no version dependent Ant scripts that will cause problems.

SteveSchafer-Innovent commented 2 years ago

I added a script to get rid of "-SNAPSHOT" in all the pom.xml files and now I get this error:

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-packaging-plugin:2.5.0:validate-version (default-validate-version) on project org.eclipse.birt: Maven version 4.9.0 must have -SNAPSHOT qualifier for SNAPSHOT builds -> [Help 1]

I get the same thing if I build locally. I wondered how to tell Tycho to do a non-snapshot build but then I read that it's necessary to also remove ".qualifier" in the manifests and also saw @claesrosell 's comment, so I will go back to doing it the other way. I found an article on how to do it with maven and tycho: https://www.lorenzobettini.it/2020/02/remove-snapshot-and-qualifier-in-maven-tycho-builds/

SteveSchafer-Innovent commented 2 years ago

At first I added tycho-versions-plugin:set-version after the clean goal and got errors and also the log indicated that it was still trying to build snapshot versions. So then I moved the tycho-versions-plugin:set-version to a separate build step and now it's definitely building non-snapshot versions and it gets a lot further (build 26), but now it fails with an ant error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.0.0:run (default) on project NLpack1-birt-charts: An Ant BuildException has occured: src '/home/jenkins/agent/workspace/birt_4.9.0_maven/build/org.eclipse.birt.p2updatesite/target/org.eclipse.birt.p2updatesite-4.9.0-SNAPSHOT.zip' doesn't exist.
[ERROR] around Ant part ...<unzip src="..\..\..\org.eclipse.birt.p2updatesite\target\org.eclipse.birt.p2updatesite-4.9.0-SNAPSHOT.zip" dest="/home/jenkins/agent/workspace/birt_4.9.0_maven/build/birt-packages/birt-nl/birt-charts/target/p2updatesite" />... @ 4:229 in /home/jenkins/agent/workspace/birt_4.9.0_maven/build/birt-packages/birt-nl/birt-charts/target/antrun/build-main.xml

I'm investigating this...

Update: Yes indeed "-SNAPSHOT" is hard-coded into a pom.xml file in the maven-antrun-plugin. Looks like I'll need to use sed in addition to tycho set-version.

wimjongman commented 2 years ago

Thanks for your tenacity, Steve.

SteveSchafer-Innovent commented 2 years ago

Looks like we are very close. I think it's all building correctly but now we appear to be violating some publishing rule:

[ERROR] Failed to execute goal org.sonatype.plugins:nexus-staging-maven-plugin:1.6.12:deploy (injected-nexus-deploy) on project org.eclipse.birt.xtab-parent: Remote staging failed: Staging rules failure! -> [Help 1]

I'll try to figure out what rule that is.

SteveSchafer-Innovent commented 2 years ago

Although the build never succeeded, BIRT 4.9.0 appears to be published anyway. At least it appears in sonatype releases. I'll try some experiments to see if it's actually usable.

https://oss.sonatype.org/#view-repositories;releases~browsestorage~org/eclipse/birt

JacquesLeRoux commented 2 years ago

Hi @SteveSchafer-Innovent ,

I must be missing something, I don't see "BIRT 4.9.0 in sonatype releases", at least not when following your link. I wanted to help by trying in OFBiz too.

SteveSchafer-Innovent commented 2 years ago

It appears the link doesn't really work. It just takes you to sonatype but doesn't expand the browser tree to org/eclipse/birt, but you can still manually browse to org/eclipse/birt.

But, in the not-so-great-news department, it appears that what got published is the entirety of birt but not the birt runtime pojo jars. The pojo jars are there, but there inside the birt-runtime zip which is useless from a maven standpoint. I believe one could still utilize the repository for the runtime but it would probably need to be an osgi configuration rather than a pojo configuration. In order to have the pojo jars be mavenized I think we are going to create a new project and publish it to org.eclipse.birt.runtime, as has evidently been done in the past, up to version 4.6.0 and as I did a couple of years ago as org.innoventsolutions.birt.runtime.

JacquesLeRoux commented 2 years ago

Thanks for the info, looking forward :)