Open pipebaum opened 3 years ago
Hi @SteveSchafer-Innovent,
We are ready to create a new release of Apache OFBiz. We just miss this new publish on Maven to fix a CVE due to Birt used as a plugin. We can avoid to use the Birt plugin for this release but would, of course, prefer not. How many time do you think it will need to publish on Maven?
TIA
I'm having problems getting the jenkins job to build. I created the branch BIRT_4.9.0_maven and created a jenkins job that is basically the same as birt-master except it refers to that branch, specifies the gpg keyring file, and invokes the release profile that I added to the pom.xml file. It fails with this error:
[ERROR] Failed to execute goal org.eclipse.tycho:tycho-p2-plugin:2.5.0:p2-metadata (attached-p2-metadata) on project org.apache.batik.dom.birt.ext: Execution attached-p2-metadata of goal org.eclipse.tycho:tycho-p2-plugin:2.5.0:p2-metadata failed. IllegalArgumentException
which seems strange because org.apache.batik.dom.birt.ext doesn't invoke the tycho-p2-plugin plugin. I get this error whether I invoke the build-server profile or not.
More info:
If I remove the "release" profile it should, theoretically, build as it originally did, however instead it gets a test failure:
Tests run: 241, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 46.863 s <<< FAILURE! - in org.eclipse.birt.chart.tests.AllTests 2dline(org.eclipse.birt.chart.tests.device.render.ImageOutputBaseTest) Time elapsed: 5.815 s <<< FAILURE! junit.framework.AssertionFailedError at org.eclipse.birt.chart.tests.device.render.ImageOutputBaseTest.runTest(ImageOutputBaseTest.java:113)
Perhaps this is related to a change I had to make to org.eclipse.birt.target.target. The repository https://download.eclipse.org/eclipse.updates/4.23-I-builds no longer exists, so I changed it to 4.23.
I'm getting 504 errors on ci.eclipse.org/birt. I think I broke it.
haha. Well done. It works fine for me: https://ci.eclipse.org/birt/
It's working for me now too. I deleted a project and then saw that it wasn't deleted but just disabled. When I tried to re-enable it, it went out to lunch. I'm going to try it again. Update: deleting the project worked this time.
Can I push a change to BIRT_4.9.0 branch to change https://download.eclipse.org/eclipse/updates/4.23-I-builds to https://download.eclipse.org/eclipse/updates/4.23 in org.eclipse.birt.target.target because the former no longer exists and is keeping it from building in jenkins?
Yes, that is fine. It should be that before we released it.
I actually got the birt_4.9.0_maven branch to build by re-creating the project and removing the "release" profile. When I added the release profile back, it got the tycho-p2-plugin error I mentioned earlier. It's a clue! But what does it mean?
I see that it complains about a source that is already attached. In your profile I see this line:
Maybe this is not needed.
The batik project contains the jar-signer in its own pom.xml https://github.com/eclipse/birt/tree/BIRT_4.9.0_maven/common/org.apache.batik.dom.birt.ext
Do you think that this can cause it?
Removing the jarsigner from org.apache.batik.dom.birt.ext didn't fix the error.
@wimjongman thanks for adding the tracebacks to the jenkins log. I see the illegal argument exception is coming from P2GeneratorImpl.getCanonicalArtifact and you can see the code here, but it doesn't mean anything to me at this point. I will try digging further.
I am new to BIRT and I also don't know much about tycho and eclipse plugins. But I see one differeence in the poms between the two projects that work (java.xml.rpc
, org.apache.axis
) and org.apache.batik.dom.birt
:
There is no groupid in the working projects.
I don't know if this is of any help, but maybe you should give it a try as the groupid is already in the parent pom and is redundant.
The build.properties
look also pretty different.
Mayby the trailing \
and following newlines produce the error.
Since there is no hint on the internet on this IllegalArgumentException
my experience tells me this is no general issue, but more like a uncommon input, to cause this.
Thanks, Alex! Those kinds of things could just be it.
I removed the groupId from org.apache.batik.dom.birt.ext/pom.xml and changed build.properties a bit but it still got the error. I don't think there's anything wrong with build.properties. On the next line after the trailing backslash there's a dot so I think that's not an error. Anyway, more importantly, the contents of org.apache.batik.dom.birt.ext are identical to the 4.10 version so it seems unlikely that anything within that project is causing the error. Maybe something about p2 configuration?
I am not very good with Maven or Tycho, but I have tinkered with it a little in the past.
I think that the:
[ERROR] Failed to execute goal org.eclipse.tycho:tycho-p2-plugin:2.5.0:p2-metadata (attached-p2-metadata) on project org.apache.batik.dom.birt.ext: Execution attached-p2-metadata of goal org.eclipse.tycho:tycho-p2-plugin:2.5.0:p2-metadata failed. IllegalArgumentException -> [Help 1]
error happens because of missing metadata in org.apache.batik.dom.birt.ext-1.14.0-SNAPSHOT-sources.jar
this in turn happens because we, since the introduction of the "maven-source-plugin" in our parent pom.xml, generate the source bundles twice.
First with we generate the sources with "tycho-source-plugin" and then that generated source bundle is overwritten when we generate the sources again with the "maven-source-plugin".
What happens if we remove the "maven-source-plugin" from the release profile? I think that we can remove the "maven-javadoc-plugin" as well since we generate the javadocs elsewhere (with Ant).
Maybe it is Tycho and we try bumping the Tycho version to 2.7.1
When I remove maven-source-plugin and maven-javadoc-plugin, it gets this error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.0.0:run (javadoc) on project org.eclipse.birt.doc.isv: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] /home/jenkins/agent/workspace/birt_4.9.0_maven/docs/org.eclipse.birt.doc.isv/BuildISVDoc.xml:201: Replace: source file /home/jenkins/agent/workspace/birt_4.9.0_maven/docs/org.eclipse.birt.doc.isv/engine/api/index.html doesn't exist
[ERROR] around Ant part ...
I will try Wim's suggestion also.
BuildISVDoc.xml is an ant file that depends on the javadoc being generated by the maven-javadoc-plugin.
@SteveSchafer-Innovent, this is actually progress. This is the same problem that, for unknown reasons, happend in master a couple of weeks ago. And was temporarily fixed with this bug: #917. @wimjongman : It is my understanding that the javadocs that BuildISVDoc.xml is dependent on is generated in BuildISVDoc.xml itself, it is not dependent on : maven-javadoc-plugin.
@claesrosell that could be, I just derived from what I saw. Apologies for the confusion.
I copied the changes from pr 920. Now we have this:
[ERROR] Failed to execute goal org.eclipse.tycho:tycho-packaging-plugin:2.7.1:package-feature (default-package-feature) on project org.eclipse.birt.chart: Execution default-package-feature of goal org.eclipse.tycho:tycho-packaging-plugin:2.7.1:package-feature failed: Duplicate key org.eclipse.swt (attempted merging values org.eclipse.tycho.model.Feature$ImportRef@d05993c2 and org.eclipse.tycho.model.Feature$ImportRef@387574f4) -> [Help 1]
Note: we are currently using Tycho 2.7.1. I should probably switch it back to 2.5.0 to eliminate that difference from 4.10.
Amazing! After I switched back to Tycho 2.5.0 the build worked. Now to see what happened on Maven Central...
I'm not seeing org.eclipse.birt on sonatype. I've left a question on the github issue.
From what I can see in the build log (https://ci.eclipse.org/birt/job/birt_4.9.0_maven/16/) we are still calling:
mvn clean verify
I think that one need to call
mvn clean deploy
To deploy to a external repository. I might be wrong though, I have never done this myself.
Yes, that is correct. Good catch.
I changed that and started the build again.
We are getting very close:
[INFO] Deploying remotely...
[INFO] Bulk deploying locally gathered artifacts from directory:
[INFO] * Bulk deploying locally gathered snapshot artifacts
[WARNING] Failed to upload checksum to org/eclipse/birt/birt-runtime/4.9.0-SNAPSHOT/birt-runtime-4.9.0-20220427.100452-1.zip.sha1
org.apache.maven.wagon.TransferFailedException: transfer failed for https://oss.sonatype.org/content/repositories/snapshots/org/eclipse/birt/birt-runtime/4.9.0-SNAPSHOT/birt-runtime-4.9.0-20220427.100452-1.zip.sha1, status: 502 Bad Gateway
at org.apache.maven.wagon.providers.http.wagon.shared.AbstractHttpClientWagon.put (AbstractHttpClientWagon.java:835)
at org.apache.maven.wagon.providers.http.wagon.shared.AbstractHttpClientWagon.put (AbstractHttpClientWagon.java:750)
I saw that everything was deployed to snapshots so I removed "-SNAPSHOT" from all the version numbers and pushed. I did that before noticing wim's last comment and now I see that it's building wim's changes. Is there a way to cancel my build?
Why did you remove SNAPSHOT and where did you remove it from?
You can cancel a build by pressing the red icon next to it (you must be logged in to ci)
Oh that explains it. I didn't notice I wasn't logged in.
I changed "4.9.0-SNAPSHOT" to "4.9.0" everywhere. All in pom.xml files. That's what you should do to release to maven central according to this:
If you feel we are not ready to release to maven central, I can revert my changes. Let me know.
Don't remove it from the pom.xml:
https://stackoverflow.com/questions/23501119/remove-snapshot-from-project-version-in-pom
We should go ahead and release it.
I reverted the changes and added versions:set -DremoveSnapshot to the maven goals in the jenkins configuration.
That doesn't seem to work. I'll leave this here to see if you have any ideas how to accomplish this other than changing the pom's.
I've been conversing with Frederick Gurr at gitlab.eclipse.org and he says that it has to be manually released the first time and that I should be able to get permission for sschafer@innoventsolutions.com to release org.eclipse.birt. I'm trying to get that permission now.
Okay, I think the step to take now is to build with out "-SNAPSHOT" in the versions. Sorry but this whole process is a bit confusing. You can see my conversation with Frederick Gurr at https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/1077.
@wimjongman, Adding versions:set -DremoveSnapshot to the maven goals didn't work. At this point I don't know what to do other than changing all the pom.xml's. Do you have another suggestion?
I think you have to use the maven versions plugin before it will work.
Modifying all poms before a release does not seem right.
However, if it is really needed, we can do it in the build. Use grep or something to mass replace snapshot instead of commiting it.
Best regards,
Wim Jongman
From: Steve Schafer @.> Sent: Thursday, April 28, 2022 8:47:05 PM To: eclipse/birt @.> Cc: Wim Jongman @.>; Mention @.> Subject: Re: [eclipse/birt] Publish Maven Build To Central (#625)
Okay, I think the step to take now is to build with out "-SNAPSHOT" in the versions. Sorry but this whole process is a bit confusing. You can see my conversation with Frederick Gurr at https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/1077.
@wimjongmanhttps://github.com/wimjongman, Adding versions:set -DremoveSnapshot to the maven goals didn't work. At this point I don't know what to do other than changing all the pom.xml's. Do you have another suggestion?
— Reply to this email directly, view it on GitHubhttps://github.com/eclipse/birt/issues/625#issuecomment-1112544930, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AABMF2JSTALP6CZER4EBDH3VHLMKTANCNFSM4ZYBQNVA. You are receiving this because you were mentioned.Message ID: @.***>
I think that using versions:set is using the versions plugin, but now that I look at the documentation I see it's a very complicated plugin and I may not be using it correctly. I'll experiment.
"versions:set -Dremovenapshot" doesn't appear to be working because it's still trying to build everything with "-SNAPSHOT" and it's also getting an error because of not finding dependencies with "-SNAPSHOT", so it's doing something but not the right thing. I may need to try using sed in a script to edit the pom.xml files.
Since we are building with Tycho, and the bundle versions and POM.xml versions most much, I think that we need to use the tycho-versions-plugin. I have never used it my self though. Links
It does not seem to have the "remove snapshot", one need to specify the complete version.
Hopefully there are no version dependent Ant scripts that will cause problems.
I added a script to get rid of "-SNAPSHOT" in all the pom.xml files and now I get this error:
[ERROR] Failed to execute goal org.eclipse.tycho:tycho-packaging-plugin:2.5.0:validate-version (default-validate-version) on project org.eclipse.birt: Maven version 4.9.0 must have -SNAPSHOT qualifier for SNAPSHOT builds -> [Help 1]
I get the same thing if I build locally. I wondered how to tell Tycho to do a non-snapshot build but then I read that it's necessary to also remove ".qualifier" in the manifests and also saw @claesrosell 's comment, so I will go back to doing it the other way. I found an article on how to do it with maven and tycho: https://www.lorenzobettini.it/2020/02/remove-snapshot-and-qualifier-in-maven-tycho-builds/
At first I added tycho-versions-plugin:set-version after the clean goal and got errors and also the log indicated that it was still trying to build snapshot versions. So then I moved the tycho-versions-plugin:set-version to a separate build step and now it's definitely building non-snapshot versions and it gets a lot further (build 26), but now it fails with an ant error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.0.0:run (default) on project NLpack1-birt-charts: An Ant BuildException has occured: src '/home/jenkins/agent/workspace/birt_4.9.0_maven/build/org.eclipse.birt.p2updatesite/target/org.eclipse.birt.p2updatesite-4.9.0-SNAPSHOT.zip' doesn't exist.
[ERROR] around Ant part ...<unzip src="..\..\..\org.eclipse.birt.p2updatesite\target\org.eclipse.birt.p2updatesite-4.9.0-SNAPSHOT.zip" dest="/home/jenkins/agent/workspace/birt_4.9.0_maven/build/birt-packages/birt-nl/birt-charts/target/p2updatesite" />... @ 4:229 in /home/jenkins/agent/workspace/birt_4.9.0_maven/build/birt-packages/birt-nl/birt-charts/target/antrun/build-main.xml
I'm investigating this...
Update: Yes indeed "-SNAPSHOT" is hard-coded into a pom.xml file in the maven-antrun-plugin. Looks like I'll need to use sed in addition to tycho set-version.
Thanks for your tenacity, Steve.
Looks like we are very close. I think it's all building correctly but now we appear to be violating some publishing rule:
[ERROR] Failed to execute goal org.sonatype.plugins:nexus-staging-maven-plugin:1.6.12:deploy (injected-nexus-deploy) on project org.eclipse.birt.xtab-parent: Remote staging failed: Staging rules failure! -> [Help 1]
I'll try to figure out what rule that is.
Although the build never succeeded, BIRT 4.9.0 appears to be published anyway. At least it appears in sonatype releases. I'll try some experiments to see if it's actually usable.
https://oss.sonatype.org/#view-repositories;releases~browsestorage~org/eclipse/birt
Hi @SteveSchafer-Innovent ,
I must be missing something, I don't see "BIRT 4.9.0 in sonatype releases", at least not when following your link. I wanted to help by trying in OFBiz too.
It appears the link doesn't really work. It just takes you to sonatype but doesn't expand the browser tree to org/eclipse/birt, but you can still manually browse to org/eclipse/birt.
But, in the not-so-great-news department, it appears that what got published is the entirety of birt but not the birt runtime pojo jars. The pojo jars are there, but there inside the birt-runtime zip which is useless from a maven standpoint. I believe one could still utilize the repository for the runtime but it would probably need to be an osgi configuration rather than a pojo configuration. In order to have the pojo jars be mavenized I think we are going to create a new project and publish it to org.eclipse.birt.runtime, as has evidently been done in the past, up to version 4.6.0 and as I did a couple of years ago as org.innoventsolutions.birt.runtime.
Thanks for the info, looking forward :)
Currently, there is no automated feature to publish the BIRT-Runtime to Maven. Wouldn't it be nice to fix this.