Problems posting maven artifacts to maven central

[planetf1]

Whilst our release 'shipped' on github 10 days ago (c. 17 Jan), and the release pipeline completed to push artifacts to jfrog artifactory & then bintray - from where they should be published to maven central, in fact the process hit some issues that we had no knowledge of.

Maven central publishing can take a few days for indexes to be rebuilt.

I raised IT issue https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-18759?sda_source=notification-email 7 days ago ("IT-18759 ODPi Egeria - release completed in Azure. Not on maven central") and have had the following update from the support team:

The LF team are investigating updating the necessary pipelines to help with some of this:

Some further notes at: https://confluence.linuxfoundation.org/pages/viewpage.action?pageId=41582704

In the meantime the artifacts were handled manually with the following issues arising:

All the 1.3 packages have been synced to Maven-Central with the exception of the following:

New Packages Have been Requested to be included in JCenter and are waiting on approval from JFrog:

audit-log-event-topic-connector
audit-log-slf4j-connector
cassandra-data-store-connector
cassandra-metadata-extractor-connector
data-platform-services-connector
discovery-service-connectors
gaf-metadata-client

open-connector-archives
Packages Missing JavaDoc:

open-types-test-generator
Packages Missing Sources:

governance-action-connectors
stewardship-action-connectors
Packages with Invalid POM file:

gaf-metadata-management
ui-chassis
Packages with Corrupt JAR:

open-lineage-janus-connector
ui-chassis-spring
I'll attempt to redistribute open-lineage-janus-connector and ui-chassis-spring from Artifactory to see if that resolves those package issues. Once we get confirmation from JFrog on the requested packages for JCenter we'll try syncing those out as well.```

[planetf1]

I should have access to jcenter to help in future (if anyone else needs this add a note here - @cmgrote )

• We need to figure out the process around adding a package. Can we detect during the normal dev process, or is it left to final validation?
• We do try and always build javadoc - though there is a discrepancy above? Where did this go wrong
• We also add sources - how did this fail?
• What is an invalid pom when we don't see build issues
• Can we automatically fix corruption by re-posting?? And was this error transient or permanent?

As we formulate answers to these questions I'll raise any necessary PRs for quick fixes or issues for longer discussions,

cc: @mandy-chessell @cmgrote @bramwelt

[planetf1]

@bramwelt I shared my bintray id. Hopefully I will be able to see more details on the exact validation failures that may require code/build changes to fix

[planetf1]

Note https://github.com/odpi/egeria/blob/master/developer-resources/Release-Process.md should be updated with any changes/clarification to the release process. Related docs may need updating with reference to new dependencies or build requirements.

[planetf1]

Processing for syncing from bintray to maven central -> https://confluence.linuxfoundation.org/pages/viewpage.action?pageId=41582704

• this is the initial way to see errors have occurred with a specific package

REST API for single package -> https://www.jfrog.com/confluence/display/BT/Bintray+REST+API#BintrayRESTAPI-SyncVersionArtifactstoMavenCentral

Adding a new artifact -> https://www.jfrog.com/confluence/display/BT/Central+Repositories#CentralRepositories-IncludingyourPackageinJCenter

Given this info will look at fixing up the remaining issues.

[planetf1]

Outstanding packages with issues:

  egeria-connector-apache-atlas
  egeria-connector-apache-atlas-adapter
  egeria-connector-apache-atlas-package
  gaf-metadata-management
  governance-action-connectors
  open-lineage-janus-connector
  open-types-test-generator
  stewardship-action-connectors
  ui-chassis
  ui-chassis-spring

[planetf1]

Still looking at this.. Taking the ui-chassis-spring the artifact on bintray looks as expected - it has a regular jar file - containing classes, jars etc; a sources jar (java source code); javadoc (html). No problem.

It isn't synced to maven central as it needs to be added to jcenter [a manual step unfortunately for each artifact added in the release]. (route is bintray->jcenter->maven central), but this fails with:

JAR file is corrupted.
A binary file (jar, aar, wat, apk) should be part of the package.
Could not validate jar file.
Package should include sources as part of the package.

None of these are true. Trying to chase up with LF/jfrog

[planetf1]

Other failures

• ui-chassis - A binary file (jar, aar, wat, apk) should be part of the package. Package should include sources as part of the package.
  • stewardship-action-connectors - Package should include sources as part of the package.
  • gaf-metadata-management - A binary file (jar, aar, wat, apk) should be part of the package. Package should include sources as part of the package.
• governance-action-connectors - Package should include sources as part of the package.
• open-lineage-janus-connector - JAR file is corrupted. A binary file (jar, aar, wat, apk) should be part of the package. Could not validate jar file. Package should include sources as part of the package.

• open-types-test-generator - An odd one here -- according to bintray this IS in jcenter - however the 'sync status' states 'Missing: no javadoc jar found in folder '/org/odpi/egeria/open-types-test-generator/1.3' Dropping existing partial staging repository.'

This last one is a correct error - the artifact doesn't have javadoc. Javadoc hit issues with the nature of this code. A fix will be needed in egeria to either build something or add a zero length file. There's no real impact though to this artifact being missing as it's unlikely anyone would use it for integration

[planetf1]

It looks like governance-action-connectors, gaf-metadata-management & stewardship-action-connectors are pom projects only. Of these two are correct. However 'governance-action-connectors' is missing the `pom' - whilst the default is jar.

[planetf1]

Posting question on SO https://stackoverflow.com/questions/59984421/jar-file-is-corrupted-missing-source-javadoc-pushing-maven-artifact-from-bin

Also contacted support@bintray.com & @bintray

[planetf1]

Had a reply from jfrog

Thank you for reaching JFrog Support. We have received your request and actively working on it. As of now we see you have the files on bintry at this location. The files seems to be fine at quick scan on the files. However we are checking further with complete analysis and why JCenter is throwing the invalid JAR error for the files. >

.. following up ....

[planetf1]

A further update from jfrog

Thank you for your patience. We found the cause of the issue. There are some timeouts happening unexpectedly while validating the jar files. Our team is investigating further and working toward the fix. We will get back to you at the earliest.

I suspect it is to do with the fact some of our jar files are non trivial - larger in size than many... yet again we push the boundaries and break other technology!

[planetf1]

We have increased the timeout values during the file processing. Kindly check if you are still seeing the issue or the jars validation is passing.

... the janus connector and ui-chassis-spring now pass validation

the open types generator passes, but should fail (and does at maven central) due to missing javadoc (we also need to fix this)

The other artifacts above still fail with 'POM project file is not valid'

Have updated jfrog with findings..

[bramwelt]

@planetf1 Thanks for the updates!

As a note I removed the apache-atlas and other connector packages from https://bintray.com/odpi/egeria as they were moved over (and redistributed from Artifactory) to https://bintray.com/odpi/egeria-connector-apache-atlas and https://bintray.com/odpi/egeria-connector-ibm-information-server respectively. Though not clearly documented, JCenter inclusion requires the packages come from the VCS repository linked to the Bintray repo.

I believe there are similar issues (missing Javadoc/Sources) with those packages that @cmgrote is working on.

[cmgrote]

I believe there are similar issues (missing Javadoc/Sources) with those packages that @cmgrote is working on.

Yes, there were some gaps, but I cleaned them up in the connectors and everything there is now synced 😄

[planetf1]

@bramwellt Thanks for the change ref connector. Good to hear they are sorted.

On the core egeria artifacts - for two of them they were actually correct! :-) but JCenter validation was wrong....

We have one artifact which is clearly wrong - our fault. Funnily enough this passed jcenter validation (wrong) but failed maven validation (correct), whilst the remaining issues are with pom only projects which are reporting an invalid pom (yes the checkbox is ticked). Of course maven doesn't report any such problem, and the error message doesn't clarify what the error is. I've tried to compare but not yet noticed a problem. I've gone back to jfrog..

[planetf1]

No response to the remaining issues on fri/w'end. Have chased up.

[planetf1]

Further response from jfrog

I have worked with our internal team and after analysis we could say that the behaviour is expected. Let me elaborate further on this. 

1. ui-chassis When a Project is marked as pom project, then the pom file should have a reference to any external sources like github for the binaries. In the provided pom file, there is no reference to external sources. And is why JCenter validation is failing and showing the validation message "A binary file (jar, aar, wat, apk) should be part of the package. Package should include sources as part of the package."

2. stewardship-action-connectors This package does not contain sources.jar file. A premium customer has the privilege to link the package to JCenter without sources.jar. In your case, we request you to add sources.jar file. Hence is it showing the validation message "Package should include sources as part of the package"

3. gaf-metadata-management The issue is same as 1. The pom file does not contain a reference to external sources. 

4. governance-action-connectors This is also same as 2. There is no sources.jar file and hence the validation is failing. 

We validate that the pom include section like mentioned here: http://maven.apache.org/scm/maven-scm-plugin/usage.html. I hope this clarifies your query. Let us know if you have further queries. We look forward to your response.

2,4 are explained by the far they originally had no section, thus defaulting to jar. And jars must have source. In fact they should be the same as 1,3 - as without any code we'd expect packaging=pom - and thus hit the first error.

JFrog appear to be saying that if there's no code included they at least need a code reference, seeming to point to saying they mandate the section. We have this in our top level pom, but no others.

Whilst some of the packaging=pom modules will change to packaging=jar once code is included, the fact remains that some packages such as 'user-interfaces' are only umbrella packages for a series of modules

It seems as if we now need to at least try some changes to make JCenter work, but ideally do also what is appropriate

The options appear to be

• Make all poms one of the non-pom types

Whilst this is correct in some cases - ie placeholders where we haven't added the code yet, it's never going to be true across all of our poms. It would also require adding a dummy empty jar or similar (like we already for for sources/javadoc)

• Try adding SCM references to all poms - at least for packaging=pom

This seems low risk/harmless though adds clutter.

I'm going to try adding the scm ref to all failing poms for 1.3, and resubmit the release. I will then share the results with jfrog as I am unhappy about their level of documentation, and the 'correctness' of the validation. In addition it's unclear if scm is the cause, since other packaging=pom artifacts are already on JCenter without scm tags. But there seems little choice to try complying for the next test

If the test passes I'll patch for 1.4, and make a change FOR ALL poms for master

[planetf1]

@mandy-chessell Once built I'd like to ask we merge this -- I need to redo the release cycle & get the new packages through to the validation step in artifactory/JCenter.

[planetf1]

Merged fixes so far and started a new release:

However this is failing

https://dev.azure.com/ODPi/Egeria/_releaseProgress?_a=release-environment-logs&releaseId=110&environmentId=322

2020-02-11T13:56:29.3182944Z ##[section]Starting: Artifactory Build Promotion - Dry Run
2020-02-11T13:56:29.3505901Z ==============================================================================
2020-02-11T13:56:29.3506226Z Task         : Artifactory Build Promotion
2020-02-11T13:56:29.3506364Z Description  : Promote a published build in Artifactory.
2020-02-11T13:56:29.3506527Z Version      : 1.8.1
2020-02-11T13:56:29.3506676Z Author       : JFrog
2020-02-11T13:56:29.3506778Z Help         : 
2020-02-11T13:56:29.3506941Z ==============================================================================
2020-02-11T13:56:29.9416182Z Downloading: https://api.bintray.com/content/jfrog/jfrog-cli-go/1.31.2/jfrog-cli-linux-amd64/jfrog?bt_package=jfrog-cli-linux-amd64
2020-02-11T13:56:31.4120606Z Caching tool: jfrog 1.31.2 x64
2020-02-11T13:56:31.4346997Z Running jfrog-cli from /opt/hostedtoolcache/jfrog/1.31.2/x64/jfrog.
2020-02-11T13:56:31.4477650Z JFrog CLI version: 1.31.2
2020-02-11T13:56:31.5174277Z [Info] [Dry run] Promoting build...
2020-02-11T13:56:34.2743402Z [Error] Artifactory response: 400 Bad Request
2020-02-11T13:56:34.2744175Z {
2020-02-11T13:56:34.2745024Z   "messages": [
2020-02-11T13:56:34.2745289Z     {
2020-02-11T13:56:34.2745455Z       "level": "ERROR",
2020-02-11T13:56:34.2746497Z       "message": "User doesn't have permissions to override 'egeria-release-local:org/odpi/egeria/repository-services/1.3'. Needs delete permissions."
2020-02-11T13:56:34.2746763Z     },
2020-02-11T13:56:34.2746922Z     {
2020-02-11T13:56:34.2747029Z       "level": "INFO",
2020-02-11T13:56:34.2747231Z       "message": "Skipping promotion status update: item promotion was completed with errors and warnings."
2020-02-11T13:56:34.2747426Z     }
2020-02-11T13:56:34.2747571Z   ]
2020-02-11T13:56:34.2747671Z }
2020-02-11T13:56:34.2841513Z ##[error]Error: Command failed: /opt/hostedtoolcache/jfrog/1.31.2/x64/jfrog rt bpr "Egeria-Stage" "bd8ee26f1" "egeria-release-local" --url="https://odpi.jfrog.io/odpi" --user=*** --password=*** --include-dependencies=false --copy=false --dry-run=true
2020-02-11T13:56:34.2928319Z ##[section]Finishing: Artifactory Build Promotion - Dry Run

This may mean the artifacts need manually deleting.

Can you help @bramwelt ?

[bramwelt]

Yeah artifacts would need to be deleted in this case. I'm not sure if that's the right approach as it goes against the 'never delete a release' process we have in place. I'm going to talk this over with the team and get back to you shortly.

[planetf1]

I think ideally we wouldn't want to -- correctly... and if there was a new release it would be reversioned.

We have a particular case here where we're still debugging the process... not sure how else to do this other than rerunning the release... I'm hoping the latest notes on the pom structure will mean we'll be in a better state for next time (says each release....)

We have 1.4 queued also....

[bramwelt]

Right. Working on re-running the 1.3 release now.

[bramwelt]

gaf-metadata-management
governance-action-connectors
stewardship-action-connectors
ui-chassis

These packages have been requested to be included in JCenter after I successfully re-ran the 1.3 release. One we hear back from JCenter, we should be able to trigger the Maven-Sync step again and get them sent out.

open-types-test-generator

Appears to still be missing javadoc - Though it sounds like this can be followed up for the 1.4 release, per the note about it not being used in integration?

[planetf1]

Thanks. It seems as if those packages are on jcenter, so I've kicked off the last stage again.

open-types-test-generator we can skip for 1.3

I'm going to make the same changes to 1.4 & master including fixing this last module (I thought I did - but my artifact indeed has missing javadoc) -- and will likely try 1.4 also. (bearing in mind to recheck/rerun after as needed).

Thanks

[planetf1]

• Verified completion of sync job -- only types gen failed
• cherry-picked changes from 1.3 forward to 1.4
• fixed javadoc for last package (test gen) in 1.4

[planetf1]

MASTER

For the code in master I've gone for a slightly different change rather than cherry pick the prior changes

We know:

• JFrog requires this tag in all pom projects.
• Jfrog doesn't entirely seem to base it's parsing on the packaging type
• we have some poms that whilst currently empty, are of type jar - and will be added to later. (for example gaf, stewardship)
• we need to clearly demonstrate the pattern of what should be in all poms for safety.
• more generally the docs./errors are not that clear, but they have been specific about

Therefore for master I propose adding the tag to EVERY pom. At worst it's unnecessary/incomplete - can't be 100% sure until the next cycle It also doesn't seem unreasonable to add info to every artifact that points back to our git repo. The main risk is that we will still need packaging=pom in a few incomplete projects

I have also corrected javadoc for the open types test gen project When we release 1.5 I will review the results

PR shortly with these proposed changes.

[bramwelt]

@planetf1 Glad to see we're down to one!

[planetf1]

1.4 is currently syncing.

[planetf1]

The 1.4 sync job failed after 5 1/2 hours - perhaps a timeout https://dev.azure.com/ODPi/Egeria/_releaseProgress?_a=release-environment-logs&releaseId=111&environmentId=332

The sync was progressing at around 1.5 minutes per artifact - given we have 200 or so this would be very slow

I was able to restart the pipeline, and the sync appears to be continuing from where it left off

[planetf1]

The 'sync missing packages to maven central' has done a further 8 artifacts in 1hr 53 min.

It seems to run the sync for every artifact in a release - is that needed on jcenter? or just for artifacts not existing at all (ie no previous releases)? @bramwelt

[planetf1]

Release 1.4 has now completed pushing after being restarted. I've manually cross-checked and all our artifacts appear to be on maven central.

@bramwelt the only remaining question is whether we can increase timeout or speed up (less likely) the sync.

moving milestone to 1.5 for now, pior to closure

[planetf1]

I checked the release pipeline and there is already a 0 timeout specified.

I'm going to close this issue for now, and will revisit when we release 1.5 (towards the end of the month)