search

opensearch-project / opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems
Apache License 2.0
136 stars 271 forks source link

[release]: JDBC driver shadow jar v.1.3.0.0 #3614

Closed Yury-Fridlyand closed 1 year ago

Yury-Fridlyand commented 1 year ago

Did you read the on-boarding document

-

What is the name of your component?

JDBC driver

What is the link to your GitHub repo?

https://github.com/opensearch-project/sql-jdbc

Targeted release date

-

Where should we publish this component?

artifacts: https://artifacts.opensearch.org/ download page: https://opensearch.org/downloads.html#drivers

What type of artifact(s) will be generated for this component?

Shadow jar

Have you completed the required reviews including security reviews, UX reviews?

+

Have you on-boarded automated security scanning for the GitHub repo associated with this component?

+

Additional context

sql-jdbc.zip JDBC driver v.1.3.0.0 was released on maven, but we should also publish shadow jar of this version on the main website. See attached artifact for this version. This jar should be signed prior to publish.

Yury-Fridlyand commented 1 year ago

This blocks Tableau connector release. @gaiksaya, could you have a look please?

gaiksaya commented 1 year ago

Hi @Yury-Fridlyand ,

Few questions:

  1. The zip only needs to be published on website or also on maven?
  2. In order to sign and publish the artifact anywhere we would automate the artifact creation, signing and uploading. Hence, wondering if this can be included in 1-click release process of sql-jdbc release a well?
  3. Is this one time process or there are further releases planned for this?
Yury-Fridlyand commented 1 year ago
  1. Shadow jar goes to website only. Note: a jar should be published, not a zip. GH doesn't allow to upload jar files. *
  2. Yes, it would be awesome.

  3. No, we should do this with every release of JDBC.

    Non shadow jar is already on maven and it is ok, we should publish both of them.

gaiksaya commented 1 year ago

Thanks! Only 1.13.0.0 jar needs to be published to website but going forward both jars can be published to maven as well as website if I am not wrong?

The link on the website jar is still pointing to opensearch-sql-jdbc-1.1.0.1.jar which is very old. Is someone following up with website team on this?

We would need to make changes to add the shadow publication jar to this tarball https://github.com/opensearch-project/sql-jdbc/blob/main/.github/workflows/release-drafter.yml#L24 Signing and publishing to maven is automatically taken care of by jenkins file

However, we need to add publishToArtifactsProdBucket to jenkins file that will take care of uploading the artifacts to S3 bucket/website automatically.

Tagging @rishabh6788 who would be taking care of this.

Thanks!

Yury-Fridlyand commented 1 year ago

going forward both jars can be published to maven as well

shadow jar goes to website (to artifacts actually) non shadow jar goes to maven (as it works now)

I think I can trigger updating link on the website on my own, once it is uploaded to artifacts.

Thanks @gaiksaya and @rishabh6788!

rishabh6788 commented 1 year ago

@Yury-Fridlyand Could you please share the command to generate shadow jars and also does it generate the artifacts in the repository directory or is it on a different path?

Yury-Fridlyand commented 1 year ago

./gradlew shadowJar generates opensearch-sql-jdbc-1.3.0.0.jar in build/libs.

rishabh6788 commented 1 year ago

I tried the above mentioned command and it generated opensearch-sql-jdbc-1.4.0.0.jar jar in build/libs dir. I then ran ./gradlew publishPublishMavenPublicationToLocalRepoRepository and it also generated the new jar with same name and overwrote the one generated using shadowJar command.

  1. Can we rename the shadow jar to something like opensearch-sql-jdbc-1.4.0.0-shadow.jar to avoid the confusion of having two jars at different locations with the same name?
  2. Is the behavior wrt to publishPublishMavenPublicationToLocalRepoRepository overwriting the jar created using shadowJar expected?
Yury-Fridlyand commented 1 year ago

@rishabh6788 you probably checked out main, you need to check out 1.3.0.0 release. Answering your questions:

  1. I think yes
  2. Yes, both commands create an artifact

There is no need to do publishPublishMavenPublicationToLocalRepoRepository for getting a shadow jar.

Yury-Fridlyand commented 1 year ago

This is required for #3631

gaiksaya commented 1 year ago

There is no need to do publishPublishMavenPublicationToLocalRepoRepository for getting a shadow jar.

Hey @Yury-Fridlyand 1.13.0.0 apart we are trying to include this in 1-click release process which sql-jdbc is already onboarded too. So the idea is to include it in the same workflow and just bundle both (shadow, non-shadow jars) into artifacts.tar.gz to be supplied to jenkins workflow. Here to be precise: https://github.com/opensearch-project/sql-jdbc/blob/main/.github/workflows/release-drafter.yml#L24 So from next release, both will be released simultaneously.

rishabh6788 commented 1 year ago

@Yury-Fridlyand If I remember correctly tableau needs the artifacts to be signed using JarSigner with timestamp if you are planning to upload to Tableau marketplace. What we are using is a PGP signer to upload the signed jar to artifacts bucket. Can you please confirm the signing requirement?

Yury-Fridlyand commented 1 year ago

Oh, that is very good point. Right, Tableau require JarSigner's signature. Is is possible to apply two signatures and have them valid on a single jar? If not, we have to publish 2 shadow jars - signed by JarSigner and by PGP signer. Both should be available for public to download.

rishabh6788 commented 1 year ago

I believe even the jar signed by JarSigner can be used by public, both forms signing represent that it comes from a valid source and there is no change in the functionality.

rishabh6788 commented 1 year ago

Since the already uploaded Jar on opensearch.org has been signed using PGP, in order to not break the signing method mentioned on the website we will have to sign it using the existing PGP method and also upload a JarSigner signed copy as well for Tableau. @gaiksaya @bbarani Let me know your thoughts.

Yury-Fridlyand commented 1 year ago

Great I also confirm that jar file name could be renamed for Tableau submission, if is needed.

As I see total JDBC release process it would like this:

./gradlew publishPublishMavenPublicationToLocalRepoRepository
# sign
...
# upload to maven
...

./gradlew shadowJar
# PGP sign
...
# Upload to artifacts
...

./gradlew shadowJar
# rename
mv opensearch-sql-jdbc-$VERSION.jar opensearch-sql-jdbc-$VERSION-shadow-jarsigner.jar
# JarSigner sign
...
# Upload to artifacts
...

# Update downloads page
...
rishabh6788 commented 1 year ago

opensearch-sql-jdbc-1.3.0.0-shadow.jar.zip @Yury-Fridlyand We decided to go with one shadow jar that can be used by public and tableau, signed using Jarsigner. We need to update the documentation accordingly on the Downloads page. I have attached the signed jar, can you please download and verify from your end and make sure it is working as expected before we upload it to artifacts bucket.

Yury-Fridlyand commented 1 year ago

Yes, I confirm that it works, but the signature is incorrect. It is signed by Amazon Web Services, Inc., but should be OpenSearch Project.

bbarani commented 1 year ago

@Yury-Fridlyand Currently we use the AWS signing pipeline as we are yet to implement OpenSource solution for our signing pipeline. It will be signed by Amazon Web Services, Inc. until that happens.

rishabh6788 commented 1 year ago

@Yury-Fridlyand Signed shadow jar has been uploaded to artifacts bucket.

Yury-Fridlyand commented 1 year ago

Awesome, I can download it: https://artifacts.opensearch.org/opensearch-clients/jdbc/opensearch-sql-jdbc-1.3.0.0-shadow.jar Can I proceed with updating website link or we have to publish signature verification instructions for jarsigner first?

rishabh6788 commented 1 year ago

For now let us remove the existing signature verification steps mentioned and instead state that the artifact is signed using JarSigner. @bbarani

acarbonetto commented 1 year ago

@rishabh6788 the jarsigned artefact is verified and ready to put on the website. Would you like us to raise a PR to update the website?

gaiksaya commented 1 year ago

For now let us remove the existing signature verification steps mentioned and instead state that the artifact is signed using JarSigner. @bbarani

Maybe add steps to verify that signature? @Yury-Fridlyand We can start with creating the PR to update the website. Thanks!

Yury-Fridlyand commented 1 year ago

Awesome!

rishabh6788 commented 1 year ago

@rishabh6788 the jarsigned artefact is verified and ready to put on the website. Would you like us to raise a PR to update the website?

Yes, please go ahead. @acarbonetto Do you need the command to verify using JarSigner that we can add for users to verify?

Yury-Fridlyand commented 1 year ago

I used jarsigner -verify <jar> and jarsigner -verify -verbose <jar> to see the signer information. Issue to publish JDBC on website for tracking: https://github.com/opensearch-project/project-website/issues/1721

gaiksaya commented 1 year ago

Hi @Yury-Fridlyand ,

Can you create a new issue to onboard the jdbc shadow jar publication automation to 1-click release process? I can close this issue as the artifacts are release and related website PR is merged as well.

Thanks!

Yury-Fridlyand commented 1 year ago

Created https://github.com/opensearch-project/sql-jdbc/issues/98 for this. Thanks!