search

apache / pekko

Build highly concurrent, distributed, and resilient message-driven applications using Java/Scala
https://pekko.apache.org/
Apache License 2.0
1.17k stars 139 forks source link

Create a `sbt-paradox-pekko` sbt plugin #92

Closed mdedetrich closed 1 year ago

mdedetrich commented 1 year ago

With https://github.com/apache/incubator-pekko/pull/91 we have managed to replace the sbt-paradox-akka theme with the standard one from paradox however it was just done by manually putting in all of the sbt-paradox-* plugins. Ideally however we would like to create our own sbt-paradox-pekko sbt plugin akin to sbt-paradox-akka. This will become especially important when we actually want to create our own bespoke pekko doc theme which we will share within all of our pekko modules.

The only slightly complicating factor is that currently pekko core sbt build enforces JDK 1.8 (sbt doesn't even load unless SBT is its run with JDK 1.8 on the path). The problem here is due to the design of one of sbt-paradox's core dependencies parboiled, you can only use specific versions of parboiled on JDK 1.8 versus later. In summary, pekko core only only use an outdated set of sbt-paradox-* dependencies where as other pekko modules which are built with newer JDK's will only support the newer sbt-paradox-* dependencies.

Thankfully I have managed to resolve this issue, assuming you have sbt-paradox-pekko with the newer sbt-paradox-* dependencies you can do this in pekko core's build.sbt

// We have to deliberately use older versions of sbt-paradox because current Pekko sbt build
// only loads on JDK 1.8 so we need to bring in older versions of parboiled which support JDK 1.8
addSbtPlugin("org.apache.pekko" % "sbt-paradox-pekko" % "HEAD+20230107-1453" excludeAll(
  "com.lightbend.paradox","sbt-paradox",
  "com.lightbend.paradox" % "sbt-paradox-apidoc",
  "com.lightbend.paradox" % "sbt-paradox-project-info"
))
addSbtPlugin("com.lightbend.paradox" % "sbt-paradox" % "0.9.2" force())
addSbtPlugin("com.lightbend.paradox" % "sbt-paradox-apidoc" % "0.10.1" force())
addSbtPlugin("com.lightbend.paradox" % "sbt-paradox-project-info" % "2.0.0" force())

I already have a prototype sbt-paradox-pekko plugin locally, it doesn't implement any theme but at least it brings in the sbt-paradox-* dependencies and will allow us to enforce a consistent theme.

pjfanning commented 1 year ago

you don't need an INFRA ticket - they have created https://selfserve.apache.org/ - I can create the extra repo.

pjfanning commented 1 year ago

created https://github.com/apache/incubator-pekko-sbt-paradox - it has to start with incubator-pekko - the main branch is not yet protected

mdedetrich commented 1 year ago

Thanks for organizing this! Will get onto it in the next few days

jrudolph commented 1 year ago

The problem here is due to the design of one of sbt-paradox's core dependencies parboiled, you can only use specific versions of parboiled on JDK 1.8 versus later.

Ok, that's because the latest paradox depends on parboiled 1.4.1 which is the first version to remove Java 8 support and consequently paradox itself had to remove Java 8 support (otherwise we could just pin parboiled to 1.4.0 here). I guess that was something of an oversight when releasing paradox 0.10.0 because it could (and maybe should) have stayed on parboiled 1.4.0 to keep Java 8 compatibility (if only to make upgrading to latest paradox easier for Akka itself going forward).

pjfanning commented 1 year ago

The incubator-pekko-sbt-paradox plugin will need to add the following to the HTML pages we generate.

jrudolph commented 1 year ago

I'm looking into setting this up. One problem, is that we have a kind of chicken and egg problem here, that AFAIU we cannot easily just make a public release of any kind of plugin even if it's just for internal consumption (especially, as it will likely need some iterations to stabilize). What options do we have?

I'd start with setting docs up in the main repo first, so that we are not blocked by a missing docs page for the first release.

mdedetrich commented 1 year ago

I'm looking into setting this up.

So I already have a local plugin as a template, should I just send it over to you or should I just push what I have? What I have locally doesn't have any theme, it just contains the other sbt plugins (i.e. project-info) and its been adapted from https://github.com/akka/akka-paradox

One problem, is that we have a kind of chicken and egg problem here, that AFAIU we cannot easily just make a public release of any kind of plugin even if it's just for internal consumption (especially, as it will likely need some iterations to stabilize). What options do we have?

To me the most pragmatic solution would be to just publish it as a snapshot as the org.apache.pekko org into the Apache Nexus Maven repo just as we have done with Pekko. I would set it up to just publish snapshots after something is pushed into main, that way its not so annoying to deal with incremental snapshot versions.

justinmclean commented 1 year ago

Hi,

Out of interest what’s stopping you from making a pubic source release?

Justin

jrudolph commented 1 year ago

Out of interest what’s stopping you from making a pubic source release?

We need a ivy/Maven binary release of the plugin to be able to consume it (which is somewhat difficult in itself, because sbt plugins usually need an ivy repository to publish to (though, Maven usually also works even if it uses non-standard naming)).

In general, I do not exactly know what it would entail to make a public release of any kind for a tool used for only developer consumption (does it have to go through the IPMC voting process etc.?) but I also do not have time to figure that out right for something that is just a potential convenience later on.

mdedetrich commented 1 year ago

Out of interest what’s stopping you from making a pubic source release?

The reason why we are not looking at a public source release initially, is that similar to a JVM library like Apache Commons dependency resolution rely on resolving artifacts from Maven, not from a local source directory. Specifically with sbt, it uses addSbtPlugin which works by resolving a maven repository (see https://www.scala-sbt.org/1.x/docs/Using-Plugins.html). Do note that this is an plugin for sbt which is a build tool, its not going to be code that will be executed as part of using the library.

I would imagine this is not dissimilar to projects like Apache Commons which even though has to make a source release to comply with ASF guidelines, since its a library the vast majority of users are not going to download the source package. They will add it as a dependency in their pom.xml, build.sbt, build.grade etc etc which will download the binary from ASF Nexus repo.

This is one of the disconnects I was talking about earlier, if the Apache Project happens to be a library and not an "executable application" almost no users will download the source package and in cases like this its not even technically feasible (or desirable) to use a source package and Pekko is not unique in this regard, other TLP's which are libraries like Apache Commons are also treated in the same way.

justinmclean commented 1 year ago

Hi,

Apache makes source releases, binaries can be created to help users but they are optional. 3rd parties (which could involve people in the project) can also provide binary releases based on Apache source releases.

This may help https://infra.apache.org/publishing-maven-artifacts.html https://infra.apache.org/publishing-maven-artifacts.html

Kind Regards, Justin

mdedetrich commented 1 year ago

A source release package can be made for this sbt plugin, but doing it right now is incredibly premature. Its also an exceptional case because we are dealing with a plugin whose goal is to remove boilerplate from pekko sbt projects. If we just repeated the boilerplate that this plugin would be abstracting over in every sbt build definition we wouldn't even be having this discussion.

Apache makes source releases, binaries can be created to help users but they are optional. 3rd parties (which could involve people in the project) can also provide binary releases based on Apache source releases.

In the case of JVM libraries, from the perspective of users the binaries are not optional because the entire JVM ecosystem is built around resolving libraries as binaries. Almost all users in this context will ignore source packages, not willingly or out of malice but because none of the build tools for JVM projects are built around and/or expect source packages. This especially so in the case of sbt, as I said before sbt plugins only work with binaries published to maven.

This is the disconnect I am talking about and in my view it would be more appropriate to discuss this in a proper channel (I don't even know what channel this is, INFRA?)

jrudolph commented 1 year ago

Let's not get misled that we need a release (in Apache terms) of the plugin discussed here. The whole idea of the plugin is just an optimization for us developers. In the end, creating a package of the plugin is just like caching some intermediate results, so let's treat it like that and try to find a solution that does not require an Apache release or which would treat the plugin as a full-fledged module of pekko.

mdedetrich commented 1 year ago

Let's not get misled that we need a release (in Apache terms) of the plugin discussed here. The whole idea of the plugin is just an optimization for us developers. In the end, creating a package of the plugin is just like caching some intermediate results, so let's treat it like that and try to find a solution that does not require an Apache release or which would treat the plugin as a full-fledged module of pekko.

Agreed, I also do not want to treat this as an official ASF project. The best way to think of it is a common script that is shared between multiple Pekko projects.

There might (and real emphasis on might) be a reason to make it a proper release, but this would be way down the line, again re-iterating its premature now.

pjfanning commented 1 year ago

One option would be for someone to just create this repo as a personal, non-ASF, repo.

I don't see the harm in keeping this as an ASF repo and using snapshot jars for the next while, while the plugin code is still changing a lot. We could later do a 1.0.0 ASF release and include bin and src distributions. Once we learn how to build those for the main pekko repos, it should be straightforward to apply that here.

FYI, I plan to develop https://github.com/pjfanning/sbt-source-dist as a non-ASF project (at least, for now).

mdedetrich commented 1 year ago

FYI, I plan to develop https://github.com/pjfanning/sbt-source-dist as a non-ASF project (at least, for now).

I was also planning to develop/prototype a sbt-apache-release plugin which would use sbt-sonatype/sbt-pgp along with your new sbt-source-dist as a way to collect all of the steps (that make sense to do so) to create a single sbt release step. I will wait until your sbt-source-dist plugin is usable in sbt though

mdedetrich commented 1 year ago

@jrudolph I have just pushed a template to https://github.com/apache/incubator-pekko-sbt-paradox and also created an INFRA ticket to set it up for Apache Nexus deployment (see https://issues.apache.org/jira/browse/INFRA-24157)

pjfanning commented 1 year ago

https://github.com/apache/incubator-pekko-sbt-paradox exists - we should use that repo (issues/PRs/discussions) to track status of that work

mdedetrich commented 1 year ago

@justinmclean I just had a discussion with @Claudenw so I just want to clarify some things.

Earlier when I (and others) was talking about a ivy/Maven binary release, this terminology might be a bit confusing because it doesn't just contain a binary (specifically for JVM we are talking about .class files) but also sources and other files. To illustrate this, in the https://github.com/apache/incubator-pekko-sbt-paradox project if you run sbt publishLocal it will create the artifacts and deploy them into a local repository, i.e.

[info] :: delivering :: org.apache.pekko#pekko-sbt-paradox;0.0.0+18-8f2ba638-SNAPSHOT :: 0.0.0+18-8f2ba638-SNAPSHOT :: integration :: Wed Feb 15 11:39:15 CET 2023
[info]  delivering ivy file to /Users/mdedetrich/github/incubator-pekko-sbt-paradox/plugin/target/scala-2.12/sbt-1.0/ivy-0.0.0+18-8f2ba638-SNAPSHOT.xml
[info]  published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/poms/pekko-sbt-paradox.pom
[info]  published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/jars/pekko-sbt-paradox.jar
[info]  published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/srcs/pekko-sbt-paradox-sources.jar
[info]  published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/docs/pekko-sbt-paradox-javadoc.jar

As you can see it creates a pekko-sbt-paradox-sources.jar and this jar file does contain the sources used to build the plugin (actually in contrast to a standard Apache source distribution, the *-sources.jar file is far more accurate because it contains ONLY the sources which are used to build the plugin and nothing else). For each of these jar files there is also .sha1 and .md5 hashes, i.e.

<~/.i/l/o/p/s/s/0/srcs>-> ls
pekko-sbt-paradox-sources.jar       pekko-sbt-paradox-sources.jar.md5   pekko-sbt-paradox-sources.jar.sha1

Additionally if you were to create a signed package (i.e. publishLocalSigned) which signs it via gpg/pgp it would also contain expected signature files (such as .asc) which you can use to verify the signature.

I have attached an example distribution 0.0.0+18-8f2ba638-SNAPSHOT.zip, note that .jar files are just standard zip files if you want to introspect them.

In summary in regards to sbt and hence pekko and these plugins that we talk about, there is nothing special/bespoke/unique about this. The sbt build tool by default will make sure that not only the binaries but the sources and other relevant meta information are distributed in Maven Repositories (whether that be Apache's Maven repo, Sonatypes OSS repo or a local folder repository as used in the previous example).

The only exception to this is the various LICENSE/NOTICE/DISCLAIMER and other files which as you have noticed from the other repositories are being handled additionally.

pjfanning commented 1 year ago

Since this is an Apache project, we will still need to produce source releases (using sbt-source-dist) and go through the release process, including votes in the Pekko PPMC and the Incubator PMC.

I don't see anything significantly different in the Maven publish of this project - compared to incubator-pekko, for instance.

mdedetrich commented 1 year ago

Since this is an Apache project, we will still need to produce source releases (using sbt-source-dist) and go through the release process, including votes in the Pekko PPMC and the Incubator PMC.

Yes, I was just clarifying that when we talked about a "Maven binary release", its strictly speaking not just a pure binary.

I don't see anything significantly different in the Maven publish of this project - compared to incubator-pekko, for instance.

Agreed