Closed mdedetrich closed 1 year ago
you don't need an INFRA ticket - they have created https://selfserve.apache.org/ - I can create the extra repo.
created https://github.com/apache/incubator-pekko-sbt-paradox - it has to start with incubator-pekko - the main branch is not yet protected
Thanks for organizing this! Will get onto it in the next few days
The problem here is due to the design of one of
sbt-paradox
's core dependencies parboiled, you can only use specific versions of parboiled on JDK 1.8 versus later.
Ok, that's because the latest paradox depends on parboiled 1.4.1 which is the first version to remove Java 8 support and consequently paradox itself had to remove Java 8 support (otherwise we could just pin parboiled to 1.4.0 here). I guess that was something of an oversight when releasing paradox 0.10.0 because it could (and maybe should) have stayed on parboiled 1.4.0 to keep Java 8 compatibility (if only to make upgrading to latest paradox easier for Akka itself going forward).
The incubator-pekko-sbt-paradox plugin will need to add the following to the HTML pages we generate.
I'm looking into setting this up. One problem, is that we have a kind of chicken and egg problem here, that AFAIU we cannot easily just make a public release of any kind of plugin even if it's just for internal consumption (especially, as it will likely need some iterations to stabilize). What options do we have?
I'd start with setting docs up in the main repo first, so that we are not blocked by a missing docs page for the first release.
I'm looking into setting this up.
So I already have a local plugin as a template, should I just send it over to you or should I just push what I have? What I have locally doesn't have any theme, it just contains the other sbt plugins (i.e. project-info) and its been adapted from https://github.com/akka/akka-paradox
One problem, is that we have a kind of chicken and egg problem here, that AFAIU we cannot easily just make a public release of any kind of plugin even if it's just for internal consumption (especially, as it will likely need some iterations to stabilize). What options do we have?
To me the most pragmatic solution would be to just publish it as a snapshot as the org.apache.pekko
org into the Apache Nexus Maven repo just as we have done with Pekko. I would set it up to just publish snapshots after something is pushed into main, that way its not so annoying to deal with incremental snapshot versions.
Hi,
Out of interest what’s stopping you from making a pubic source release?
Justin
Out of interest what’s stopping you from making a pubic source release?
We need a ivy/Maven binary release of the plugin to be able to consume it (which is somewhat difficult in itself, because sbt plugins usually need an ivy repository to publish to (though, Maven usually also works even if it uses non-standard naming)).
In general, I do not exactly know what it would entail to make a public release of any kind for a tool used for only developer consumption (does it have to go through the IPMC voting process etc.?) but I also do not have time to figure that out right for something that is just a potential convenience later on.
Out of interest what’s stopping you from making a pubic source release?
The reason why we are not looking at a public source release initially, is that similar to a JVM library like Apache Commons dependency resolution rely on resolving artifacts from Maven, not from a local source directory. Specifically with sbt, it uses addSbtPlugin
which works by resolving a maven repository (see https://www.scala-sbt.org/1.x/docs/Using-Plugins.html). Do note that this is an plugin for sbt which is a build tool, its not going to be code that will be executed as part of using the library.
I would imagine this is not dissimilar to projects like Apache Commons which even though has to make a source release to comply with ASF guidelines, since its a library the vast majority of users are not going to download the source package. They will add it as a dependency in their pom.xml
, build.sbt
, build.grade
etc etc which will download the binary from ASF Nexus repo.
This is one of the disconnects I was talking about earlier, if the Apache Project happens to be a library and not an "executable application" almost no users will download the source package and in cases like this its not even technically feasible (or desirable) to use a source package and Pekko is not unique in this regard, other TLP's which are libraries like Apache Commons are also treated in the same way.
Hi,
Apache makes source releases, binaries can be created to help users but they are optional. 3rd parties (which could involve people in the project) can also provide binary releases based on Apache source releases.
This may help https://infra.apache.org/publishing-maven-artifacts.html https://infra.apache.org/publishing-maven-artifacts.html
Kind Regards, Justin
A source release package can be made for this sbt plugin, but doing it right now is incredibly premature. Its also an exceptional case because we are dealing with a plugin whose goal is to remove boilerplate from pekko sbt projects. If we just repeated the boilerplate that this plugin would be abstracting over in every sbt build definition we wouldn't even be having this discussion.
Apache makes source releases, binaries can be created to help users but they are optional. 3rd parties (which could involve people in the project) can also provide binary releases based on Apache source releases.
In the case of JVM libraries, from the perspective of users the binaries are not optional because the entire JVM ecosystem is built around resolving libraries as binaries. Almost all users in this context will ignore source packages, not willingly or out of malice but because none of the build tools for JVM projects are built around and/or expect source packages. This especially so in the case of sbt, as I said before sbt plugins only work with binaries published to maven.
This is the disconnect I am talking about and in my view it would be more appropriate to discuss this in a proper channel (I don't even know what channel this is, INFRA?)
Let's not get misled that we need a release (in Apache terms) of the plugin discussed here. The whole idea of the plugin is just an optimization for us developers. In the end, creating a package of the plugin is just like caching some intermediate results, so let's treat it like that and try to find a solution that does not require an Apache release or which would treat the plugin as a full-fledged module of pekko.
Let's not get misled that we need a release (in Apache terms) of the plugin discussed here. The whole idea of the plugin is just an optimization for us developers. In the end, creating a package of the plugin is just like caching some intermediate results, so let's treat it like that and try to find a solution that does not require an Apache release or which would treat the plugin as a full-fledged module of pekko.
Agreed, I also do not want to treat this as an official ASF project. The best way to think of it is a common script that is shared between multiple Pekko projects.
There might (and real emphasis on might) be a reason to make it a proper release, but this would be way down the line, again re-iterating its premature now.
One option would be for someone to just create this repo as a personal, non-ASF, repo.
I don't see the harm in keeping this as an ASF repo and using snapshot jars for the next while, while the plugin code is still changing a lot. We could later do a 1.0.0 ASF release and include bin and src distributions. Once we learn how to build those for the main pekko repos, it should be straightforward to apply that here.
FYI, I plan to develop https://github.com/pjfanning/sbt-source-dist as a non-ASF project (at least, for now).
FYI, I plan to develop https://github.com/pjfanning/sbt-source-dist as a non-ASF project (at least, for now).
I was also planning to develop/prototype a sbt-apache-release
plugin which would use sbt-sonatype/sbt-pgp along with your new sbt-source-dist as a way to collect all of the steps (that make sense to do so) to create a single sbt release
step. I will wait until your sbt-source-dist plugin is usable in sbt though
@jrudolph I have just pushed a template to https://github.com/apache/incubator-pekko-sbt-paradox and also created an INFRA ticket to set it up for Apache Nexus deployment (see https://issues.apache.org/jira/browse/INFRA-24157)
https://github.com/apache/incubator-pekko-sbt-paradox exists - we should use that repo (issues/PRs/discussions) to track status of that work
@justinmclean I just had a discussion with @Claudenw so I just want to clarify some things.
Earlier when I (and others) was talking about a ivy/Maven binary release, this terminology might be a bit confusing because it doesn't just contain a binary (specifically for JVM we are talking about .class
files) but also sources and other files. To illustrate this, in the https://github.com/apache/incubator-pekko-sbt-paradox project if you run sbt publishLocal
it will create the artifacts and deploy them into a local repository, i.e.
[info] :: delivering :: org.apache.pekko#pekko-sbt-paradox;0.0.0+18-8f2ba638-SNAPSHOT :: 0.0.0+18-8f2ba638-SNAPSHOT :: integration :: Wed Feb 15 11:39:15 CET 2023
[info] delivering ivy file to /Users/mdedetrich/github/incubator-pekko-sbt-paradox/plugin/target/scala-2.12/sbt-1.0/ivy-0.0.0+18-8f2ba638-SNAPSHOT.xml
[info] published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/poms/pekko-sbt-paradox.pom
[info] published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/jars/pekko-sbt-paradox.jar
[info] published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/srcs/pekko-sbt-paradox-sources.jar
[info] published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/docs/pekko-sbt-paradox-javadoc.jar
As you can see it creates a pekko-sbt-paradox-sources.jar
and this jar file does contain the sources used to build the plugin (actually in contrast to a standard Apache source distribution, the *-sources.jar
file is far more accurate because it contains ONLY the sources which are used to build the plugin and nothing else). For each of these jar files there is also .sha1
and .md5
hashes, i.e.
<~/.i/l/o/p/s/s/0/srcs>-> ls
pekko-sbt-paradox-sources.jar pekko-sbt-paradox-sources.jar.md5 pekko-sbt-paradox-sources.jar.sha1
Additionally if you were to create a signed package (i.e. publishLocalSigned
) which signs it via gpg/pgp it would also contain expected signature files (such as .asc
) which you can use to verify the signature.
I have attached an example distribution 0.0.0+18-8f2ba638-SNAPSHOT.zip, note that .jar
files are just standard zip files if you want to introspect them.
In summary in regards to sbt and hence pekko and these plugins that we talk about, there is nothing special/bespoke/unique about this. The sbt build tool by default will make sure that not only the binaries but the sources and other relevant meta information are distributed in Maven Repositories (whether that be Apache's Maven repo, Sonatypes OSS repo or a local folder repository as used in the previous example).
The only exception to this is the various LICENSE
/NOTICE
/DISCLAIMER
and other files which as you have noticed from the other repositories are being handled additionally.
Since this is an Apache project, we will still need to produce source releases (using sbt-source-dist) and go through the release process, including votes in the Pekko PPMC and the Incubator PMC.
I don't see anything significantly different in the Maven publish of this project - compared to incubator-pekko, for instance.
Since this is an Apache project, we will still need to produce source releases (using sbt-source-dist) and go through the release process, including votes in the Pekko PPMC and the Incubator PMC.
Yes, I was just clarifying that when we talked about a "Maven binary release", its strictly speaking not just a pure binary.
I don't see anything significantly different in the Maven publish of this project - compared to incubator-pekko, for instance.
Agreed
With https://github.com/apache/incubator-pekko/pull/91 we have managed to replace the
sbt-paradox-akka
theme with the standard one from paradox however it was just done by manually putting in all of thesbt-paradox-*
plugins. Ideally however we would like to create our ownsbt-paradox-pekko
sbt plugin akin tosbt-paradox-akka
. This will become especially important when we actually want to create our own bespoke pekko doc theme which we will share within all of our pekko modules.The only slightly complicating factor is that currently pekko core sbt build enforces JDK 1.8 (sbt doesn't even load unless SBT is its run with JDK 1.8 on the path). The problem here is due to the design of one of
sbt-paradox
's core dependencies parboiled, you can only use specific versions of parboiled on JDK 1.8 versus later. In summary, pekko core only only use an outdated set ofsbt-paradox-*
dependencies where as other pekko modules which are built with newer JDK's will only support the newersbt-paradox-*
dependencies.Thankfully I have managed to resolve this issue, assuming you have
sbt-paradox-pekko
with the newersbt-paradox-*
dependencies you can do this in pekko core'sbuild.sbt
I already have a prototype
sbt-paradox-pekko
plugin locally, it doesn't implement any theme but at least it brings in thesbt-paradox-*
dependencies and will allow us to enforce a consistent theme.sbt-paradox-pekko
sbt plugin from local git repo to a temporary orgincubator-sbt-paradox-pekko
git reposbt-paradox-pekko
sbt plugin to the apache maven repo so it can be used in pekko repos. Can also use this simple project as a test bed for https://github.com/apache/incubator-pekko/issues/85