Upload paradox/scala+java docs into nightlies folder

[mdedetrich]

As discussed in https://the-asf.slack.com/archives/C0491A12EP2/p1675850262146849, at least initially there is a preliminary decision to upload generated paradox/scala+java docs into the apache nightlies (i.e. https://nightlies.apache.org/). The idea is the main website (which is separate and hosted at http://pekko.apache.org/) would point to the docs uploaded to nightlies. The reasons why to do this are

• Nightlies will automatically host any content that happens to have an index.html in a root folder. This makes it really trivial to host generated docs (see https://nightlies.apache.org/flink/flink-ml-docs-master/ as an example)
• Uploading to nightlies is very easy and almost all of the work has already been done, see https://github.com/apache/incubator-pekko/blob/main/.github/actions/sync-nightlies/action.yml. The only changes we would need to do is to run sbt unidoc paradox and change the LOCAL_PATH and REMOTE_PATH)
  • Each pekko module would have its own folder (i.e. pekko core would sit in /pekko folder, pekko-http would sit in /pekko-http)
  • Under that module folder there would be another one which would designate the version, i.e. 1.0.0 would sit under /pekko/1.0.0. Same deal with snapshots but they would have an expiry.
  • There will also be a /pekko/current/<content> folder which will always point to the latest release
• The core website at http://pekko.apache.org/ will then link to these docs

The Flink TLP project uses this method (go to https://flink.apache.org/ and on the left menu navigate to Documentation), so there is prior art for this and we can also ask them any questions. The reason why I am advocating for this solution is that its by far the simplest and easiest to get going, its quite similar to what other Scala/OS projects do (especially Scala projects that use Paradox), in fact Lightbend used almost exactly the same method with Akka (thats why Akka/Pekko contains https://github.com/lightbend/sbt-publish-rsync). Its more principled in design especially when taking into account the fact that docs generated from a release (i.e. paradox/scala+java doc) have a different life/update cycle compared to the main website at https://github.com/apache/incubator-pekko-site. The main site would be updated on the https://github.com/apache/incubator-pekko-site git repo where as the doc updating would be handed by a github action/release manager within sbt.

The potential drawbacks are the size of the generated docs (~510 megabytes) although there isn't much we can do about this (scala/javadocs take up almost all of the size and its just how they are generated) and whether putting release documentation in Apache nightlies is problematic (considering that Flink does this I don't think its a concern 🤷 ). Irregardless we should also look into this.

[jrudolph]

Alternatively, here's a script draft to upload generated docs and apidocs to https://github.com/apache/incubator-pekko-site:

#!/bin/sh

set -ex

# Expects generated docs in these folders
# target/javaunidoc
# target/scala-2.13/unidoc
# docs/target/paradox

TARGET_REPO="https://github.com/apache/incubator-pekko-site"
REPO="https://github.com/jrudolph/incubator-pekko-site"
TMPDOCS="/tmp/docstmp"
BRANCH="new-docs"
TARGET_BRANCH="asf-staging"
GITOPTS="--git-dir $TMPDOCS/.git --work-tree $TMPDOCS"

mkdir $TMPDOCS

git clone --depth 1 -b $TARGET_BRANCH $TARGET_REPO $TMPDOCS
git $GITOPTS remote add my $REPO
git $GITOPTS checkout -b $BRANCH
git $GITOPTS rm -r content/api content/japi content/docs

cp -r target/javaunidoc $TMPDOCS/content/japi
cp -r target/scala-2.13/unidoc $TMPDOCS/content/api
cp -r docs/target/paradox $TMPDOCS/content/docs
git $GITOPTS add -A content
git $GITOPTS commit -m "publish docs"
git $GITOPTS push my $BRANCH

echo "Now open $REPO/pull/new/$BRANCH to open PR"

[mdedetrich]

Thanks! Because it makes sense to upload docs for snapshots to nightly (along with an expiry due to their size) I will go ahead and do this anyways. Since the flow is the same we can then also put docs for releases into nightlies and point to that, if there is an ASF process/incubator issue then we can look into using this script.

I also think that the script is not enough, we would also have to either have to document or automate changing the relevant paradox/unidoc settings in sbt to update the absolute link host (since its going to be in a different location), this is one advantage that using nightlies has is that we can just (for now) hardcode the URI in the sbt settings

[jrudolph]

One consideration regarding regular site vs. nightlies is also how these domains are served and how CDNs get involved.

[mdedetrich]

One consideration regarding regular site vs. nightlies is also how these domains are served and how CDNs get involved.

True, I would assume that since Flink (one of the most popular ASF TLP's) is doing it its a non concern but thats a bold assumption