SBT may publish to Maven Central lot more efficiently

[pshirshov]

Many people are experiencing issues publishing their projects to Maven Central with SBT.

Typical complaints are:

1. It takes a lot of time
2. Many staging repositories may be open (see https://github.com/xerial/sbt-sonatype/issues/83) and it may be impossible to promote a release in case, for example, a jar and it's signature file got into separate repositories.

Many projects (ZIO, for example) suffer from these problems.

I've investigated the issue a bit and found that just using Nexus-specific APIs may drastically improve user experience. So, in our case I got a 40x speedup - from 80 minutes with SBT to 2 minutes with Nexus publisher.

The following simple ANT script does the job:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:staging="antlib:org.sonatype.nexus.ant.staging">
    <property environment="env"/>
    <!-- https://oss.sonatype.org/ -->
    <property name="nexus.uri" value="${env.NEXUS_URI}" />
    <property name="nexus.user" value="${env.NEXUS_USER}" />
    <property name="nexus.pass" value="${env.NEXUS_PASS}" />
    <property name="dir.repo" value="${env.DIR_REPO}" />
    <property name="nexus.profile" value="${env.NEXUS_PROFILE}" />

    <taskdef uri="antlib:org.sonatype.nexus.ant.staging"
            resource="org/sonatype/nexus/ant/staging/antlib.xml">
    <classpath>
        <fileset dir="tasks" includes="nexus-staging-ant-tasks-*uber.jar" />
    </classpath>
    </taskdef>

    <staging:nexusStagingInfo id="target-nexus" stagingDirectory="target/staging">
        <staging:projectInfo
                groupId="org.sonatype.nexus.ant"
                artifactId="nexus-staging-ant-tasks"
                stagingProfileId="${nexus.profile}"
                version="1.0" />
        <staging:connectionInfo
            baseUrl="${nexus.uri}">
            <staging:authentication
            username="${nexus.user}"
            password="${nexus.pass}" />
        </staging:connectionInfo>
    </staging:nexusStagingInfo>

    <target name="deploy" description="Deploy: Local and Remote Staging">
        <staging:stageLocally>
            <staging:nexusStagingInfo refid="target-nexus" />
            <fileset dir="${dir.repo}" includes="**/*.*" />
        </staging:stageLocally>

        <staging:stageRemotely>
            <staging:nexusStagingInfo refid="target-nexus" />
        </staging:stageRemotely>

        <staging:releaseStagingRepository>
            <staging:nexusStagingInfo refid="target-nexus" />
        </staging:releaseStagingRepository>

        <staging:dropStagingRepository>
            <staging:nexusStagingInfo refid="target-nexus" />
        </staging:dropStagingRepository>
    </target>

</project>

More details may be found here

In order to prepare input for the script you have to publish to a local directory first with publishTo := Some(Resolver.file("file", new File("/path/to/repo")))

I think that SBT must support Nexus APIs and be able to publish to Central efficiently in transactional manner. Countless hours are being wasted because everyone publishes the usual way. And it should be relatively easy to do it.

Also it worths investigating if the same can be done for Artifactory.

[eed3si9n]

Have you seen https://github.com/xerial/sbt-sonatype? I think this would be an interesting area that can be explored + iterated with plugins without being bound to sbt's release cadence.

[pshirshov]

Of course, we all use sbt-sonatype and I've even linked an issue in its bugtracker. Though it's just a simple wrapper around repo management endpoints, it does not solve publishing problems I mentioned.

A plugin may solve the problem, though it's kinda problematic to achieve an ideal result. You may have a project with multiple aggregating projects in it which aren't parts of one project tree. So we need a global hook allowing us to run our code after they all build. Also it's not so easy to hook into publishing in sbt and we need to prepare a proper maven layout locally. All doable but not just trivial.

At the same time such a functionality is so basic and important (actually there are just two major artifact repositories) so it may be worth considering to be included into the core.

[eed3si9n]

I wonder if part of the issue here has something to do with chattiness of the protocol and/or network latencies between US-elsewhere. For example in https://github.com/sbt/sbt/issues/4199 someone based in Berlin said it sometimes takes 800s (13min) to upload 1MB of artifact, so depending on the size sbt used to timeout at 60 minutes.

I've never seen that myself, but I mostly work along US East coast. I publish things either to staging or to Bintray, and release them afterwards, and none of these operations take 60 min. This is not to say it's not a real problem, but it's an interesting observation that something is at play that makes the UX of Sonatype OSS much worse for some people.

A plugin may solve the problem, though it's kinda problematic to achieve an ideal result. You may have a project with multiple aggregating projects in it which aren't parts of one project tree. So we need a global hook allowing us to run our code after the they all build.

As per global hook is concerned, the primitive I'd recommend for that is a command. If you have some time, I go over that in the first 10 min of sbt core concepts talk I gave.

sbt itself is released using a command:

  commands += Command.command("release") { state =>
    "clean" ::
      "conscriptConfigs" ::
      "compile" ::
      "publishSigned" ::
      "bundledLauncherProj/publishLauncher" ::
      state
  },

[pshirshov]

Nope, it's not related to regions. We have CI nodes in AWS in the US and it's the same.

Though it may be related to "chattiness" - I suspect that sonatype may be rate-limiting our requests.

Re the hook. Thank you for this idea, we may look at this later. For now we are gonna use our ant workaround, but maybe we may write a useful plugin later.

[SethTisue]

2. it may be impossible to promote a release in case, for example, a jar and it's signature file got into separate repositories

fwiw, I have definitely experienced this repeatedly when publishing Scala modules. (I should have opened an issue on it long ago? but I guess I just assumed, without really thinking about it, that it was somehow an inherent issue at Sonatype's end.)

[pshirshov]

@SethTisue : nope, it's not Sonatype, it the combination of basic maven protocol and staging repositories on Sonatype (and possibly rate limiters on Sonatype.

You may try to employ my workaround and enjoy the outcome.

Though I think such a functionality must be provided by SBT (maybe as a plugin) and most likely reimplemented from scratch (fortunately it's not so hard) - the ant plugin is not maintained anymore.

[dwijnand]

Thanks @pshirshov for bringing attention to this!

For the thread: see https://github.com/xerial/sbt-sonatype/issues/89 where "bundle upload" support was added to sbt-sonatype 3.1+

(Eventually, we should move this behaviour into sbt, IMO, perhaps for sbt 1.4.)

[yawaramin]

I suggest label area/performance for this. Possibly help wanted (if that is the case).

[eed3si9n]

Labeled.