search

djspiewak / sbt-github-packages

A simple sbt plugin for publishing to GitHub Packages, in the style of sbt-sonatype and sbt-bintray
Apache License 2.0
175 stars 27 forks source link

GitHub Packages doesn't seem to support sbt (yet) #1

Closed djspiewak closed 4 years ago

djspiewak commented 5 years ago

Sbt's publication model just doesn't appear to work with GitHub Packages, for reasons that aren't entirely clear. This tweet seems to imply that this is a known issue, or at the very least something that isn't entirely unexpected. Once GitHub adds support for sbt, or at least stops breaking it, we'll close this issue and do a proper release of the working plugin.

For the record, the error looks like this:

[info] [info] Set current project to sbt-github-packages-tests-publish (in build file:/private/var/folders/vm/h_lhw5wn573cw70qd6l0ljt80000gn/T/sbt_78381fbc/publish/)
[info] [info] Packaging /private/var/folders/vm/h_lhw5wn573cw70qd6l0ljt80000gn/T/sbt_78381fbc/publish/target/scala-2.12/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT-sources.jar ...
[info] [info] Updating ...
[info] [info] Done packaging.
[info] [info] Wrote /private/var/folders/vm/h_lhw5wn573cw70qd6l0ljt80000gn/T/sbt_78381fbc/publish/target/scala-2.12/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT.pom
[info] [info] Done updating.
[info] [info] Packaging /private/var/folders/vm/h_lhw5wn573cw70qd6l0ljt80000gn/T/sbt_78381fbc/publish/target/scala-2.12/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT-javadoc.jar ...
[info] [info] Done packaging.
[info] [info] Packaging /private/var/folders/vm/h_lhw5wn573cw70qd6l0ljt80000gn/T/sbt_78381fbc/publish/target/scala-2.12/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT.jar ...
[info] [info] Done packaging.
[info] [info]   published sbt-github-packages-tests-publish_2.12 to https://maven.pkg.github.com/djspiewak/sbt-github-packages/com/codecommit/sbt-github-packages-tests-publish_2.12/0.1-SNAPSHOT/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT.pom
[info] [error] java.io.IOException: Access to URL https://maven.pkg.github.com/djspiewak/sbt-github-packages/com/codecommit/sbt-github-packages-tests-publish_2.12/0.1-SNAPSHOT/sbt-github-packages-tests-publish_2.12-0.1-SNAPSHOT.jar was refused by the server: Forbidden
[info] [error]  at org.apache.ivy.util.url.AbstractURLHandler.validatePutStatusCode(AbstractURLHandler.java:79)
[info] [error]  at sbt.internal.librarymanagement.ivyint.GigahorseUrlHandler.upload(GigahorseUrlHandler.scala:191)
[info] [error]  at org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
[info] [error]  at org.apache.ivy.util.FileUtil.copy(FileUtil.java:150)
[info] [error]  at org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:84)
[info] [error]  at sbt.internal.librarymanagement.ConvertResolver$LocalIfFileRepo.put(ConvertResolver.scala:366)
[info] [error]  at org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
[info] [error]  at sbt.internal.librarymanagement.ConvertResolver$ChecksumFriendlyURLResolver.put(ConvertResolver.scala:118)
[info] [error]  at sbt.internal.librarymanagement.ConvertResolver$ChecksumFriendlyURLResolver.put$(ConvertResolver.scala:105)
[info] [error]  at sbt.internal.librarymanagement.ConvertResolver$$anonfun$defaultConvert$lzycompute$1$PluginCapableResolver$1.put(ConvertResolver.scala:165)
[info] [error]  at org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:216)
[info] [error]  at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$5(IvyActions.scala:497)
[info] [error]  at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$5$adapted(IvyActions.scala:496)
[info] [error]  at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:788)
[info] [error]  at scala.collection.Iterator.foreach(Iterator.scala:937)
[info] [error]  at scala.collection.Iterator.foreach$(Iterator.scala:937)
[info] [error]  at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
[info] [error]  at scala.collection.IterableLike.foreach(IterableLike.scala:70)
[info] [error]  at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
[info] [error]  at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[info] [error]  at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:787)
[info] [error]  at sbt.internal.librarymanagement.IvyActions$.publish(IvyActions.scala:496)
[info] [error]  at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$3(IvyActions.scala:144)
[info] [error]  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[info] [error]  at sbt.internal.librarymanagement.IvyActions$.withChecksums(IvyActions.scala:157)
[info] [error]  at sbt.internal.librarymanagement.IvyActions$.withChecksums(IvyActions.scala:151)
[info] [error]  at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$1(IvyActions.scala:144)
[info] [error]  at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$1$adapted(IvyActions.scala:134)
[info] [error]  at sbt.internal.librarymanagement.IvySbt$Module.$anonfun$withModule$1(Ivy.scala:239)
[info] [error]  at sbt.internal.librarymanagement.IvySbt.$anonfun$withIvy$1(Ivy.scala:204)
[info] [error]  at sbt.internal.librarymanagement.IvySbt.sbt$internal$librarymanagement$IvySbt$$action$1(Ivy.scala:70)
[info] [error]  at sbt.internal.librarymanagement.IvySbt$$anon$3.call(Ivy.scala:77)
[info] [error]  at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:95)
[info] [error]  at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:80)
[info] [error]  at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:99)
[info] [error]  at xsbt.boot.Using$.withResource(Using.scala:10)
[info] [error]  at xsbt.boot.Using$.apply(Using.scala:9)
[info] [error]  at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:60)
[info] [error]  at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:50)
[info] [error]  at xsbt.boot.Locks$.apply0(Locks.scala:31)
[info] [error]  at xsbt.boot.Locks$.apply(Locks.scala:28)
[info] [error]  at sbt.internal.librarymanagement.IvySbt.withDefaultLogger(Ivy.scala:77)
[info] [error]  at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:199)
[info] [error]  at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:196)
[info] [error]  at sbt.internal.librarymanagement.IvySbt$Module.withModule(Ivy.scala:238)
[info] [error]  at sbt.internal.librarymanagement.IvyActions$.publish(IvyActions.scala:134)
[info] [error]  at sbt.Classpaths$.$anonfun$publishTask$4(Defaults.scala:2416)
[info] [error]  at sbt.Classpaths$.$anonfun$publishTask$4$adapted(Defaults.scala:2416)
[info] [error]  at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[info] [error]  at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:40)
[info] [error]  at sbt.std.Transform$$anon$4.work(System.scala:67)
[info] [error]  at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
[info] [error]  at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[info] [error]  at sbt.Execute.work(Execute.scala:278)
[info] [error]  at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
[info] [error]  at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[info] [error]  at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[info] [error]  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info] [error]  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[info] [error]  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info] [error]  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info] [error]  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info] [error]  at java.lang.Thread.run(Thread.java:748)

If you look towards the top, you'll see that the pom successfully published, but the first jar file was rejected with Forbidden. @alexarchambault reports that the ordering of the publishes is irrelevant, and jars are rejected regardless.

lokkju commented 5 years ago

As far as I can tell, -sources, -javadoc, -tests are not supported

using sbt-aether-deploy, I have successfully published to github packages from sbt

project/plugins.sbt

addSbtPlugin("no.arktekk.sbt" % "aether-deploy" % "0.23.0")
addSbtPlugin("com.dwijnand" % "sbt-dynver" % "4.0.0")

build.sbt

externalResolvers += "GitHub lokkju Apache Maven Packages" at "https://maven.pkg.github.com/lokkju/github-action-sbt"
publishTo := Some("GitHub lokkju Apache Maven Packages" at "https://maven.pkg.github.com/lokkju/github-action-sbt")
credentials += Credentials("GitHub Package Registry", "maven.pkg.github.com", "lokkju", "<GITHUB_TOKEN>")

// GitHub package repo isn't supporting javadoc and sources
publishArtifact in (Compile, packageDoc) := false
publishArtifact in (Compile, packageSrc) := false

sbt aetherDeploy

lokkju commented 5 years ago

An additional aspect; if I use the Charles Web Debugging Proxy, everything works fine. Without it, I occasionally get errors about the server failing to response.

djspiewak commented 5 years ago

@lokkju Hmm, what about multi-module builds?

lokkju commented 5 years ago

I haven't tried them; key here seems to be using aether-deploy, though it doesn't entirely fix it. I'm suspecting it's an issue with the SSL support in the client versions used by each tool.

I'm going to try a few different java http libraries, and see if I can replacate a simple working and not working configuration. I'll keep you updated.

lokkju commented 5 years ago

So, after more exploration, I've found some interesting symptoms.

Just using curl to try to upload packages, I've found;

The key takeaway currently is that you must upload the JAR artifact first, then publish the POM.

I have an email into support, we'll see if they provide more information.

djspiewak commented 5 years ago

Hmm, so no artifact signing either, then?

lokkju commented 5 years ago

It auto-generates the .md5 and .sha1 for any files uploaded.

Also, found this: https://github.com/sbt/librarymanagement/blob/d09f9f81b664baaac15054730fbcb51e1b240de2/ivy/src/main/scala/sbt/internal/librarymanagement/IvyActions.scala#L123

Artifactory deals with the publishing (and republishing) of SNAPSHOTs using a strict rule on the order of publishing. The ;build.timestamp=... suffix is the alternative.

The strict rule is:

publish the main artefact (which has no classifier)
publish the POM/ivy.xml file
publish additional artefacts which have a classifier

I can't seem to get classifiers deployed at all so far, which is bothersome, and part of what I have a question into github about.

I'll probably create a pure maven/java project and see if it works then

francisdb commented 5 years ago

Just tried deploying a snapshot using the default publish and got this:

[error] stack trace is suppressed; run last publish for the full output
[error] (publish) java.io.IOException: PUT operation to URL https://maven.pkg.github.com/company/foo/bar/utils/3.24.0-SNAPSHOT/utils-3.24.0-SNAPSHOT.pom failed with status code 400: Bad Request

No mention of snapshots in the docs

In fact I have the same for normal packages:

java.io.IOException: PUT operation to URL https://maven.pkg.github.com/company/foo/bar/utils/utils/0.0.1/utils-0.0.1.pom failed with status code 400: Bad Request
[error] java.io.IOException: PUT operation to URL https://maven.pkg.github.com/company/foo/bar/utils/utils/0.0.1/utils-0.0.1.pom failed with status code 400: Bad Request
[error]     at org.apache.ivy.util.url.AbstractURLHandler.validatePutStatusCode(AbstractURLHandler.java:82)
[error]     at org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:264)
[error]     at org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
[error]     at org.apache.ivy.util.FileUtil.copy(FileUtil.java:150)

I wonder what the body of that 400 looks like

tovbinm commented 5 years ago

Are there any updates on this?

djspiewak commented 5 years ago

No updates yet! @lokkju's experiments are the latest, I think. I want to play around with it some more, but honestly it sounds like GitHub Packages still doesn't support hosting files which are part and parcel to a standard Maven distributable (such as signatures, documentation, and classifiers). It's hard to do anything practical with it as an artifact host until they correct these deficiencies.

francisdb commented 5 years ago

Some more info, testing with plain sbt 1.3.2 (no extra plugins)

You need to publish to the repository of your project: 

publishTo := Some("GitHub Package Registry" at "https://maven.pkg.github.com/[username]/[project]")

credentails should be defined like this

credentials += Credentials("GitHub Package Registry","maven.pkg.github.com","[username]","[token]")

Your token needs the following scopes: read:packages, repo and write:packages

but once you publish you end up with these proposed package coordinates (where organization="io.test" and name="test"):

<dependency>
  <groupId>com.github.[username]/[project]</groupId>
  <artifactId>io.test.test_2.13</artifactId>
  <version>0.1</version>
</dependency>

Snapshots can be uploaded but they probably have no special handling as you can't upload them twice

Pom upload works, but the rest fails:

[info] published test_2.13 to https://maven.pkg.github.com/[username]/[project]/io/test/test_2.13/0.0.3/test_2.13-0.0.3.pom [error] java.io.IOException: Access to URL https://maven.pkg.github.com/[username]/[project]/io/test/test_2.13/0.0.3/test_2.13-0.0.3.jar was refused by the server: Forbidden response=Error: "test_2.13-0.0.3.jar" in version 0.0.3 of "io.test.test_2.13" has already been published.

lokkju commented 5 years ago

@francisdb JAR upload fails because SBT uploads the files out of order; from what I can remember, it's a long standing bug in SBT (IvyActions.scala) where the publish function uses a map type instead of seq type. With the move to the Coursier dependency management system, this could self resolve.

My suggestion would be to override the upload process itself within this plugin, making sure to upload the pom first. Of course, this still doesn't help with classifiers, which I need support for.

francisdb commented 5 years ago

looks like on sbt 1.3.2 Coursier is not in use for uploading artifacts, only for downloading

lokkju commented 5 years ago

Also, I haven't had any response from GitHub on this issue

mkurz commented 5 years ago

Hmmm... I just had a very quick look. sbt 1.2.8 is still used here everyhwere: https://github.com/djspiewak/sbt-github-packages/blob/0839b97dea06100a202f2a105cbe83873784bf77/build.sbt#L28 https://github.com/djspiewak/sbt-github-packages/blob/0839b97dea06100a202f2a105cbe83873784bf77/src/sbt-test/sbtghpackages/publish/project/build.properties#L1 https://github.com/djspiewak/sbt-github-packages/blob/0839b97dea06100a202f2a105cbe83873784bf77/project/build.properties#L1

(Maybe missed some however)

I guess these need upgrades?

mkurz commented 5 years ago

Pull request: #2

mkurz commented 5 years ago

Coursier is only used by default starting with sbt 1.3.0

francisdb commented 5 years ago

my testing was with 1.3.2

mkurz commented 5 years ago

@francisdb Got it.

francisdb commented 5 years ago

Looks like Github has changed some things, was able to do a full upload today. (vanilla sbt 1.3.3)

[info]  published test_2.13 to https://maven.pkg.github.com/francisdb/foo/com/example/test_2.13/0.0.9/test_2.13-0.0.9.pom
[info]  published test_2.13 to https://maven.pkg.github.com/francisdb/foo/com/example/test_2.13/0.0.9/test_2.13-0.0.9.jar
[info]  published test_2.13 to https://maven.pkg.github.com/francisdb/foo/com/example/test_2.13/0.0.9/test_2.13-0.0.9-sources.jar
[info]  published test_2.13 to https://maven.pkg.github.com/francisdb/foo/com/example/test_2.13/0.0.9/test_2.13-0.0.9-javadoc.jar

The web ui is still confused about the sbt suffix

<dependency>
  <groupId>com.example.test_2</groupId>
  <artifactId>13</artifactId>
  <version>0.0.9</version>
</dependency>
djspiewak commented 4 years ago

This is exciting! I'm going to do some testing today and see if I can close this out. Very very exciting.

djspiewak commented 4 years ago

I can confirm that, with a few tweaks to the plugin to use @francisdb's experimental results, things are now working! The webui issues with cross-build suffixes do not appear to affect actual resolution, just the UI. So publication and resolution now work with the plugin. I haven't yet tested things with a private repository; my guess is that they aren't quite working yet, but things are good enough that I've published 0.1.0.

Thank you, everyone, for all your hard work and experimentation! The plugin is remarkably trivial thanks to all of you.

mkurz commented 4 years ago

@francisdb

Snapshots can be uploaded but they probably have no special handling as you can't upload them twice

You can upload them twice: You can delete a snapshot in the GitHub UI and then again upload a new snapshot (meaning same snapshot version). That worked for me. A bit cumbersome, but it works.