Pull dependencies from the instance-level registry, push to the project-level registry

[L7R7]

I'd like to publish a library to Gitlab that has dependencies that must be pulled from the instance-level Gitlab repository. Do I have to configure the publish/pull URLs separately? If so, how? Or is it enough to configure the instance-level registry as resolver?

[L7R7]

After looking at the code, I tried it with

publishTo := Some( "gitlab-maven" at "https://my.gitlab.com/api/v4/projects/123/packages/maven"),
resolvers += "gitlab-maven" at "https://my.gitlab.com/api/v4/packages/maven",

But I get the following exception:

org.eclipse.aether.deployment.DeploymentException: Failed to deploy artifacts: Could not transfer artifact com.sample:foo-bar_2.13:pom:6.0.8 from/to gitlab-maven (https://my.gitlab.com/api/v4/packages/maven): PUT operation to URL https://my.gitlab.com/api/v4/packages/maven/com/sample/foo-bar_2.13/6.0.8/foo-bar_2.13-6.0.8.pom failed with status code 415: ; Response Body: {"error":"The provided content-type 'application/octet-stream' is not supported."}

Which looks weird to me, because it seems like it's trying to push to the instance level registry instead of the project level one

[listba]

Are you trying to publish from within gitlab ci or from your local machine?

If you are trying to publish from within a gitlab ci job, so long as you have the proper permissions and are trying to publish to the project the CI job is running under the only thing you should need is to include the plugin. if you are trying to publish this from outside gitlab ci you need to set the following accordingly

import com.gilcloud.sbt.gitlab.{GitlabCredentials,GitlabPlugin}

GitlabPlugin.autoImport.gitlabGroupId     :=  Some(12345)
GitlabPlugin.autoImport.gitlabProjectId   :=  Some(12345)
GitlabPlugin.autoImport.gitlabDomain      :=  "my-gitlab-host.com"
GitlabPlugin.autoImport.gitlabCredentials :=  Some(GitlabCredentials("Private-Token","<API-KEY>"))

// Alternatively for credential management 
// ideal for pulling artifacts locally and keeping tokens out of your source control
// see below for sample .credentials file
credentials += Credentials(Path.userHome / ".sbt" / ".credentials.gitlab"),

either way you shouldn't need to set the publishTo manually the plugin handles that for you.

Given the error message it looks like you might be using the sbt-aether-deploy plugin. I dont know anything about that plugin but after taking a quick glance I would not be too surprised if there is some conflict between the two.

[L7R7]

@listba Thanks for the answer! I am trying to publish from within gitlab ci.

I forgot to mention that I need to add an additional maven resolver for the dependencies (we have a proxy for maven central where we pull external dependencies).
I got it to work with pulling dependencies correctly (internal ones from Gitlab, external ones via the proxy), and I got it working with pushing to gitlab registry but without pulling dependencies from Gitlab registry (only from the proxy). When I try to do both at the same time, I get the error mentioned above.

I don't use sbt-aether-deploy, at least not directly. I will have another look though, this might be a good hint

[hedefalk]

I got it to work with pulling dependencies correctly (internal ones from Gitlab, external ones via the proxy),

@L7R7 Did you manage to get that working with a normal mvn resolver like the one above?

resolvers += "gitlab-maven" at "https://my.gitlab.com/api/v4/packages/maven",

I got the feeling that gitlab demands custom headers that this plugin works with but that means having to disable coursier?

[L7R7]

@hedefalk I'm not sure I understand your question. Pulling dependencies works "out of the box", with dependencies from Gitlab and our Nexus mirror. As soon as I want to publish, things break with the error message described above.

Coursier is disabled, as described in the docs

[L7R7]

I think I might have found a solution now. At least I'm much closer now. I originally tried it in a multi-project build which is a bit more involved than it should. I tried it with a simple sbt project and now almost everything works out of the box, like @listba said above.

I had to provide the resolver explicitly though (I'm ok with that).

Now building and publishing works as expected, but only in the CI pipeline. It doesn't work locally. I tried what is described in the readme and in #21: I have a .credentials.gitlab file with a valid API Token (double checked it via curl) and the gitlab host (example.gitlab.com, without https:// or anything, and without the suffix for api/v4/packages/...).
The file is at the correct location (There would be a warning if the file was missing), but apparently it is not used when it tries to get the dependencies from gitlab. It does try to get it, the URL I see in the logs is correct, but it's not successful.

I played around with a couple of possibilities and found out that it works when I invoke it like this: CI_SERVER_HOST=gitlab.host.com sbt compile. This seems very strange to me. It picks up the secrets from the file I provided, but not the host URL? What am I missing here?

[listba]

Would you be willing/able to share your build.sbt file in here? Hard to help without knowing what you are doing. Also make sure your credential file is formatted correctly.

realm=gitlab
host=my-git-lab-host
user=Private-Token
password=<API-KEY>

[L7R7]

Sure, here's the build.sbt file I'm using:

ThisBuild / useCoursier := false

lazy val root = (project in file("."))
  .settings(
    name := "sbt-gitlab-test",
    credentials += Credentials(Path.userHome / ".sbt" / ".credentials.gitlab"),
    libraryDependencies ++= Vector(
      "foo.bar" % "baz" % "1.2.3",
    ),
    resolvers += ("gitlab-maven" at "https://my.gitlab.com/api/v4/packages/maven"),
    resolvers += "nexus-mirror" at "https://nexus.foo.com",
  )

I tried to simplify the project as much as I can, so I have one dependency that comes from gitlab, the sbt-gitlab plugin is the only one that's enabled.

My credentials file looks like this:

realm=gitlab
host=my.gitlab.com
user=Private-Token
password=...

which looks good to me (I wasn't sure about the host field: Should I include https://... or anything in there?) The token works fine, and it reads the file correctly (if I set CI_SERVER_HOST, it pulls the dependencies just fine so in that case it uses the token to authenticate)

[L7R7]

Update: it works locally when I set:

GitlabPlugin.autoImport.gitlabDomain :=  "my.gitlab.com"

But with that, it fails in CI when publishing the package:

[error] (publish) java.io.IOException: PUT operation to URL https://my.gitlab.com/api/v4/packages/maven/sbt-gitlab-test/sbt-gitlab-test_2.12/0.1.0-SNAPSHOT/sbt-gitlab-test_2.12-0.1.0-SNAPSHOT.pom failed with status code 415: ; Response Body: {"error":"The provided content-type 'application/octet-stream' is not supported."}

full stacktrace

``` error] java.io.IOException: PUT operation to URL https://my.gitlab.com/api/v4/packages/maven/sbt-gitlab-test/sbt-gitlab-test_2.12/0.1.0-SNAPSHOT/sbt-gitlab-test_2.12-0.1.0-SNAPSHOT.pom failed with status code 415: ; Response Body: {"error":"The provided content-type 'application/octet-stream' is not supported."} [error] at org.apache.ivy.util.url.AbstractURLHandler.validatePutStatusCode(AbstractURLHandler.java:82) [error] at sbt.internal.librarymanagement.ivyint.GigahorseUrlHandler.validatePutStatusCode(GigahorseUrlHandler.scala:262) [error] at sbt.internal.librarymanagement.ivyint.GigahorseUrlHandler.upload(GigahorseUrlHandler.scala:198) [error] at org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82) [error] at org.apache.ivy.util.FileUtil.copy(FileUtil.java:150) [error] at org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:84) [error] at sbt.internal.librarymanagement.ConvertResolver$LocalIfFileRepo.put(ConvertResolver.scala:368) [error] at org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130) [error] at sbt.internal.librarymanagement.ConvertResolver$ChecksumFriendlyURLResolver.put(ConvertResolver.scala:118) [error] at sbt.internal.librarymanagement.ConvertResolver$ChecksumFriendlyURLResolver.put$(ConvertResolver.scala:105) [error] at sbt.internal.librarymanagement.ConvertResolver$$anonfun$defaultConvert$lzycompute$1$PluginCapableResolver$1.put(ConvertResolver.scala:165) [error] at org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:216) [error] at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$5(IvyActions.scala:504) [error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) [error] at scala.util.Try$.apply(Try.scala:213) [error] at sbt.internal.librarymanagement.IvyUtil$.retryWithBackoff(IvyUtil.scala:22) [error] at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$4(IvyActions.scala:503) [error] at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$4$adapted(IvyActions.scala:501) [error] at scala.collection.Iterator.foreach(Iterator.scala:943) [error] at scala.collection.Iterator.foreach$(Iterator.scala:943) [error] at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) [error] at scala.collection.IterableLike.foreach(IterableLike.scala:74) [error] at scala.collection.IterableLike.foreach$(IterableLike.scala:73) [error] at scala.collection.AbstractIterable.foreach(Iterable.scala:56) [error] at sbt.internal.librarymanagement.IvyActions$.publish(IvyActions.scala:501) [error] at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$3(IvyActions.scala:143) [error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) [error] at sbt.internal.librarymanagement.IvyActions$.withChecksums(IvyActions.scala:157) [error] at sbt.internal.librarymanagement.IvyActions$.withChecksums(IvyActions.scala:150) [error] at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$1(IvyActions.scala:143) [error] at sbt.internal.librarymanagement.IvyActions$.$anonfun$publish$1$adapted(IvyActions.scala:133) [error] at sbt.internal.librarymanagement.IvySbt$Module.$anonfun$withModule$1(Ivy.scala:251) [error] at sbt.internal.librarymanagement.IvySbt.$anonfun$withIvy$1(Ivy.scala:215) [error] at sbt.internal.librarymanagement.IvySbt.sbt$internal$librarymanagement$IvySbt$$action$1(Ivy.scala:77) [error] at sbt.internal.librarymanagement.IvySbt$$anon$1.call(Ivy.scala:87) [error] at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:113) [error] at xsbt.boot.Locks$GlobalLock.withChannelRetries$1(Locks.scala:91) [error] at xsbt.boot.Locks$GlobalLock.$anonfun$withFileLock$1(Locks.scala:119) [error] at xsbt.boot.Using$.withResource(Using.scala:12) [error] at xsbt.boot.Using$.apply(Using.scala:9) [error] at xsbt.boot.Locks$GlobalLock.withFileLock(Locks.scala:119) [error] at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:71) [error] at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:59) [error] at xsbt.boot.Locks$.apply0(Locks.scala:47) [error] at xsbt.boot.Locks$.apply(Locks.scala:36) [error] at sbt.internal.librarymanagement.IvySbt.withDefaultLogger(Ivy.scala:87) [error] at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:209) [error] at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:206) [error] at sbt.internal.librarymanagement.IvySbt$Module.withModule(Ivy.scala:250) [error] at sbt.internal.librarymanagement.IvyActions$.publish(IvyActions.scala:133) [error] at sbt.Classpaths$.$anonfun$publishTask$4(Defaults.scala:3544) [error] at sbt.Classpaths$.$anonfun$publishTask$4$adapted(Defaults.scala:3537) [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49) [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62) [error] at sbt.std.Transform$$anon$4.work(Transform.scala:68) [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:282) [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23) [error] at sbt.Execute.work(Execute.scala:291) [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:282) [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:64) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [error] at java.base/java.lang.Thread.run(Thread.java:834) [error] (publish) java.io.IOException: PUT operation to URL https://my.gitlab.com/api/v4/packages/maven/sbt-gitlab-test/sbt-gitlab-test_2.12/0.1.0-SNAPSHOT/sbt-gitlab-test_2.12-0.1.0-SNAPSHOT.pom failed with status code 415: ; Response Body: {"error":"The provided content-type 'application/octet-stream' is not supported."} ```

[listba]

Ah ok, so I think the problem lies here

resolvers += ("gitlab-maven" at "https://my.gitlab.com/api/v4/packages/maven"),

I don't know the internals of sbt very well, but the plugin uses gitlab-maven as the key for the publishTo setting, and I suspect the issue lies in you adding a resolver with the same key, likely overwriting the publish location.

tldr: try changing that to something unique like gitlab-maven-pull. If that fixes it ill try to see if I can get that changed to something more specific like gilcloud-sbt-gitlab-maven so its less likely to collide with user resolvers

[L7R7]

Oh, thank you very much! Your explanations totally make sense. Renaming the resolver works, at least for the simple project. I'll test it tomorrow with the actual one, but I'm pretty sure it will work.

So the learning is that I have to set the gitlab domain explicitly. I can live with that, though. It's just one line and it won't ever change for my use cases.

[L7R7]

As expected, it works now also for proper "real world" projects. Thanks again for helping!