Missing sbt internal dependency for project

[inkytonik]

In my project Kiama (https://github.com/inkytonik/kiama) I have a multi-project structure that includes a base project with macros that are used by other sub-projects. I am using the approach from the sbt reference manual (https://www.scala-sbt.org/1.x/docs/Macro-Projects.html, section Distribution) to set it up so that the macro sub-project is published as part of the main sub-project, rather than being published on its own.

I've been using Bloop via VSCode + Metals to develop Kiama for at least a couple of years with no problems, but unfortunately I can't build Kiama in Metals in the current version (Metals 1.93 which uses Bloop 1.4.3-23-550c6c0a). I've also tested the problem outside Metals using Bloop 1.4.4 and I see the same issue.

The issue seems to be that I use an internal dependency of the main project in the macro sub-project as described in the sbt reference manual. When that dependency is used, bloopInstall omits the dependency in the generated project JSON file. As you would expect, this means that the macro sub-project is not built first, so I get lots of spurious errors in Metals. Building the same project using sbt 1.3.13 outside VSCode works fine.

Rather than have you deal with Kiama and its complexity, I've cut the problem down to a simple project (simple.zip). There are two sub-projects util and core with core depending on util. If it's a regular dependency then everything works fine. If it's an internal dependency it fails, so the dependency is omitted from the generated core.json file and I see a compile error in VSCode on the use of the Util object from the main Core object.

[plokhotnyuk]

I use internal types of dependencies even for external dependencies (not just for sub-projects): compile-internal for macros or GraalVM stubs, runtime-internal for shaded runtime dependencies.

It would be great for having full support for them. But even clear instruction for possible workarounds that use provided dependency and patch .pom files during publishing would be helpful too.

[jvican]

Hey @inkytonik @plokhotnyuk thanks for chiming in. You can read this past thread on internal sbt configurations. We currently don't support them and I don't think this will change any time soon. In the thread, I recommend users using internal dependencies (which is quite an exotic feature), use sbt tasks to remove the dependencies from the pom files directly or to replicate the specifics of the configuration you're using. That should be a valid workaround for the time being.

[inkytonik]

Thanks @jvican for your response and all of the work by everyone on Bloop.

I'm not sure which thread you mean. Is the link missing from your comment?

I agree that internal dependencies are not commonly used, but they are used in real projects, so it would be great to have support for them. I'm not keen to rewrite my build (which has been mostly unchanged for years), re-test publishing, etc, just to get Bloop to work. For the moment, an easier workaround is to manually edit the project json file to add the missing dependency.

Also, as @plokhotnyuk notes, it would be good to have clear instructions, recipes for how to avoid internal dependencies, if that is the recommenced workaround. I looked around and couldn't find anything, particularly in the multi-project build section of the sbt manual.

Finally, I'm confused by your comment that you don't support internal dependencies. I've used Metals + Bloop for quite a few years on this project. The import from my sbt build worked perfectly well until fairly recently (not sure exactly when), so the current behaviour looks like a regression.

[jvican]

@inkytonik I forgot to link the thread out, here it is https://github.com/scalacenter/bloop/issues/1138

I agree that it's annoying that internal dependencies are not supported in sbt but that's expected to be like that for the time being, there's no plan to support them now. I don't know why your project was working before but it could have been because you were relying on a bug in the sbt exporter that would over-export dependencies. This bug was fixed in the latest releases.

Also, as @plokhotnyuk notes, it would be good to have clear instructions, recipes for how to avoid internal dependencies, if that is the recommenced workaround. I looked around and couldn't find anything, particularly in the multi-project build section of the sbt manual.

Yeah, I agree this would be ideal. I don't have the time to come up with those clear instructions but if someone comes up with them I'll happily put them in the sbt docs of our website. I think whomever wants to tackle this needs to use the Compile config and then find out a way to replicate the semantics of the internal dependency.

It would be great for having full support for them. But even clear instruction for possible workarounds that use provided dependency and patch .pom files during publishing would be helpful too.

@plokhotnyuk Yeah, I should just confirm that Provided does work pretty much out of the box but you'll need to find the sbt steps to remove the dependencies from the POM files. sbt has an API for hat.

[inkytonik]

Ok, thanks @jvican. If I manage to find time in the future to work out how to do the proper workaround, I'll try to contribute some instructions for general use. This project is on the back-burner now though, so I don't think I can do anything soon.

[inkytonik]

Actually, @jvican, one more suggestion if I may: is there a way for Bloop to warn if it's ignoring internal dependencies? In my case, it took quite a bit of investigation to even work out that this aspect was related. It would be very helpful if an "internal dependencies are not currently supported" message appeared in the logs.