Archetype for java library (JavaLibPackaging?)

onsails commented 8 years ago

Our platform uses sbt-native-packager for deb packaging its core functionality. It also has an ability to load third-party or enterprise modules by specifying their fqcn's in server config. The problem is that JavaAppPackaging archetype assumes that packaged project have to be "started" via start script and that it also has a configuration. What would you say if we implement JavaLibPackaging which would only put jar's in /usr/lib/<appname> (appname – not libname) and send PullRequest to you?

onsails commented 8 years ago

Hmm, there is also a question: how can I modify default app_classpath? It seems that there is no way to add custom classpath into app start script – I want to add /usr/lib/actor/* to classpath.

muuki88 commented 8 years ago

This is a feature we have long been discussing and is wanted by a lot of users with different packaging systems (docker, rpm, debian). If I understood you correctly the main idea is to create a new archetype JavaLibPackaging which

only serves dependencies (no start scripts)
stores dependencies under, e.g for debian, /usr/lib

There are a few things we must take care of for this feature

How do we handle theses external dependencies in the original start script? (declare dependencies, runtime checking,.. ?)
How do we build this that we can adapt it for other packaging systems as debian (rpm, docker)
How should this archetype be used? -> create submodules for dependencies, generate multiple packages (e.g. dependency.deb and app.deb ) or for complete separate projects

Pull Requests are very welcome :)

muuki88 commented 8 years ago

You can add things to the classpath with the scriptClasspath setting.

There are two archetypes handling this problem.

onsails commented 8 years ago

How do we handle theses external dependencies in the original start script? (declare dependencies, runtime checking,.. ?)

For me declaring an option like javaLibraryDependencies := true which will add every jar from /usr/lib/actor/*/*.jar to classpath would work.

How do we build this that we can adapt it for other packaging systems as debian (rpm, docker)

I don't see any difference with rpm but docker is a big question. Right now we use docker in one of our enterprise deployment and we resolved problem in the following way:

package library in separate docker image and publish it in docker hub (as a private image)
give access to it to our customer
via Docker Compose we mount image with a library to /usr/lib/actor/lib-name
the original start script adds to classpath every jar from /usr/lib/actor/*/*.jar
Yay!

How should this archetype be used? -> create submodules for dependencies, generate multiple packages (e.g. dependency.deb and app.deb ) or for complete separate projects

We ended up creating complete separate project without dependency on the main project. Submodules (you mean sbt subprojects) for dependencies didn't work for us because we are not the only team who develop modules for our app. Third-party developers should be able to build libraries in their repositories too.

onsails commented 8 years ago

It is a blocker issue for us and until tomorrow we will develop some internal solution for deb packaging. We can make a pull request for javaLibraryDependencies in Debian := true and /usr/lib/actor/*/*.jar if we agree on that.

muuki88 commented 8 years ago

We ended up creating complete separate project without dependency on the main project

I think this is a good starting point as this is the most general way ( you can always integrate a separate project as an sbt-submodule). Also your solution with docker is the one that has come up in other discussions about this topic.

Regarding the implementation I think we shouldn't go for flags to set. Instead we need two AutoPlugins. One for the library part ( JavaLibArchetype) and another for the main project that uses external dependencies (e.g. JavaExternalDependenciesArchetype).

Because as you said we have to configure the scriptClasspath extensions. Even IMHO * wildcard classpaths are dangerous, because of security and classpath ordering, this is the most practical option here. So the configuration could go like this

// build.sbt in the lib project
name := "my-libraries"

enablePlugins(JavaLibArchetype)
javaLibraryPath in Debian := Some(s"/usr/lib/${packageName.value}")

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

// build.sbt in your main project
name := "my-project"

enablePlugins(JavaAppArchetype, JavaExternalDependenciesArchetype)
javaLibraryPath in Debian := Some(s"/usr/lib/my-libraries")

javaLibraryPath ( settingKey[Option[String]]) defines the path where the libraries are stored.
JavaExternalDependenciesArchetype uses this setting to extend the scriptClasspath setting appropriately. If javaLibraryPath is None, then nothing happens (e.g. if you use it for debian, but not for docker)

WDYT?

onsails commented 8 years ago

ok, I got it. We are on it, wait for pull request in a few days.

muuki88 commented 8 years ago

Awesome! Looking forward to it :) If you need any help, don't hesitate to ask. There is a developer guide that describes how to run tests and build the docs

muuki88 commented 8 years ago

Also see https://github.com/sbt/sbt-native-packager/pull/368

onsails commented 8 years ago

ok. A bit delayed here, still on my way.

muuki88 commented 8 years ago

Questions from gitter:

If library has its own dependencies, how it should “notify” project with JavaExternalDependenciesArchetype about them?

what should I do with dependency collisions? Let’s say we have two library installed: libA and libB. Both of them depend on cats-0.3.0 after installing debs/rpms we get /usr/lib/app-libraries/libA/org.spire-math.cats-0.3.0.jar and /usr/lib/app-libraries/libB/org.spire-math.cats-0.3.0.jar is it OK for JavaExternalDependenciesArchetype detect this collisions and add to classpath only the first one? if jar names are equal

another question: should I put libs in /usr/share and them symlink them to /usr/lib? with the second approach mapGenericFilesToLinux in LinuxPlugin will become a bit more complex… same for DebianPlugin and RpmPlugin. With the first approach lib will be able to have some assets, and implementation will be easier by " the second approach” I have meant putting files to /usr/lib/libname directly, lost that part somehow

muuki88 commented 8 years ago

To make sure I understood your questions correctly, I will try answer them by using this example setup:

my-app/
  - depend on the "my-libs"  project, but mark it as provided
    - "your.company" % "my-libs" % "0.3.13" % provided
  - standalone project
my-libs/
  - dependencies
    - "cats" % "0.3"
    - "logback" % "1.0"
  - standalone project

If library has its own dependencies, how it should “notify” project with JavaExternalDependenciesArchetype about them?

Both projects are separate, one carrying only the application code, assuming all dependencies are available in the library path. So you have either

check at build time that you have the latest version of my-libs
add rpm/debian requirements or pre-requesits hooks to check if this package is installed

what should I do with dependency collisions? Let’s say we have two library installed: libA and libB. Both of them depend on cats-0.3.0 after installing debs/rpms we get /usr/lib/app-libraries/libA/org.spire-math.cats-0.3.0.jar and /usr/lib/app-libraries/libB/org.spire-math.cats-0.3.0.jar is it OK for JavaExternalDependenciesArchetype detect this collisions and add to classpath only the first one? if jar names are equal

First of all, we shouldn't try to do any jar-collision-detection. This is way to complex and error prone. However I'm not sure I understand the problem correctly. Why should there be any collision? If I install my example project, I would have

/usr/share/my-app
  bin/my-app
  lib/my-app.jar
/usr/lib/my-libs
  cats-0.3.jar
  logback-1.0.jar

And my startscript will have a classpath like this -cp /usr/share/myapp/lib/my-jar,/usr/lib/my-libs/. I wonder if we even could add the full classpath if we depend on that library ( even though dependencies are marked as provided)

So if I install another app (my-app2), which is referencing my-libs then nothing changes. If I install another lib project (my-libs2), the jars should end up in another folder. No issue here as well. What am I missing?

another question: should I put libs in /usr/share and them symlink them to /usr/lib?

why that? I haven't looked into the linux mappings for quite some time.

rockjam commented 8 years ago

Currently working on this feature, I stuck with implementation. I can't get around Def.Initialize and dynamic reference error.

https://github.com/rockjam/sbt-native-packager/blob/97d5890bc2185a5f80b9f26de8ceb4e4a83ee5a2/src/main/scala/com/typesafe/sbt/packager/archetypes/DependenciesOps.scala#L20

Here is function that returns dependencies of JavaLib projects with absolute path, as they will appear in main package's classpath.

And here is usage of it. https://github.com/rockjam/sbt-native-packager/blob/97d5890bc2185a5f80b9f26de8ceb4e4a83ee5a2/src/main/scala/com/typesafe/sbt/packager/archetypes/JavaExternalDeps.scala#L75

I know it is wrong usage, but I also tried to solve it other way. Usually I get Illegal dynamic dependency or Illegal dynamic reference.

Can someone tell me, is there something fundamentally wrong in defenition of javaLibsAbsolutePath, or I should find some workaround with sbt?

muuki88 commented 8 years ago

I recommed not using the <<= operators and Def. utils as they are complicating things more than actually needed. So if you have your method:

// defintion
def javaLibsAbsolutePath: Def.Initialize[Def.Initialize[Task[Seq[String]]]] = 
  (sbt.Keys.buildDependencies, sbt.Keys.thisProjectRef, sbt.Keys.state) apply { (build, thisProject, stateTask) => ???

// call site
scriptClasspath <<= (javaLibsAbsolutePath) map { v =>
      v.value
},

I would recommend changing it to

// definition
def javaLibsAbsolutePath(build: BuildDependencies, thisProject: ProjectRef, stateTask: State): Seq[String] = {

}

// call site
scriptClasspath := javaLibsAbsolutePath(buildDependencies.value, thisProjectRef.value, state.value)

This approach has a few pros that I found very usefull

You can test your actually logic with unit tests
SBT specific stuff stays in the AutoPlugin (depending on tasks, settings)
All your task/setting dependencies are in one place

I will try to look a bit closer at your current working state and give feedback if I am able to.

rockjam commented 8 years ago

It's still work in progress, I will certainly make cleanup in code, and improve code reuse, when got working solution.

rockjam commented 8 years ago

Thanks for advice!

rockjam commented 8 years ago

Well, leaving all sbt magic in AutoPugin really helped to make code easier, but anyway I couldn't solve Illegal reference issue. Latest commit in branch issue-723 reflects current state.

muuki88 commented 8 years ago

I will try to run your code ASAP

rockjam commented 8 years ago

:+1: Thanks

muuki88 commented 8 years ago

@rockjam I think I found the solution. See https://github.com/sbt/sbt/issues/780 For this special case it seems we have to use Def. ( and I hope we can avoid it anywhere else ) This is the example from the issue:

Def.settingDyn {
   sourceDirectories.all( ScopeFilter(configurations = confFilter.value) )
}

rockjam commented 8 years ago

@muuki88 I got a question on project structure. How should it be described in main build file? This is how I "imagine" it:

Here is a project tree:

.
├── build.sbt
├── src
│   └── main
├── subpr1
│   ├── build.sbt
│   ├── src
├── subpr2
│   ├── build.sbt
│   ├── src
└── test

in build.sbt

enablePlugins(JavaExternalDepsPackaging)
name := "external-deps-test"
version := "0.1.0"

lazy val subproject1 = project.in(file("subpr1"))

lazy val subproject2 = project.in(file("subpr2"))

in subpr1/build.sbt

name := "subproject-1"

enablePlugins(JavaLibPackaging)
javaLibraryPath := Some(s"/usr/share/${name.value}/lib")

version := "0.1.0"

libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.4.2"

in subpr2/build.sbt

name := "subproject-2"

enablePlugins(JavaLibPackaging)
javaLibraryPath := Some(s"/usr/share/${name.value}/lib")

version := "0.1.0"

libraryDependencies += "org.typelevel" %% "cats" % "0.4.1"

muuki88 commented 8 years ago

@rockjam this looks exactly how I hope the API can look like.

muuki88 commented 8 years ago

@rockjam do you need more help on this?

ostronom commented 8 years ago

@muuki88 Hello. I am responsible for this task now :)

Could you be more specific on how the Def.settingDyn could be used against illegal dynamic reference here? Thank you.

muuki88 commented 8 years ago

I haven't used it myself :( Just found the sbt issue.

sbt / sbt-native-packager

Archetype for java library (JavaLibPackaging?) #723