search

palantir / sls-packaging

A set of Gradle plugins for creating SLS-compatible packages
Apache License 2.0
38 stars 74 forks source link
gradle octo-correct-managed

readme

Autorelease

SLS Distribution Gradle Plugins

Build Status Gradle Plugin Portal

A set of Gradle plugins that facilitate packaging projects for distributions conforming to Palantir's Service Layout Specification. This project was formerly known as gradle-java-distribution.

The Java Service and Asset plugins cannot both be applied to the same gradle project, and distributions from both are produced as a gzipped tar named [service-name]-[project-version].sls.tgz.

Java Service Distribution Gradle Plugin

Similar to the standard application plugin, this plugin helps package Java Gradle projects for easy distribution and execution. This distribution conforms with Palantir's SLS service layout conventions that attempt to split immutable files from mutable state and configuration.

In particular, this plugin packages a project into a common deployment structure with a simple start script, daemonizing script, and, a manifest describing the content of the package. The package will follow this structure:

[service-name]-[service-version]/
    deployment/
        manifest.yml                      # simple package manifest
    service/
        bin/
            [service-name]                # Bash start script
            [service-name].bat            # Windows start script
            init.sh                       # daemonizing script
            darwin-amd64/go-java-launcher # Native Java launcher binary (MacOS)
            linux-amd64/go-java-launcher  # Native Java launcher binary (Linux x86_64)
            linux-arm64/go-java-launcher  # Native Java launcher binary (Linux arm64)
            launcher-static.yml           # generated configuration for go-java-launcher
            launcher-check.yml            # generated configuration for check.sh go-java-launcher
        lib/
            [jars]
        monitoring/
            bin/
                check.sh                  # monitoring script
    var/                                  # application configuration and data

The service/bin/ directory contains both Gradle-generated launcher scripts ([service-name] and [service-name].bat) and go-java-launcher launcher binaries. See below for usage.

Asset Distribution Gradle Plugin

This plugin helps package static files and directories into a distribution that conforms with Palantir's SLS asset layout conventions. Asset distributions differ from service distributions in that they do not have a top-level service or var directory, and instead utilize a top-level asset directory that can contain arbitrary files. See below for usage.

Usage

Product dependencies

'Product dependencies' are declarative metadata about the products your product/asset requires in order to function. When you run ./gradlew distTar, your product dependencies are embedded in the resultant dist in the deployment/manifest.yml file.

Most of your product dependencies should be inferred automatically from on the libraries you depend on. Any one of these jars may contain an embedded 'recommended product dependency' in its MANIFEST.MF (embedded using the Recommended Product Dependencies Plugin).

However, you can also use the productDependency block to specify these manually (although this is no longer considered a best-practise). Please note: you can add further restrictions to existing constraints, but you can't broaden them:

distribution {
    productDependency {
        productGroup = "com.palantir.group"
        productName = "my-service"
        minimumVersion = "1.0.0"
        maximumVersion = "1.x.x"
        recommendedVersion = "1.2.1"
        optional = false
    }
}

sls-packaging also maintains a lockfile, product-dependencies.lock, which should be checked in to Git. This file is an accurate reflection of all the inferred and explicitly defined product dependencies. Run ./gradlew --write-locks or ./gradlew writeProductDependenciesLocks to update it. e.g.

# Run ./gradlew writeProductDependenciesLocks to regenerate this file
com.palantir.auth:auth-service (1.2.0, 1.6.x)
com.palantir.storage:storage-service (3.56.0, 3.x.x)
com.palantir.email:email-service (1.200.3, 2.x.x) optional
com.palantir.foo:foo-service ($projectVersion, 1.x.x)

The $projectVersion string is a placeholder that will appear if your repo publishes multiple services, and one of them depends on another. The actual manifest will contain a concrete version.

The suffix optional will be added for optional = true in the productDependency declaration. All dependencies are required by default.

It's possible to further restrict the acceptable version range for a dependency by declaring a tighter constraint in a productDependency block - this will be merged with any constraints detected from other jars. If all the constraints on a given product don't overlap, then an error will the thrown: Could not merge recommended product dependencies as their version ranges do not overlap.

It's also possible to explicitly ignore a dependency or mark it as optional if it comes as a recommendation from a jar:

distribution {
    productDependency {
        // ...
    }
    ignoredProductDependency('other-group3', 'other-service3')
    optionalProductDependency('other-group4', 'other-service4')
}

Dependencies marked as optional will appear with the optional suffix in the lockfile.

Accessing product dependencies

You can programmatically access the minimum product dependency version as follows:

def myDependency = getMinimumProductVersion('com.palantir.service:my-service')

More often though, you probably just want to get the minimum product dependencies as a gradle configuration that you can depend on from other projects. For this purpose, there is a configuration called productDependencies that is published from each SLS project.

You can then use this together with gradle-docker to inject your product dependencies into the docker-compose templating, for instance.

For example, given a dist project, :my-service, you can collect wire up docker :

// from another project
apply plugin: 'com.palantir.docker'
dependencies {
    docker project(path: ':my-service', configuration: 'productDependencies')
}

Container Images

You can specify container images that your product requires using the artifact declaration on the distribution extension. For example, if a container could be pulled from registry.example.io/foo/bar:v1.3.0, you could add the following to the distribution extension. The entry should contain the URI for the artifact in the uri field and should always contain 'oci' (the only currently supported type) in the type field:

distribution {
   artifact {
        type = 'oci'
        uri = 'registry.example.io/foo/bar:v1.3.0'
    }
}

The result will be embedded in the deployment/manifest.yml file. The file will look something like this:

{
  "manifest-version" : "1.0",
  "product-type" : "service.v1",
  "product-group" : "com.example.foo",
  "product-name" : "bar",
  "product-version" : "1.1.0",
  "extensions" : {
    "product-dependencies" : [ {
      "product-group" : "com.example",
      "product-name" : "dependency",
      "minimum-version" : "1.0.0",
      "recommended-version" : "1.0.0",
      "maximum-version" : "1.x.x",
      "optional" : false
    } ],
    "artifacts" : [ {
       "type": "oci",
       "uri": "registry.example.io/foo/bar:v1.1.0"
    } ]
  }
}

Schema versions

sls-packaging also maintains a lockfile, schema-versions.lock, which should be checked in to Git. This file is an accurate reflection of the schema versions specified in the manifest. The file can be used to easily determine what schema versions are supported by a particular version of the code. Run ./gradlew --write-locks or ./gradlew writeSchemaVersionLocks to update it.

---
comment: "Run ./gradlew writeSchemaVersionLocks to regenerate this file"
schemaMigrations:
- type: "online"
  from: 100
- type: "online"
  from: 101
version: 1

Packaging plugins

These plugins require at least Gradle 4.10.

Java Service Distribution plugin

Apply the plugin using standard Gradle convention:

plugins {
    id 'com.palantir.sls-java-service-distribution'
}

Additionally, declare the version of go-java-launcher to use:

# Add to 'versions.props'
com.palantir.launching:* = 1.18.0

A sample configuration for the Service plugin:

distribution {
    serviceName 'my-service'
    serviceGroup 'my.service.group'
    mainClass 'com.palantir.foo.bar.MyServiceMainClass'
    args 'server', 'var/conf/my-service.yml'
    env 'KEY1': 'value1', 'KEY2': 'value1'
    manifestExtensions 'KEY3': 'value2'
    productDependency {
        productGroup = "other-group"
        productName = "other-service"
        minimumVersion = "1.1.0"
        maximumVersion = "1.5.x"      // optional, defaults to "1.x.x" (same major version as minimumVersion)
        recommendedVersion = "1.3.0"  // optional
    }
}

And the complete list of configurable properties:

JVM Options

The list of JVM options passed to the Java processes launched through a package's start-up scripts is obtained by concatenating the following list of hard-coded required options and the list of options specified in distribution.defaultJvmOpts:

Hard-coded required JVM options:

The go-java-launcher and init.sh launchers additionally append the list of JVM options specified in the var/conf/launcher-custom.yml configuration file. Note that later options typically override earlier options (although this behavior is undefined and may be JVM-specific); this allows users to override the hard-coded options.

JDK Inclusion

The distribution extension provides a configuration point to add JDKs; internally, gradle-sls-docker discovers the appropriate JDKs used in the service's associated container image then configures these JDKs to be included in the dist. There are more details in gradle-sls-docker readme.

For each included JDK major version X, the launcher-static.yml run by go-java-launcher has the corresponding JAVA_X_HOME environment variable set to be a relative path to the JDK's location in the dist. The javaHome option is also set to the relative path in the same manner. There can only be one version of each JDK major version included in the dist.

Runtime environment variables

Environment variables can be configured through the env blocks of launcher-static.yml and launcher-custom.yml as described in configuration file. They are set by the launcher process before the Java process is executed.

Directories created at runtime

The plugin configures go-java-launcher to create the following directories before starting the service:

Additionally, the following directories are created in every SLS distribution created:

Asset Distribution plugin

Apply the plugin using standard Gradle convention:

plugins {
    id 'com.palantir.sls-asset-distribution'
}

A sample configuration for the Asset plugin:

distribution {
    serviceName 'my-assets'
    assets 'relative/path/to/assets', 'relocated/path/in/dist'
    assets 'another/path, 'another/relocated/path'
}

The complete list of configurable properties:

The example above, when applied to a project rooted at ~/project, would create a distribution with the following structure:

[service-name]-[service-version]/
    deployment/
        manifest.yml                      # simple package manifest
    asset/
        relocated/path/in/dist            # contents from `~/project/relative/path/to/assets/`
        another/relocated/path            # contents from `~/project/another/path`

Note that repeated calls to assets are processed in-order, and as such, it is possible to overwrite resources by specifying that a later invocation be relocated to a previously used destination's ancestor directory.

Packaging

To create a compressed, gzipped tar file of the distribution, run the distTar task. To create a compressed, gzipped tar file of the deployment metadata for the distribution, run the configTar task.

The plugins expose the tar file as an artifact in the sls configuration, making it easy to share the artifact between sibling Gradle projects. For example:

configurations { tarballs }

dependencies {
    tarballs project(path: ':other-project', configuration: 'sls')
}

As part of package creation, the Java Service plugin will additionally create three shell scripts:

Furthermore, the Java Service plugin will merge the entire contents of ${projectDir}/service and ${projectDir}/var into the package.

Tasks

Specific to the Java Service plugin:

Recommended Product Dependencies Plugin

This plugin allows API jars to declare the recommended product dependencies an SLS service distribution should take.

An example application of this plugin might look as follows:

apply plugin: 'java'
apply plugin: 'com.palantir.sls-recommended-dependencies'

recommendedProductDependencies {
    productDependency {
        productGroup = 'com.foo.bar.group'
        productName = 'product'
        minimumVersion = rootProject.version
        maximumVersion = "${rootProject.version.tokenize('.')[0].toInteger()}.x.x"
        recommendedVersion = rootProject.version
    }
}

The recommended product dependencies will be serialized into the jar manifest of the jar that the project produces. The SLS distribution and asset plugins will inspect the manifest of all jars in the server or asset and extract the recommended product dependencies.

Marking a product dependency as optional in recommendedProductDependencies is not currently supported.

License

This plugin is made available under the Apache 2.0 License.