= Spring Batch Lightmin
:toc: left
:imagesdir: /spring-batch-lightmin-documentation/src/main/doc/images
== About
Spring Batch Lightmin is a client/server batch and scheduling platform for the Spring Boot stack.
The batch stack based on Spring Batch.
== Architecture
image::lightmin_architecture.001.jpeg[]
== Documentation
The full documentation can be found at
== Samples
All samples for Spring Batch Lightmin can be found at https://github.com/tuxdevelop/spring-batch-lightmin-samples[Sample Applications]
== Getting Started
[IMPORTANT]
Spring 5.1.x set the the property spring.main.allow-bean-definition-overriding to false as a default.
In order to use the current version of Spring Batch Lightmin, this property has to be set to true.
This issue can hopefully be solved in the upcoming releases.
The following section describes a quick start guide to setup the client server architecture.
If your want to get an overview of the configuration options, please read the documentation.
The getting started guide for the Embedded Mode can be found in the section Embedded Lightmin.
=== Spring Batch Lightmin BOM
Add the Spring Batch Lightmin BOM to the dependency management section of your project.
[source,xml]
org.tuxdevelop
spring-batch-lightmin-bom
${spring-batch-lightmin.version}
pom
import
=== Setting up the Server
The server is the component which is used to configure clients and schedule jobs.
[IMPORTANT]
The server can be operated in standalone and cluster mode.
The standalone mode does not support any synchronisation between multiple cluster instances.
Also the server cluster scheduler are working totally independent from each other in standalone mode.
The server itself is also a Spring Boot application.
==== Maven Dependencies
===== Standalone
The following dependency has to added to your project.
[source,xml]
org.tuxdevelop
spring-batch-lightmin-server-standalone
${spring-batch-lightmin.version}
===== Cluster
The project currently provides one cluster implementation based on Infinispan.
====== Infinispan
[source,xml]
org.tuxdevelop
spring-batch-lightmin-server-cluster-infinispan
${spring-batch-lightmin.version}
==== Enable the Server
In order to enable the server, the annotation @EnableLightminServer has to added to one of the @Configuration classes.
[source,java]
@EnableLightminServer
@SpringBootApplication
public class LightminServer {
}
==== Enable the Cluster
If you added the cluster dependencies the following actions have to be done:
===== Infinispan
In order to enable the server, the annotation @EnableLightminServer has to added to one of the @Configuration classes.
[source,java]
@EnableLightminServer
@EnableServerClusterInfinispan
@SpringBootApplication
public class LightminServer {
}
The Infinispan cluster configuration will provide the following clustered features:
- Synchronized poller for server scheduler and clean up jobs
- LightminServerLockManager implenentation for Infinispan
- Distributed Repositories for
JobExecutionEventRepository
JournalRepository
** LightminApplicationRepository
If the following properties are set to true, additional distributed repositories will be provided:
[source,yaml]
spring:
batch:
lightmin:
server:
cluster:
infinispan:
repository:
init-scheduler-configuration-repository: true # default false <1>
init-scheduler-execution-repository: true # default false <2>
<1> if true, a distributed _SchedulerConfigurationRepository_ will be provided
<2> if true, a distributed _SchedulerExecutionRepository_ will be provided
[IMPORTANT]
====
It is highly recommended to use persistent repositories for _SchedulerConfigurationRepository_ and _SchedulerExecutionRepository_ like the _JDBC_ repositories.
The distributed repositories will contain the stored values as long as one cluster member is up and running.
====
====== Infinispan Global Configuration
In order to use the infinispan cluster, a _GlobalConfiguration_ has to be provided.
The documentation of _Infinispan_ and _jGroups_ will give you insides how to configure it for your need.
[source,java]
----
@Configuration
public class MyConfiguration {
@Bean
public GlobalConfiguration globalConfiguration() {
return new GlobalConfigurationBuilder()
// Configure the global configuration
.build();
}
}
----
==== Server Scheduler
Since lightmin version _2.1.1_ the server provides a scheduler to trigger jobs at the clients.
In order to provide this feature, the scheduler configurations and executions have to be stored.
The _SchedulerConfigurationRepository_ and _SchedulerExecutionRepository_ are responsible for the doing the job.
Per dafault, the _standalone server_ will configure inmemory repositories, which are cleared after each restart.
===== Configuration Properties
The behavior of the _Server Scheduler_ can be changed via application properties.
[source,yaml]
----
spring:
batch:
lightmin:
server:
scheduler:
enabled: true # default true <1>
create-new-executions-on-failure: false # default false <2>
create-new-executions-on-lost: false # default false <3>
fail-on-instance-execution-count: true # default true <4>
poller-period: 1000 # default 1000 <5>
poller-period-retry: 1000 # default 1000 <6>
repository:
delete-poller-period: PT10m # default 10 minutes <7>
delete-failed: true # default true <8>
delete-finished: true # default true <9>
delete-lost: true # default true <10>
keep-failed: PT24h # default 24 hours <11>
keep-finished: PT24h # default 24 hours <12>
keep-lost: PT24h # default 24 hours <13>
----
<1> If set to false, no schedulers will be executed
<2> If set to true, a new executions based on the configuration will be created on failure.
<3> If set to true, a new executions based on the configuration will be created on lost after configured retries failed.
<4> If true, the execution will fail if not enough instances are available.
<5> Time in millis to fetch new executions to be executed by the scheduler
<6> Time in millis to fetch executions to be retried by the scheduler
<7> Duration of the clean up poller for the repositories
<8> If set to true, the failed executions will be deleted automatically.
<9> If set to true, the finished executions will be deleted automatically.
<10> If set to true, the lost executions will be deleted automatically.
<11> Duration how long failed executions should be kept before getting deleted automatically.
<12> Duration how long finished executions should be kept before getting deleted automatically.
<13> Duration how long lost executions should be kept before getting deleted automatically.
[IMPORTANT]
====
The properties
create-new-executions-on-failure: false
create-new-executions-on-lost: false
fail-on-instance-execution-count: true
are global properties.
There is the possibility that future releases will support this configuration option per execution.
====
===== Server Scheduler Jdbc Repositories
The database schema for the repositories are available in the dependency at _org.tuxdevelop.spring.batch.lightmin.server.scheduler.repository_
====== Maven Dependency
[source,xml]
----
org.tuxdevelop
spring-batch-lightmin-server-scheduler-repository-jdbc
${spring-batch-lightmin.version}
----
====== Enable the JDBC Repository
[source,java]
----
@EnableLightminServer
@EnableServerSchedulerJdbcRepository
@SpringBootApplication
public class MyApplication {
...
}
----
==== Service Discovery
Depending on the clients, the server is also able to find the clients via service discovery.
To enable the feature, a _DiscoveryClient_ implementation has to added and configured(e.g. Consul, Eureka, etc...) and the following property has to set to true
[source,yaml]
----
spring::
batch:
lightmin:
server:
discovery-enabled: true
----
==== Server behind a Proxy
The server frontend uses redirects and the _HOST_ header is taken to create the redirect urls.
This is the default behaviour of Spring MVC.
If the server frontend is running behind a proxy, the implementation of the server takes care, that the _X-FORWARD-PREFIX_ header is used as well.
If the proxy cannot pass the _HOST_ header or you do not want to change the defaults, e.g. _Zuul_, the following property can force the server to use the _X-FORWARDED-HOST_ header to build the redirect urls.
[source,yaml]
----
spring:
batch:
lightmin:
server:
use-x-forwarded-headers: true
----
=== Setting up a Client
The client applications are responsible to provide _Spring Batch Job_ definitions.
The _Spring Batch Lightmin_ client framework provides all the configurations to set up _Spring Batch_ and the communication with the server.
==== Step one - Client type
The type of the client decides how the registration to the server should be done.
Currently to types are supported, classic and via service discovery.
===== Classic
The classic client has to know where the servers are located and will send a registration request after the start up.
[source,xml]
-----
org.tuxdevelop
spring-batch-lightmin-client-classic
${spring-batch-lightmin.version}
-----
The dependency above will provide everything which is required for the classic client.
The annotation _@EnableLightminClientClassic_ has to added to one of the configuration classes.
[source,java]
----
@SpringBootApplication
@EnableLightminClientClassic
public class ClientApplication {
public static void main(final String[] args) {
SpringApplication.run(ClientApplication.class, args);
}
}
----
The following configuration properties have to be present
[source,yaml]
----
spring:
application:
name: my-client-application <1>
batch:
lightmin:
client:
classic:
server:
url: http://myserver1.domain:8180, http://myserver2.domain:8180 <2>
----
<1> The _spring.application.name_ is used to identify a client and handle a cluster of the instances.
<2> The _url_ property is a list of server to which the registration request should be send.
===== Service Discovery
_Spring Batch Lightmin_ provides two implementations for the discovery client type.
Both implementations add a tags to the underlying service discovery technology, so the server could identify lightmin clients.
====== Consul Client
The following dependency has to added for the consul client
[source,xml]
-----
org.tuxdevelop
spring-batch-lightmin-client-discovery-consul
${spring-batch-lightmin.version}
-----
The annotation _@EnableLightminClientConsul_ enables the fully integration with Consul.
[source,java]
----
@SpringBootApplication
@EnableLightminClientConsul
public class ClientApplication {
public static void main(final String[] args) {
SpringApplication.run(ClientApplication.class, args);
}
}
----
The configuration options for Consul can be found in the _Spring Cloud Consul_ documentation.
The client is sending per default events to the server, in order to find the server via service discovery, the following property has to be set.
[source,yaml]
----
spring:
batch:
lightmin:
client:
discovery:
server-discovery-name: lightmin-server <1>
----
<1> The service discovery name of the server.
====== Eureka Client
The following dependency has to added for the eureka client
[source,xml]
-----
org.tuxdevelop
spring-batch-lightmin-client-discovery-eureka
${spring-batch-lightmin.version}
-----
The annotation _@EnableLightminClientEureka_ enables the fully integration with Eureka.
[source,java]
----
@SpringBootApplication
@EnableLightminClientEureka
public class ClientApplication {
public static void main(final String[] args) {
SpringApplication.run(ClientApplication.class, args);
}
}
----
The configuration options for Eureka can be found in the _Spring Cloud Netflix_ documentation.
The client is sending per default events to the server, in order to find the server via service discovery, the following property has to be set.
[source,yaml]
----
spring:
batch:
lightmin:
client:
discovery:
server-discovery-name: lightmin-server <1>
----
<1> The service discovery name of the server.
==== Step two - The Configuration Repository
The _Configuration Repository_ is the component which stores the scheduler and listener configurations of the lightmin clients.
This configurations are loaded at start time and can be managed with the server frontend or API calls.
_Spring Batch Lightmin_ provides three implementation of the repository
* *map* - In memory repository, all changes will be gone after a restart.
* *jdbc* - The client fetches and stores the configurations in a database.
* *remote* - The client fetches and stores the configurations via API calls to a repository server.
===== Map Repository
[source,xml]
----
org.tuxdevelop
spring-batch-lightmin-repository-map
${spring-batch-lightmin.version}
----
The annotation _@EnableLightminMapConfigurationRepository_ enables the in memory repository.
[source,java]
----
@SpringBootApplication
@EnableLightminClientConsul
@EnableLightminMapConfigurationRepository
public class ClientApplication {
public static void main(final String[] args) {
SpringApplication.run(ClientApplication.class, args);
}
}
----
===== Jdbc Repository
[source,xml]
----
org.tuxdevelop
spring-batch-lightmin-repository-jdbc
${spring-batch-lightmin.version}
----
The annotation _@EnableLightminJdbcConfigurationRepository_ enables the jdbc repository.
[source,java]
----
@SpringBootApplication
@EnableLightminClientConsul
@EnableLightminJdbcConfigurationRepository
public class ClientApplication {
public static void main(final String[] args) {
SpringApplication.run(ClientApplication.class, args);
}
}
----
The jdbc repository requires a configured datasource bean with the name _dataSource_.
If the project configuration requires a specific datasource for the lightmin repository, the bean name can set via configuration property.
More configuration options can be found in the documentation.
[source,yaml]
----
spring:
batch:
lightmin:
repository:
jdbc:
data-source-name: myDataSource <1>
----
<1> Overriding the default datasource name.
The database schema ddl and drop scripts for various databases are located in the dependency above under the path:
[source,yaml]
----
org/tuxdevelop/spring/batch/lightmin/repository
----
===== Remote Repository
[source,xml]
----
org.tuxdevelop
spring-batch-lightmin-repository-remote
${spring-batch-lightmin.version}
----
The annotation _@EnableLightminRemoteConfigurationRepository_ enables the remote repository.
[source,java]
----
@SpringBootApplication
@EnableLightminClientConsul
@EnableLightminRemoteConfigurationRepository
public class ClientApplication {
public static void main(final String[] args) {
SpringApplication.run(ClientApplication.class, args);
}
}
----
The remote repository can be located via url and service discovery.
For the url way, the following properties has to be set:
[source,yaml]
----
spring:
batch:
lightmin:
repository:
remote:
server-url: http://my-server.domain:8280 <1>
----
<1> The url to the remote repository server
For the service discovery approach, the following properties has to set and a _DiscoveryClient_ bean has to be present.
[source,yaml]
----
spring:
batch:
lightmin:
repository:
remote:
discover-remote-repository: true <1>
server-discovery-name: remoteRepositorySever <2>
----
<1> Enables the discovery feature
<2> The discovery name of the remote repository server
Further configuration options can be found in the documentation.
==== Step three - Configure Spring Batch
The client configurations are enabling the _Spring Batch_ stack as well.
_Spring Batch_ itself has to have a configured _JobRepository_.
This _JobRepository_ can be in memory via map or jdbc.
The configuration of the _JobRepository_ can be done via properties, so _Spring Batch Lightmin_ knows what to configure.
===== Map JobRepository
For the map repository, the following configuration is enough:
[source,yaml]
----
spring:
batch:
lightmin:
batch:
repository-type: map
----
===== Jdbc Repository
For the jdbc repository, the following configuration is enough:
[source,yaml]
----
spring:
batch:
lightmin:
batch:
repository-type: jdbc
----
Properties like dataSource name, table prefix etc. can be overridden as well.
Please check the documentation for more details.
==== Clients in Containers
If a client runs inside a container like _Docker_, the dns name of the host systems has to be transferred to server on registration time.
For this use case, a property is available.
[source,yaml]
----
spring:
batch:
lightmin:
client:
hostname: FQDN of the host
----
==== Lightmin Client and JPA
If your clients uses JPA, the following annotation is required to initialize the _JPA_ specific _Lightmin BatchConfigurer_.
[source,java]
----
@Configuration
@EnableLightminBatchJpa <1>
public class MyConfiguration {
}
----
<1> @EnableLightminBatchJpa configures the JPA specific BatchConfigurer
== Remote Repository Server
The _Remote Repository Server_ is a _Spring Boot_ application which provides a REST API for clients.
The server itself needs a job configuration repository itself.
The server supports map and jdbc.
=== Maven
For the Jdbc repository, the following dependencies have to be added.
[source,xml]
----
org.tuxdevelop
spring-batch-lightmin-repository-server
${spring-batch-lightmin.version}
org.tuxdevelop
spring-batch-lightmin-repository-jdbc
${spring-batch-lightmin.version}
----
=== Configuration
The configuration of the used repository can be found in the client section.
The annotation _@EnableLightminRepositoryServer_ enables the server and the corresponding annotation the _job configuration repository_.
[source,java]
----
@SpringBootApplication
@EnableLightminRepositoryServer
@EnableLightminJdbcConfigurationRepository
public class RepositoryServerApplication {
public static void main(final String[] args) {
SpringApplication.run(RepositoryServerApplication.class, args);
}
}
----
== Embedded Lightmin
If the client server architecture does not fit the requirements, _Spring Batch Lightmin_ also provides am embedded mode, which ships the client and server in one package.
=== Maven
[source,xml]
----
org.tuxdevelop
spring-batch-lightmin-embedded
${spring-batch-lightmin.version}
----
=== Configuration
In this case, a specific client does not have to be configured.
The _Job Configuration Repository_ and _Spring Batch_ have to be configured like for a regular client.
[source,java]
----
@SpringBootApplication
@EnableLightminEmbedded
@EnableLightminMapConfigurationRepository
public class EmbeddedLightminApplication {
public static void main(final String[] args) {
SpringApplication.run(EmbeddedLightminApplication.class, args);
}
}
----
[source,yaml]
----
spring:
batch:
lightmin:
batch:
repository-type: map
----
== Spring Batch Lightmin Metrics
Lightmin Metrics introduces some custom metrics for the Lightmin-Platform.
It uses Micrometer to collect metrics.
The metrics are available for client and server applications.
=== Enabling Client Metrics
In order to collect metrics of a _Lightmin_ client application, the following property has to be to _true_.
[source,yaml]
----
spring:
batch:
lightmin:
client:
metrics-enabled: true # default true
----
[IMPORTANT]
====
If you want to use runtime information of a client on a server, the following property has to be set to true.
The information send will be transformed to metrics on the server, if the sever metrics feature is enabled.
====
[source,yaml]
----
spring:
batch:
lightmin:
client:
publish-job-events: true # default true
----
=== Enabling Server Metrics
In order to collect metrics of a _Lightmin_ server , the following property has to be set to _true_.
[source,yaml]
----
spring:
batch:
lightmin:
server:
metrics-enabled: true # default true
----
=== Exposing Metrics via actuator
In order to expose the collected metrics, a _Micrometer_ registry dependencies has to be added to the client and the server.
For prometheus, the following dependencies has to be added
[source,xml]
----
io.micrometer
micrometer-registry-prometheus
${micrometer.version}
----
[IMPORTANT]
====
The lightmin dependencies do not provide any registries out of the box
====
=== Modifying Lighmin Metrics
Metrics will be published over prometheus actuator and can be modified by adding MeterFilter into the metrics configuration.
Refer to Micrometer Documentation for more information.
[source,java]
----
@Configuration
public class Config{
final static String name = "EXAMPLE";
@Bean
public MeterRegistryCustomizer
metricsCommonTags() {
return registry -> {
if (name != null) {
registry.config().meterFilter(
new MeterFilter() {
@Override
public DistributionStatisticConfig configure(Meter.Id id, DistributionStatisticConfig config) {
if (id.getName().startsWith("lightmin")) {
// Example Filter - Activating histogram on all lightmin metrics
return DistributionStatisticConfig.builder().percentilesHistogram(true)
.percentiles(0.95)
.build()
.merge(config);
}
return config;
}
})
.commonTags("example_tag", name);
}
};
}
}
----
=== Grafana Dashboards
Lightmin Metrics introduces prebuild Grafana Dashboards for server and client.
The json files are available in the resources folder of the lightmin-metrics project.
In order to use the plugin, please install the following Plugins:
* Statusmap by Flant JSC
==== Lightmin Dashboard
The _Spring Batch Lightmin_ Dashboard provides information about all Lightmin clients which are configured as target in Prometheus.
The following variables are available:
* Service - taken from the tag _APPLICATION_NAME_ (Lightmin Clients should be listed here)
* Job Name - Name of all jobs, or all jobs of the selected _Services_
* Job Count Interval - Time period for the overall _SUCCESSFUL_ and _FAILED_ overview
image::grafana_all.png[]
If a specific Job Name is selected, the details about the job and step execution will be displayed.
image::grafana_job.png[]
==== Lightmin Server Dashboard
The Server Dashboard provides an overview of all Job Executions, which are send from the clients to the server.
The Lightmin server(s) should be configured as target in Prometheus.
image::grafana_server_all.png[]
[IMPORTANT]
====
The boards require a Prometheus datasource.
====
== Server User Interface
=== Applications
The start page of the _SpringBatchLightmin_ shows all register applications.
The status icon shows the current health status of the application.
image::applications.png[]
=== Application Instance Dashboard
The application dashboard is the entry point to the monitoring and administration of a client application instance.
The overview shows the important endpoints, all known _Spring Batch Jobs_ and configured external links of the client application.
image::dashboard.png[]
=== Batch Jobs
The batch jobs overview shows all registered batch jobs of the application instance and the execution count of them.
image::batch-jobs.png[]
=== Batch Job Executions
The view shows an overview of all executions for the selected job.
To get details of the job execution, click on the desired id.
image::batch-job.png[]
=== Job Execution
The job execution view shows you a detailed overview about the job and step executions of the selected job execution.
image::job-execution.png[]
=== Job Scheduler
_Job Scheduler Configurations_ are cron or time based scheduler.
image::scheduler.png[]
==== Add Job Scheduler Configuration
===== Period Scheduler
image::scheduler-period-add.png[]
===== Cron Scheduler
image::scheduler-cron-add.png[]
=== Job Listener
image::listener.png[]
==== Add Job Listener configuration
image::listener-add.png[]
=== Job Launcher
image::job-launcher.png[]
=== Job Execution Events
image::job-execution-events.png[]
=== Journal
image::journal.png[]
=== Server Scheduler
==== Server Scheduler Executions
image::server-scheduler-executions.png[]
==== Server Scheduler Configurations
image::server-scheduler-configurations.png[]
==== Add Server Scheduler Configuration
image::server-scheduler-configuration-add.png[]