Java VTL: Java implementation of VTL

The Java VTL project is an open source java implementation of the VTL 1.1 draft specification. It follows the JSR-223 Java Scripting API and exposes a simple connector interface one can implement in order to integrate with any data stores.

Visit the interactive reference manual for more information.

Modules

The project is divided in modules;

java-vtl-parent
- java-vtl-parser, contains the lexer and parser for VTL.
- java-vtl-model, VTL data model.
- java-vtl-script, JSR-223 (ScriptEngine) implementation.
- java-vtl-connector, connector API.
- java-vtl-tools, various tools.

Usage

Add a dependency to the maven project

<dependency>
    <groupId>no.ssb.vtl</groupId>
    <artifactId>java-vtl-script</artifactId>
    <version>0.1.13-SNAPSHOT</version>
</dependency>

Evaluate VTL expressions

ScriptEngine engine = new VTLScriptEngine(connector);

Bindings bindings = engine.getBindings(ScriptContext.ENGINE_SCOPE);
engine.eval("ds1 := get(\"foo\")" +
            "ds2 := get(\"bar\")" +
            "ds3 := [ds1, ds2] {" +
            "   filter ds1.id = \"string\"," +
            "   total := ds1.measure + ds2.measure" +
            "}");

System.out.println(bindings.get("ds3"))

Connect to external systems

VTL Java uses the no.ssb.vtl.connector.Connector interface to access and export data from and to external systems.

The Connector interface defines three methods:

public interface Connector {

    boolean canHandle(String identifier);

    Dataset getDataset(String identifier) throws ConnectorException;

    Dataset putDataset(String identifier, Dataset dataset) throws ConnectorException;

}

The method canHandle(String identifier) is used by the engine to find which connector is able to provide a Dataset for a given identifier.

The method getDataset(String identifier) is then called to get the dataset. Example implementations can be found in the java-vtl-ssb-api-connector module but a very crude implementation could be as such:

class StaticDataset implements Dataset {

    private final DataStructure structure = DataStructure.builder()
            .put("id", Role.IDENTIFIER, String.class)
            .put("period", Role.IDENTIFIER, Instant.class)
            .put("measure", Role.MEASURE, Long.class)
            .put("attribute", Role.ATTRIBUTE, String.class)
            .build();

    @Override
    public Stream<DataPoint> getData() {

        List<Map<String, Object>> data = new ArrayList<>();
        HashMap<String, Object> row = new HashMap<>();
        Instant period = Instant.now();
        for (int i = 0; i < 100; i++) {
            row.put("id", "id #" + i);
            row.put("period", period);
            row.put("measure", Long.valueOf(i));
            row.put("attribute", "attribute #" + i);
            data.add(row);
        }

        return data.stream().map(structure::wrap);
    }

    @Override
    public Optional<Map<String, Integer>> getDistinctValuesCount() {
        return Optional.empty();
    }

    @Override
    public Optional<Long> getSize() {
        return Optional.of(100L);
    }

    @Override
    public DataStructure getDataStructure() {
        return structure;
    }
}

Implementation roadmap

This is an overview of the implementation progress.

Group	Operators	Comment
General purpose	round parenthesis
General purpose	:= (assignment)
General purpose	membership
General purpose	get	The keep, filter and aggregate options are not implemented.
General purpose	put	Defined in the grammar but not implemented
Join expression	[]{}
Join clause	filter
Join clause	keep
Join clause	drop
Join clause	fold
Join clause	unfold
Join clause	rename
Join clause	:= (assignment)
Join clause	. (membership)
Clauses	rename
Clauses	filter
Clauses	keep
Clauses	calc
Clauses	attrcalc
Clauses	aggregate
Conditional	if-then-else
Conditional	nvl
Validation	Comparisons (>,<,>=,<=,=,<>)
Validation	in,not in, between
Validation	isnull	Implemented syntax are `isnull(value)`, `value is null` and `value is not null`
Validation	exist_in, not_exist_in
Validation	exist_in_all, not_exist_in_all
Validation	check	The boolean dataset must be built manually (no lifting).
Validation	match_characters
Validation	match_values
Statistical	min, max
Statistical	hierarchy	The inline definition is not supported. A dataset that has a correct structure can be used instead.
Statistical	aggregate
Relational	union
Relational	intersect
Relational	symdiff
Relational	setdiff
Relational	merge
Boolean	and	Only inside join expression (no lifting).
Boolean	or	Only inside join expression (no lifting).
Boolean	xor	Only inside join expression (no lifting).
Boolean	not	Only inside join expression (no lifting).
Mathematical	unary plus and minus
Mathematical	addition, substraction
Mathematical	multiplication, division
Mathematical	round, ceil, floor
Mathematical	abs
Mathematical	trunc
Mathematical	power, exp, nroot
Mathematical	ln, log
Mathematical	mod
String	length
String	concatenation
String	trim
String	upper/lower case
String	substr	No lifting.
String	indexof
String	date_from_string	Dataset as input not implemented. Only YYYY date format accepted.
Outside specification	integer_from_string
Outside specification	float_from_string
Outside specification	string_from_number

statisticsnorway / java-vtl