Proposal: a framework to parse and manipulate ADL files with java

[lazurinke]

An application description language (ADL) file is generated by the compiler to describe the streaming application which was compiled .

On some cases it is very useful to get an intimate knowledge about how the application is structured especially when you are not the vendor of the application but just need to "take a peek" under the hood to understand the plumbing .

I propose to contribute a framework which I have developed using java 8 and jax-b which can parse and create object mappings based on the an ADL file. for more information please see illustration 1 .

Usage example:

String file = "my/adl/file/path";
ApplicationSet adlAsClass = generator.generate(file);
SplApplication splApplication = adlAsClass.getSplApplication() ;
List<TupleType> tupleTypes = splApplication.getTupleTypes();

Ilustration 1: interrogating our ADL using java .

[nasgaard]

-1 The problem is that the ADL file is a private API. It can and does change from release to release, which would cause this framework to potentially break or require updates with each release. There is no plan to publish that interface. Also, manipulating that file would achieve nothing. Even if you submit the job using that file, it simply uses the location of the ADL file to find the SAB file with the same name. The real ADL file used to submit the job is contained in the SAB file.

[ddebrunner]

What is provided in addition to jaxb classes generated from the schema?

[lazurinke]

A little explanation.

@nasgaard This framework was created for internal use , to ease up problem determination and specifically isolation of performance bottlenecks within streaming applications. Obviously the input would be the SAB file which contains the ADL.

@ddebrunner The framework's classes were not automatically generated from the schema as that has resulted in quite a mess, the framework classes were created and developed carefully based on the structure of the ADL file (using jax-b for unmarshalling) as you mentioned, to mimic the schema but not to be overly complex (as can be seen in the screenshot).

The framework currently contains the following enhancements :

1. Programmatically extraction of the ADL file from the SAB file

2. intelligent wrapping and inheritance of many types of objects for example handling different types of ports (screenshot above).

3. This framework could later be easily integrated with the topology toolkit in order to incrementally build a SAB based on the data contained within the ADL file and the objects which the framework has unmarshaled from it .

The motivation behind this framework is to break up the application into small bits (operators) , then automatically building the application incrementally by adding one operator at a time and investigating the effects it has on throughput and latency by in that way we will be able to diagnose bottlenecks.

[chanskw]

Closing this one due to lack of interest and -1 vote. Please reopen if you would like to continue with this discussion. Thanks!