Note. This repository was used for the 2015 Transformation Tool Contest and is no longer maintained. If you wish to use the benchmark, visit the Train Benchmark repository, which is actively maintained.
The paper/trainbenchmark-ttc.pdf
file contains the case description.
mvn --version
).Use the scripts/init-config.sh
file to initialize the config/config.json
configuration file.
The scripts
directory contains the run.py
script which is used for the following purposes:
run.py -b
-- builds the projectsrun.py -b -s
-- builds the projects without testingrun.py -g
-- generates the instance modelsrun.py -m
-- runs the benchmarkrun.py -v
-- visualizes the results of the latest benchmarkThe config
directory contains the configuration for the scripts:
config.json
-- configuration for the model generation and the benchmark (based on the config-default.json
file)reporting.json
-- configuration for the visualizationSet the maxSize
variable to the desired value and run the run.py -g
script. With enough memory (-Xmx2G
or more), the models from size 1
to size 512
are generated in about 5 minutes.
The script runs the benchmark for the given number of runs, for the specified tools, queries and sizes.
The benchmark results are stored in a TSV (tab-separated values) file. The header for the TSV file is stored in the output/header.tsv
file.
Make sure you read the README.md
file in the reporting
directory and install all the requirements for R.
It is recommended to start with an Eclipse distribution tailored to developing EMF-based applications, e.g. Eclipse Modeling.
If you'd like to try the ATL/EMFTVM implementation, it is recommended to use Eclipse Luna. There are two ways to resolve the dependencies:
pom.xml
files). This requires the m2e Eclipse plugin (this is included in Eclipse for Java developers but is not included in Modeling distribution). The m2e plugin can be installed from the the update site of your release (Kepler/Luna).MANIFEST.MF
files).
In general, we recommend to stick to your proven build solution, else you may spend a lot of time tinkering with the build. In theory, you can build Eclipse plug-ins with the Tycho Maven plug-in, however, it has a steep learning curve and is tricky to debug. For reference, see https://github.com/FTSRG/cheat-sheets/wiki/Maven-and-Eclipse.
To implement a tool, it is recommended to start from an existing implementation. Please implement your own benchmark logic and benchmark case factory which instantiates the classes for each query defined in the benchmark.
In order to make the fewest assumptions on the specific implementations, the pattern matches are stored in the variable matches
declared as a Collection<Object>
(see the AbstractBenchmarkCase class). The framework requires the matches
collection to be unique.
To enable a consistent ordering of the matches, the framework requires a comparator class. Section 2.4.2 ("Ordering of the Match Set") in the case description defines the rules of the ordering.
For implementing a match comparator, we recommend two approaches:
To avoid confusion between the different implementations, we decided to use the Smurf Naming convention (see #21). This way, the classes in the Java implementation are named JavaBenchmarkCase
, JavaPosLength
, JavaPosLengthMatch
, JavaPosLengthTransformation
, while the classes in the EMF-IncQuery implementation are named EMFIncQueryBenchmarkCase
, EMFIncQueryPosLength
, etc. We found that relying on the package names to differentiate class names like
hu.bme.mit.trainbenchmark.ttc.benchmark.java.BenchmarkCase
andhu.bme.mit.trainbenchmark.ttc.benchmark.emfincquery.BenchmarkCase
is error-prone and should be avoided.
To ensure that the Query Explorer works without launching a runtime Eclipse instance, go to Preferences, EMF-IncQuery and Query Explorer and tick the Dynamic EMF mode checkbox.