psuzzi / parse-test

A project to benchmark and compare different parsers
0 stars 0 forks source link

Refactor benchmarks to allow parser tests even in case of different libraries versions #5

Closed psuzzi closed 2 months ago

psuzzi commented 3 months ago

This maven project has a centralized benchmark structure.

parse-test                      [pom] parent pom
├── parse-test-antlr-v4         [jar] antlr parser (java)
├── parse-test-benchmark        [jar] benchmark depend on: common, antlr, kolasu, xtext (java/kotlin)
├── parse-test-common           [jar] common classes (java)
├── parse-test-kolasu-v1-5      [jar] kolasu parser (kotlin)
└── parse.test.xtext.parent     [pom] xtext parser parent [pom]
    └── parse.test.xtext        [jar] xtext parser (java/xtend)

This means the parse-test-benchmark depends on all the 'parse-test-*project and on theparse-test-common So doing, theparse-test-benchmark` pulls all the libraries from the other projects, and a lib conflict is unavoidable. We could fix the behaviour by reversing the dependency chain.

So, we could have the parse-test-benchmark as a dependency for all the parser projects, in the same way the parser projects depend on parse-test-common, as long as the parse-test-benchmark does not bring in any parser dependency. One of the current issues, is the ArithmeticExprGrammarFuzzer is actually using Antlr v4, in order to verify the generated samples are correct. So, the benchmark has this dependency to Antlr4, that we should remove.

So for the refactoring, we should proceed as follows:

  1. remove the Antlr Dependency from parse-test-benchmark
  2. remove all the 'parse-test-*(parser) dependencies from theparse-test-benchmark`
  3. Implement a different parsing structure, where the parse-test-benchmark can be imported and used by the parsers
  4. Delegate the benchmarking to the single parse projects.

This should enable the addition of multiple parser benchmarks, even with different library versions.

psuzzi commented 3 months ago

Here is the current status of the relevant dependencies, where local projects are in blue, printed using the depgraph generator:

image

Based on analysis, we could: