search

eMoflon / benchmarx

Infrastructure for implementing benchmarx: benchmarks for bidirectional transformation (bx) tools. Also contains a collection of example benchmarx and test runners for various and diverse bx tools.
GNU General Public License v3.0
9 stars 12 forks source link

Enable FunnyQT solution in ScalabilityMeasurements #50

Closed tsdh closed 7 years ago

tsdh commented 7 years ago

I know I've said the last PR was my last but I just found out there's also a scalability measurements class. So I added my FunnyQT solution there, too, and the results in comparison to the other tools are horrible. Maybe you can extend the BXToolTimer so that it kills a tool if it doesn't finish within a given amount of time. (E.g., run the transformation in a separate thread and call the (unsafe, deprecated) Thread.stop() method.)

By the way: the case description says that every solution developer should provide a SHARE image with his fork of the benchmarx project. Is that sensible? Wouldn't it be much better to make one SHARE image with all solutions? I guess everyone will fork this project and have her solution integrated here with a PR anyway.

anthonyanjorin commented 7 years ago

No problem, and thanks again for your solution!

The BXToolTimer is just meant as a stub that must be adjusted as required. Your idea with a reasonable timeout is quite nice, however, I'll open an issue.

You're also right with SHARE. We currently provide a VM with all tools we used in our paper. The idea was for people to have a means of running comparable measurements. After TTC I'll try to extend it to include all new tools as well.

tsdh commented 7 years ago

Ok, then I'll create my own share VM.