search

martin-fleck / cra-ttc2016

The Class Responsibility Assignment Case for the Transformation Tool Contest 2016
3 stars 2 forks source link

K.Lano evaluation of Henshin #25

Open klano opened 8 years ago

klano commented 8 years ago

EvaluationSheet(1).xlsx

dstrueber commented 8 years ago

Hi, I'm surprised about the CRA scores given in the table. Have you followed the instructions to use our recommended configuration (Runner.java class)?

klano commented 8 years ago

We took those xmi files from the code repository for Henshin, Kevin

Date: Thu, 26 May 2016 07:36:51 -0700 From: notifications@github.com To: cra-ttc2016@noreply.github.com CC: klano@hotmail.com; author@noreply.github.com Subject: Re: [martin-fleck/cra-ttc2016] K.Lano evaluation of Henshin (#25)

Hi, I'm surprised about the CRA scores given in the table. Have you followed the instructions to use our recommended configuration (Runner.java class)?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub

dstrueber commented 8 years ago

Ah I see. Indeed, we have a large number of XMI files in our repository, since we tried out many configurations of our tool (as we say in the paper). But the point of all these XMI files was to find our recommended configuration that can be used to produce good results. In the front page of the repository., we provide instructions to produce high-quality output models and to find such models in the repository.

Sorry for creating confusion with those many XMI files. Can you check if you can reproduce the better scores by following the instructions to trigger the Runner.java file?

klano commented 8 years ago

Now we have checked the 'best of' files, and these have much better values for CRA, together with a reasonable number of classes per model.

dstrueber commented 8 years ago

Hi Kevin, thanks for going the extra mile and re-checking our results. These CRA scores give a more accurate account of our solution's optimality, since our experiments helped us to find a configuration that produces such outcomes consistently. But of course, we see that our presentation could have been much clearer in this respect.