[cuadrado] Translating Target to Source Constraints in M2M

[grammarware]

Submitted by @jesusc to https://github.com/grammarware/models17/tree/master/cuadrado

[ahmadsalim]

I am experiencing some difficulties with running the tool.

When I try to follow the provided instructions to run the example class2table file, I get a blank anATLyzer pane. I have tried re-clicking Add/Remove anATLyzer nature and pressing the yellow arrows a few times to recheck, but it seems that nothing is changing.

Additionally I get errors when I import the pos2pre evaluation script java project regading the anatlyzer.evaluation package which can not be resolved, and some other errors following from that.

Could you kind provide some hints on how to troubleshoot these issues?

[grammarware]

Ping @jesusc @estherguerra @jdelara @robertclariso @jcabot — could you provide some aid in helping @ahmadsalim to solve this issue? Thanks in advance!

[jesusc]

Hi,

Thanks for taking the time to check this.

It seems that you have properly installed AnATLyzer since you see the Analysis view (this is what it is blank, right?). Is the transformation opened in the ATL editor? The Analysis view only shows the analysis of the result of the currently opened transformation.

Regarding the pos2pre script, you are right. It seems that it is not working with the binary version of the tool because there is a package which is only available in the source distribution. I will package it as a jar and upload a new version of pre2post with all the dependencies included. Meanwhile, let's try to have to example working for you.

[ahmadsalim]

@jesusc Thanks for looking into this.

This is how my view looks:

[jesusc]

Ok, this is weird.

Could you tell me the Eclipse version you are using (and you operating system)? I will try to test in a similar environment.

[ahmadsalim]

I am using Eclipse Neon.3 version 4.6.3., ATL SDK 3.7.0 and anATLyzer 0.6.

[jesusc]

Looking closely to the snapshot, it seems to me that the analysis is actually working under the hood, because the marker in line 35 is produced by AnATLyzer. If you hover it you should see "Possibly unresolved binding (Attribute): Attribute". However, I am not sure why the view it is not working.

I will check with your configuration.

[ahmadsalim]

Yeah, it seems that it indeed shows the binding warning.

[jesusc]

I have downloaded Neon.3 Release (4.6.3) for Linux 64, Modelling Tools Edition, and installed AnATLyzer from the update site. I checked the four features provided by the update site. It works with the example.

The only thing that I can think of now is a problem with the operating system. Which OS as you using?

[ahmadsalim]

I am using macOS El Capitan 10.12

[ahmadsalim]

I can try doing the rest of the artifact evalution on Linux, but it is strange that it should not work on macOS.

[jesusc]

I have access to a Macbook with 10.11.6. I am going to try it here and I will let you know.

[jesusc]

I have tried and it works. Are you using starting Eclipse with Java 8? This is required for some parts of the code.

Anyway, I have zipped the configure Eclipse version and you have it available here

[wasowski]

@jesusc the zipped version might be a good idea anyway. The artifact should be possibly resistant to bit rotting over time.

[ahmadsalim]

@jesusc Thanks for configuring it, but unfortunately I get that the archive is damaged when unzipped. I am trying to install Eclipse Modelling Tools on my other computer for a fresh environment to test on, and see if that works.

[ahmadsalim]

@jesusc OK, I got the example to work on my home computer. I will wait for you to fix the evaluation script before proceeding.

[mschoettle]

Hi,

I am using a fresh copy of Eclipse Neon.3 (4.6.3) Modeling Tools on Mac OS 10.11.6 with Java 8.

Regarding anATLyzer installation, I chose all 4 features. I somehow expected ATL to be installed automatically when installing anATLyzer. I believe it should be possible to define such a dependency for the anATLyzer feature. Otherwise, a note regarding where ATL can be found would be helpful since it might not be clear to everyone (I used the Neon update site).

The example worked for me. However, I get a different result than shown in the screencast.

• the hasPKeysPrecondition already exists and the conflict is on enoughColumns, therefore I added a enoughColumnsPrecondition precondition
• when opening the dialog for the analysis result of enoughColumns the visualization does not work:

• I suggest to change "The example used ..." to "The example ATL project" in the README to make it clear that the zip file includes a project.

When running the experiment I get the following exception:

----> transformation mutants generated
java.net.URISyntaxException: Illegal character in path at index 40: file:/path/to/workspace/pos2pre.evaluation.zip_expanded/pos2pre.evaluation/resources/EVAL.ecore
    at java.net.URI$Parser.fail(URI.java:2848)
    at java.net.URI$Parser.checkChars(URI.java:3021)
    at java.net.URI$Parser.parseHierarchical(URI.java:3105)
    at java.net.URI$Parser.parse(URI.java:3053)
    at java.net.URI.<init>(URI.java:588)
    at java.net.URL.toURI(URL.java:946)
    at pos2pre.evaluation.TesterWF.initConformanceChecking(TesterWF.java:604)
    at pos2pre.evaluation.TesterWF.runEvaluation(TesterWF.java:221)
[...]

I think the illegal character refers to a space I have in the path (the name of the Eclipe app).

@ahmadsalim I was able to unzip the zipped Eclipse version. Maybe try to download it again (I used Safari)

[mschoettle]

I renamed my Eclipse app to not include spaces.

When running the "EVALUATE preconditions" I get this exception:

transML exception: URI not found
  ** /evaluation_project.zip_expanded/anatlyzer.evaluation.preconditions/er2rel/temp/conformance/EVAL.ecore
    at transML.utils.modeling.EMFUtils.loadEcoreMetamodel(EMFUtils.java:63)
    at transML.utils.modeling.EMFUtils.loadEcoreMetamodel1(EMFUtils.java:72)
    at pos2pre.evaluation.TesterWF.initConformanceChecking(TesterWF.java:609)
    at pos2pre.evaluation.TesterWF.runEvaluation(TesterWF.java:221)

Is this the same problem as mentioned above?

[jesusc]

Hi,

For the first problem, you need to install Graphviz to visualize the witness models in the dialog. I have added this dependency to the README.md. Also, I have added a pointer to the ATL update site.

Regarding the problem with the evaluation script. It seems that it only works when the project is located within the workspace. So, I have added the following instructions:

You must import the project with the seed transformations directly from the zip file using "Import -> Existing projects in to workspace -> Select archive file". This is because the evaluation script assumes that the project directory is located within the workspace. Also, make sure that there are no whitespaces in the Eclipse path.

It should work now.

Thank you.

[ahmadsalim]

@mschoettle I was able to unzip it, but not to run the .app inside (where I get corrupt archive).

[mschoettle]

@jesusc Thanks for updating the README.

For the evaluation script, I did the following steps, maybe I missed something:

1. Imported existing project into workspace from pos2pre.evaluation.zip
2. Ran second Eclipse instance
3. Imported existing project into that workspace from _evaluationproject.zip
4. Opened er2rel/er2rel.atl and performed "Target invariants analysis"
5. Right-clicked on the file and selected "EVALUATE preconditions ... with Witnessfinder"

I still get the same error. The temp directory was created and the .ecore file is there.

[jesusc]

Sorry for the trouble. I got this error before and I fixed it as commented above. It is clear that our code for handling file paths is not bulletproof :-)

I am going to look closely to this issue and try to find a workaround.

Thank you.

[jesusc]

@mschoettle I have uploaded a new version of the evaluation script. I hope it works for you now, although I cannot be sure.

The results will be generated in a couple of xls files, which are actually CSV files which should be read by Excel. However, I have just realized that formulas are written with Spanish names, so I am not sure if this will work with other systems. Just let me know.

[mschoettle]

Thanks, it works now.

And you are right, the Spanish function names in the xls files unfortunately don't work with other systems.

[artifacteryx]

Hi! I could follow all steps from the detailed instructions without problems. However, I'm a bit unsure about the consistency between the paper and the tool:

According to the paper, the main contribution is a method that takes as input a transformation with post-conditions, and generates some corresponding pre-conditions. The question is: Using the tool, how do I apply this method to a given ATL file with post-conditions to generate the pre-conditions?

The screencast is confusing in this respect: at 0:19, the pre-condition and post-condition are inserted by the user via copy and paste. However, from my understanding, the user should only insert the post-condition, and the pre-condition should be generated automatically.

[jesusc]

@artifacteryx Thank you for your comment. You are right about the contribution, however the artifact is made of several items: the implementation, the evaluation results and the script to reproduce the results.

I have added the following to the README.md:

Artifact description

The presented artifact consists of the following elements:

• The raw data obtained during evaluation of the pre-condition generation method (here)
• An script which allow to reproduce the evaluation from scratch (here)
• The implementation of the method as integrated in AnATLyzer.

The next part of the artifact description shows how AnATLyzer can be installed and used. This includes the integration of the pre-condition generation with a model finder, which is just one of the applications of the method.

Regarding how to see and copy the generated pre-conditions, this is shown in the image. If you click on a post-condition after the analysis (in the Analysis view) you should get a dialog window in which the generated pre-condition appears, and you could copy-paste it if you wish.

[artifacteryx]

@jesusc Thanks! Indeed, based on the image, I now see that the generated pre-condition is shown in the dialog. Since it's your main contribution, you should certainly mention it in the text as well.

The clarification in the Readme is indeed important. I think the documentation could be improved even further by having a short section where you explain the standalone use of your method. At least, I would expect a section header "Generating source constraints" somewhere in the text, even if it's within the description of the integration with the model finder.

[artifacteryx]

This submission presents the implementation and evaluation artifacts for a paper on the translation of target-metamodel to source-metamodel constraints for exogenous model transformations. It includes the anATLyzer tool with its support for the constraint translation, a demo example, three evaluation examples, and a benchmark framework that applies mutation testing to the evaluation examples.

(1) Is the artifact consistent with the paper?

Yes. All evaluation artifacts described in the paper are available, including outputs and summaries, and I could reproduce the constraint generation.

(2) Is the artifact as complete as possible?

Concerning the formalities of the artifact evaluation process (see https://itu.dk/research/codegrp/), the submission may not be complete yet:

• The two-page description document is not mentioned.
• The ReMoDD or FigShare submission is not mentioned.

From the perspective of reproducibility and reuse, the submission seems complete, since all artifacts to reproduce the experiments are contained in the (Github-based) submission.

(3) Is the artifact well-documented?

I was able to reproduce all described steps based on the detailed Readme files. It's very convenient that the authors provide all artifacts produced during the evaluation and the summarized results as downloads.

In the Readme, as already remarked, it would be preferable if the role of the new contribution within the considered usage scenario (constraint generation vs. its integration with a model finder) could be emphasized better.

Minor comments:

"Graphviz (optional), Zest 1.5 (optional)" - What precisely are the steps required to install these and make them work with anATLyzer? "Right-click on the ATL transformation" - Where? ("in the package explorer" should be added.) "The process will generate a temp directory with all the mutants and xls files summarizing the result." - To where is the directory generated?

The screencast is a nice idea. Unfortunately, it was not made clear if the screencast contains additional steps not shown in the repository (in other words, if the user is required to watch it).

(4) Is the artifact easy to (re)use?

The detailed documentation makes it generally easy to (re)use the artifact in the short term. For long-term use, it would be better to upload all involved artifacts to ReMoDD or Figshare.

Since the evaluation can take quite long (as correctly pointed out in the Readme), it would be best if a particular set-up could be indicated that does not take so long to execute. Unfortunately, my Eclipse crashed with heap overflow errors after several hours. So it would also be good to specify the memory requirements of the used Eclipse version.

Additional comments (concerning the UI of anATLyzer):

• "Right-click on the project -> Add anATLyzer nature" For consistency with the rest of the Eclipse world, I think this should be an entry of the "Configure" menu group.
• Analysis view: The icon for "Run analyser" is used inconsistently with the regular Eclipse use, where it signifies "Link with editor".

[ahmadsalim]

Summary

The artifact represents a component of the anATLyzer tool that infers the (OCL) preconditions necessary to impose on source models for an ATL transformation, in order for the target model constraints to be satisfied. The technique is tested on three different transformations (ER2Rel, HSM2FSM and Fact2PN in two variants), and to generalize the results, mutants are automatically generated from the target constraints of these transformations, thereby testing the adequacy of the technique; the technique is deemed adequate if it generates preconditions that are safe (so that the constraints are not violated by outputs produced from inputs that satisfy these preconditions) and still permissive (that no input that would produce valid output is excluded).

Assessment

Consistency

• All the transformations evaluated in the paper are available with the artifact.
• I was able to successfully rerun the experiments producing output Excel sheets with the desired results. The output functions were in Spanish, so I had to manually change their name to my own locale for it to work.
• The output Excel summary is still not as nicely formatted as the results presented in the paper. I think it would be helpful if it was possible to generate the tables in the paper automatically, or alternatively that there are instructions that detail the correspondence.

Completeness

• I agree with @artifacteryx that it seems that the artifact has not been uploaded to a suggested repository which would be useful.
• I think that the provided instructions (README) should be converted to a 2 page LaTeX document as stated in the requirements.

Documentation

• Although I had some issues in the beginning, after the corrections I was able to succesfully follow all instructions and run the example and experiments.
• I do not primarily work with Eclipse, and so there a few things regarding the instructions which I think could be clarified:
  • How to import a project correctly in Eclipse. I had some issues before because I imported the projects the wrong way, and it took me some tries to see that one should actually use Import... > General > Existing Project > Archive.
  • I think that Check the features that you want to install and click Finish could be more precise. I would usually expect the instructions would tell me exactly what features and dependencies are required to reproduce the results in the paper.

Reuse

• Besides archiving as mentioned by @artifacteryx , I think the artifact is very reusable. The code for the complete static analysis tool is available as open source: https://github.com/jesusc/anatlyzer and the code is reasonably documented.

[patrimrodilla]

The artefact presented consists of an implementation and evaluationof a new method for translating backwards postconditions on large models (in Model to Model transformations) to source model preconditions. The method is implemented in the anATLyzer tool and examples and a benchmark framework that applies mutation testing to the evaluation examples.

(1) Consistent with the paper

Yes. All evaluation artifacts described in the paper are available, and after some problems in the preparation of the environment, I could run the experiments.

(2) As complete as possible Apart form some minor formalities regarding AEC that other reviews mentioned before, I think the artifact is complete.

(3) Well-documented Although the README files are easy to follow and the author’s presented total disponilility for solving issues, I found a little messy all the installation and execution process. I mean, I know that these environments are complex and requires some dependencies, but maybe could be a a good idea for the future to provide also a Virtual machine image or similar mechanism in order to facilitate the pre-running process (it could also work if you plan to present the tool to different kind of audience that are not extremely familiar with Eclipse-based environments). In addition, it would be wonderful to be able to run your tiny target model (for instance only a Petry net or similar) and to see what happens in this case with the constraints treatment. Although I understand the reason of author’s in using mutations, I think this possiblity will clarify the interesting contribution of the method without other implementation and noisy aspects. I also have some interest in how the method creates the CTT if you have models with several levels of abstraction (class generalization etc) that implies a lot of possibilities in terms of building the tree and also all the backward process beyond the examples presented.

(4) Is the artifact easy to (re)use?

I think the artifact is reasonably (re)usable, although it could be necessary for different programmers and modellers profiles to encapsulate and simplify a little bit the pre-running and running process for future work.

[jesusc]

Thank you for all your comments. They have helped me to improve the artifact and I will also take them into account to improve AnATLyzer in the future.

To improve the reuse of the artifact in the long term, I have uploaded a zip file to Figshare containing: a) VirtualBox VM with AnATLyzer pre-installed and b) the raw data and the execution scripts. The link is available as part of the README.md file.

[mschoettle]

This artifact accompanies a paper on providing a translation from target to source constraints for model-to-model transformations, ensuring that no incorrect target model is produced.

The artifact consists of one use case of the source constraint generation using a model finder. It includes a simple demo project as well as a script to re-create the experiment and the required data (metamodels and transformations).

Consistency with paper

The four transformations mentioned in the paper are provided. However, the paper uses a running example (factory to petri nets), which is not included in the artifact. For consistency, I think it would be good to include this as well. This would allow to follow the same example as in the paper.

• How is the precision and recall value given in Table 5 calculated from the values provided in the summary spreadsheet?
• In order to verify the results with the ones provided in the paper, please change the used functions in the Excel spreadsheets to English ones (I believe those should work in any language versions, but even if not, it would at least allow someone else to translate it to their own).

Completeness

The links provided within the README provide all the required projects and data to re-run the experiment. Furthermore, the full evaluation data as well as summaries are provided.

A two page artifact paper seems missing still, but a virtual image has been uploaded to figshare including the plug-ins and above mentioned data.

Due to the long runtime of the experiments and large memory requirements when choosing the same settings as the authors, I could unfortunately not run them to verify the consistency of results (see also documentation comment below).

Well-documented

Following the instructions provided in the README and using the linked artifacts, it is possible to set up Eclipse, perform the analysis and run the evaluate preconditions with witness finder.

I think the instructions could be a bit more detailed in general. Furthermore, some specific comments:

• The second part of checking target constraints includes instructions to do the transformations. I think it would make sense if this comes before the explanations of the conditions, and therefore after setting up the transformation.
• The setup for the experiments should include memory requirements and approx. expected runtime for each transformation since it can run quite long.
• It is not fully clear whether the determined pre-conditions provided by the analysis have to be first added to the .atl file before running the EVALUATE preconditions. Maybe this can be clarified in the instructions.

Easy (re)use

The uploaded zip file to figshare provides all required artifacts and an Eclipse with anATLyzer installed. However, since the instructions detail a set up of Eclipse, it might be good to provide here in this repository an archived version of the required anATLyzer plug-in as well as archived versions of the zip files containing the experiment script and data linked in the README.

[grammarware]

Dear @jesusc @estherguerra @jdelara @robertclariso @jcabot

The Artifact Evaluation Committee of MoDELS 2017 has reached the conclusion during the closed committee meeting that this artifact conforms to the expectations and is hereby approved. Please use the badge instructions page to add the badge of approval to your article, and add the link to the FigShare entry with DOI https://doi.org/10.6084/m9.figshare.5211892.v1 to the camera ready version of the paper.

Thank you very much for putting extra effort into the preparation and finalising of the artifact. If any of the comments above are still not addressed, please try to accommodate them before the conference.