Strange behavior when testing incrementality in forward direction

[tbuchmann]

The test cases specified in the class IncrementalForward.java obviously have several issues, where some of them are related to the transformation tools used and which are not discussed in this issue.

To sort them out i started running them for eMoflon only. I encountered a strange behavior: The class comprises 4 methods, which implement test cases according the Issues TC 6a-6d. When run in isolation (all other tests are commented out), testIncrementalInserts and testIncrementalDeletions seem to work find. They also work if they are run together with testIncrementalRename. However, when all 4 test methods are active (no method is commented), the method "testIncrementalDeletions" fails in the postcondition, as the actual and expected PersonRegisters differ. My assumption was that tool.initiateSynchronisationDialogue() starts with an empty model each time, so that the results should be comparable, right?

Steps to reproduce: comment out all test methods in IncrementalForward.java and then remove the comment for one method in each step and check the result.

Any suggestion is welcome.

[tbuchmann]

btw: testIncrementalRename fails in the Postcondition with differing FamilyRegisters (so that's a different problem)

[anthonyanjorin]

This sounds like non-deterministic behaviour and might be due to a faulty comparator implementation (that applies some simple strategies to sort and normalise the models). E.g., the Simpson families might not be sorted deterministically.

[tbuchmann]

I also think the comparator might be faulty, but i don't think this is the reason here. I added some println's to the code and obtained the following result (when running all test methods specified in the class in a row):

Incremental Delete: Expected: PersonRegister { persons = [ Male {
fullName = "Flanders, Rod" , birthday = "2013-11-01" }, Male {
fullName = "Simpson, Bart" , birthday = "2013-11-09" }, Male {
fullName = "Simpson, Bart" , birthday = "2013-11-11" }, Male {
fullName = "Simpson, Homer" , birthday = "2013-11-09" }, Female {
fullName = "Simpson, Lisa" , birthday = "2013-11-08" }, Female {
fullName = "Simpson, Maggie" , birthday = "2013-11-07" }, Female {
fullName = "Simpson, Marge" , birthday = "2013-11-09" } ] }

Actual: PersonRegister { persons = [ Male {
fullName = "Flanders, Rod" , birthday = "2013-11-01" }, Male {
fullName = "Simpson, Bart" , birthday = "2013-11-09" }, Male {
fullName = "Simpson, Bart" , birthday = "2013-11-10" }, Male {
fullName = "Simpson, Homer" , birthday = "2013-11-09" }, Female {
fullName = "Simpson, Lisa" , birthday = "2013-11-08" }, Female {
fullName = "Simpson, Maggie" , birthday = "2013-11-07" }, Female {
fullName = "Simpson, Marge" , birthday = "2013-11-09" } ] }

Please note: when incremental rename is not called, the testIncrementalDelete passes without error. But when incrementalrename is also called, the test fails because the older son Bart is delted from the Simpsons family instead of the younger one.

[tbuchmann]

I integrated our own solution, just to make sure the test cases are correct. Our tool passes all incremental forward tests, while the other tools mostly fail. Not sure, if the tests are tailored towards our tool (which i doubt, because I strictly followed the proposals on GitHub).

[anthonyanjorin]

These are my results up till now:

• testIncrementalMove is a real failure for eMoflon. The TGG wasn't designed with such movements in mind and is not optimal (regarding least change). The moved family members are revoked and retranslated (thus losing their birthdays in the person register).
• testIncrementalDeletions is probably failing because helperFamily::deleteFirstSonBart does not guarantee that the same bart is always deleted. eMoflon can sometimes reshuffle the models (and their sorting) so it is a bit dangerous just to pick to the "first" element in the sons list. It would be better for helperFamily::deleteFirstSonBart to always delete the youngest or oldest bart (irrespective of his position in the list of sons). This also explains why the test sometimes passes.
• similar to testIncrementalDeletions, testIncrementalRename is probably failing because eMoflon shuffles the model elements and the way the different Barts are sorted is not maintained. So helperPerson::changeAllBirthdays does not guarantee which Bart get's which birthday and the final result is not as expected. Again I would suggest making sure that helperPerson::changeAllBirthdays always changes birthdays in a deterministic manner, irrespective of the sorting of Barts in the person register.

[tbuchmann]

I agree with you, but how should helperFamily::deleteFirstSonBart know, who is the youngest Bart? The Family model does not have birthdays and we can not access the correspondence model in order to figure out which person of the person register is mapped to which family member. The same holds for helperPerson::changeAllBirthdays. Actually i have no idea how to make it deterministic without accessing some kind of trace information.

[anthonyanjorin]

Perhaps one should decompose helperPerson::changeAllBirthdays into two functions that are applied at points where this is clear. E.g. when the person register only has one Bart set birthdays, then afterwards the second Bart is obviously the Bart with the default birthday.

An idea for helperFamily::deleteFirstSonBart would be to obtain a reference to the first Bart when this is clear (so before there are multiple Barts), and then keep this reference to use for deletion later.

Would these ideas work?

[tbuchmann]

I don't think it would work. According to the specification of the test cases, the birthdays should be changed AFTER the initial fwd transformation. In other words on a complete Person Register.

I will try your idea for helperFamily::deleteFirstSonBart. I think it could work.

[anthonyanjorin]

As we discussed (telco) the birthday change should be before the precondition assertion and is not part of the actual test (this was unclear/wrong in the issues). So I wouldn't care how (when, in how many steps) this is done as it is just part of the dialogue establishing a well-defined precondition.

[tbuchmann]

I modified the testcases. The precondition is now established in several steps, to allow setting the birthdate in a deterministic way. Changes are committed to the respective branch.