Sync with Democles

[anthonyanjorin]

Hi Greg,

I'm now slowly working on synchronisation...

Currently, I have the problem that I seem to be receiving multiple events for essentially the same match. Every time the handleEvent method is invoked, the MatchEvent is different but its DataFrame retrieved via event.getMatching() is exactly the same object.

Any ideas why this is happening? Is such behaviour expected or is something wrong?

The resource set seems to be perfectly ok, and the matches I am receiving make sense as well, just that I'm getting multiple events for the same match.

Cheers and thanks, Tony

[gervarro]

Hi Tony,

Is such behaviour expected or is something wrong?

No, such a behaviour would be wrong.

First thing to clarify. A MatchEvent is a triple consisting of the match itself, the producing pattern, and the delta type. If you have a central location for processing matches (i.e., one MatchEventListener is attached to all the patterns (RetePattern)), then all patterns (RetePattern) report to the same location, which means that you might receive matches (DataFrame) with identical content from different patterns. Do the observed match events come from the same source, and are they of the same type INSERT vs. DELETE? In the tested setup, I could imagine that you get match events with the same content but first from the CORR_CONTEXT, and then next from the MODELGEN.

Have not you attached more than one listeners to the same set of patterns? This could also yield to the observed behaviour. Match events are sent to all the listeners attached.

If the above two cases can be excluded, then it is probably a bug.

Bests, Greg

[anthonyanjorin]

Hi Greg,

I have now checked everything carefully. For the simple example (attached), I get four times a match event for HandleRegisters_FWD with exactly the same type (insert), the same producing pattern, and the same match (DataFrame).

Even though the patterns are fairly trivial and the pattern invocation is silly, I still expect to receive only one match for HandleRegisters_FWD.

Please tell me if you can reproduce. For the time being I'll use a set to filter the events (but I guess this is an unwanted hack that should be unnecessary).

Cheers and thanks, Tony

models.zip

[gervarro]

Hi Tony,

I checked the situation, and it is a bug caused by two issues. Some initialization is still missing in case of pattern invocations, which is the main source of the erroneous behaviour. However, I additionally noticed that the analysis module should also be implemented ASAP to provide a reliable behaviour in the TGG setup. Currently, I am not sure that the first fix alone is enough. I can fix the first issue fast, but the implementation of the analysis module is a somewhat larger step (1-2-3 weeks depending on my other duties).

Bests, Greg

[anthonyanjorin]

Hi Greg,

thanks for trying it out.

I'll try to continue working with whatever fixes you can offer. If the first fix goes fast then I'll gratefully take it and see how far I get :) I have a good student starting his Master thesis on Ibex which is why I'm trying to give gas at the moment. I'm pretty sure the model generator should be enough for the first few weeks though.

Cheers, Tony

[gervarro]

Hi Tony,

Just a short status report: I managed to fix the bugs and to implement the call graph analysis module. I will try to prepare a new version on the update site in the next couple of days. You will have to update the pattern matcher configuration at some locations. I will share the details when I am ready with the update.

Bests, Greg

[gervarro]

Hi Tony,

The promised modifications are now online. Please update the corresponding 3 features (specification, interpreter, advanced) from the update sites. You should remove the code from the Democles configuration that handles pattern invocations in the incremental case (i.e., PatternInvocationComponentBuilder must not be created in your code manually any more). My other recommendation would be to invoke the internal pattern matcher builder like in the code below (once for all pattern specifications, and not one by one for each individual specification, which would construct a new Rete network each time when you add a new specification):

// 2) EMF-independent to pattern matcher runtime (i.e., Rete network) transformation
retePatternMatcherModule.build(internalPatterns.toArray(new DefaultPattern[internalPatterns.size()]));

Bests, Greg

[anthonyanjorin]

@Arikae @FStolte : Once @erhanleblebici has performed these fixes we all have to update the features (org.gervarro.*) to get ibex running as expected.

[erhanleblebici]

After having removed the configuration for pattern invocation (which did not compile anyway), I get the following exception at runtime: Exception in thread "main" java.lang.RuntimeException: No handler exists for constraint type org.gervarro.democles.constraint.PatternInvocationConstraintType @ 7f0eb4b4

The line which I removed can be seen here (according to your recommendation, @gervarro) : https://github.com/eMoflon/emoflon-ibex/commit/e1d5a19ca64fe018a539911d5a7a80aad09d0481

The question is whether our configureDemocles() method is still proper. I have the impression that we should delete more. It's current version can be found here: https://github.com/eMoflon/emoflon-ibex/blob/master/org.emoflon.ibex.tgg.core.runtime/src/org/emoflon/ibex/tgg/operational/util/DemoclesHelper.java

[gervarro]

I do not think that the Democles configuration would be a problem, as the above error says that the handler for pattern invocations is missing. You removed exactly this line from the configuration, which should be handled automatically in the latest version.

Have you downloaded the features from both the default and the advanced update sites? Does the org.gervarro.democles.interpreter.incremental plugin contain an org.gervarro.dfs package? If you set a breakpoint to the setModule() method of the GlobalSearchPlanAlgorithmSession class, does its subclass (this variable in the Variables view in the Debug perspective) have a patternRootDepths attribute? If not, then please download the latest version. However, if the corresponding jar files (o.g.d.interpreter, o.g.d.interpreter.incremental, o.g.d.plan.incremental.leaf) have a relatively recent date (mid-March), then there might have been a problem with the deployment process.

[gervarro]

Just another comment. The EMF-based to EMF-independent transformation needs to have the patterns in the resource set. In debug mode, the pattern resources are removed in the saveDemoclesPatterns() method, which will definitely lead to errors.

[anthonyanjorin]

Hi Greg,

I have been able to update and I can run the code without getting any exceptions.

Unfortunately, the process is now non-deterministic: sometimes some matches are available (as expected), sometimes not. This has nothing to do with saveDemoclesPatterns() as I tried it also with debug = false.

It's very strange, all I can say is that if you run the code 4-5 times, one time it'll work as expected (the PM produces all matches as expected), the other times not (no matches at all). Now and then it also produces matches for some rules, but not for all.

I hope you can reproduce this; I've attached the patterns (very simple), models, and metamodels.

BTW: The exact same code was working before I updated the pattern matcher and removed the line in the configuration code as you suggested.

setup.zip

[gervarro]

Hi Tony,

This has nothing to do with the configuration.

Unfortunately, the process is now non-deterministic: sometimes some matches are available (as expected), sometimes not.

The search plan algorithm prepares different Rete networks. If a network contains a structure that is buggy at runtime, then it might result in no matches. However, if a network does not contain any buggy structures, then you get the corresponding matches. In the last commit, I changed the runtime behaviour of the engine so it might easily happen that a case is not yet correct.

Bests, Greg

[anthonyanjorin]

Ok. If I understand correctly we should wait for a fix.

BTW: I hope to start work on a testsuite (based on our existing tests for eMoflon) very soon so we can run automatic tests after every update to avoid regressions.

It's just a bit hard to test the (random) model generator so this will only make sense as soon as synchronisation is working (hopefully very soon).

Thanks for the fast reply! Tony

[gervarro]

Hi Tony,

I am planning to prepare a scenario for testing all possible Rete network substructures, but as there are many valid combinations it might take somewhat longer. I could probably fix the bug faster if you could help me identifying the cases in which the pattern matcher is erroneous.

Please add the following lines after retePatternMatcherModule.getSession().setAutoCommitMode(false); to print the Rete network structure:

for (final RetePattern retePattern : retePatternMatcherModule.getPatterns()) {
    final List<RetePatternBody> bodies = retePattern.getBodies();
        for (int i = 0; i < bodies.size(); i++) {
            final RetePatternBody body = bodies.get(i);
            System.out.println(body.getHeader().toString() + " @ " + i + ": " + body.getRuntime().toString());
        }
    }
}

Then attach the listener to all patterns and tell me the smallest one, which did not produce the expected result. Please send me the Rete network descriptions as well.

Many thanks and bests, Greg

[anthonyanjorin]

Hi Greg,

Of course! Below is the output produced for the smallest pattern that is supposed to match but doesn't:

SonToMaleRule_MODELGEN/2 @ 0: BidirectionalJoinOperation@6(BidirectionalJoinOperation@5(SonToMaleRule_SRC_CONTEXT/1->[0]->[0],UnidirectionalJoinOperation@4(BidirectionalJoinOperation@3(UnidirectionalJoinOperation@2(BidirectionalJoinOperation@1(platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//Family^U->[0],platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//Family/postalNumber^BF->[0,2]),Equal^BB->[2,4])->[0],platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//Family/name^BF->[0,3]),Equal^BB->[3,5])->[0])->[0, 2, 3],BidirectionalJoinOperation@1(platform:/resource/FamiliesToPersonsDeterministic/model/Persons.ecore#//PersonRegister^U->[1],SonToMaleRule_TRG_CONTEXT/1->[0]->[1])->[1])

Another pattern that also doesn't match but should is this:

IgnoreFamily_MODELGEN/3 @ 0: BidirectionalJoinOperation@8(BidirectionalJoinOperation@7(BidirectionalJoinOperation@1(IgnoreFamily_SRC_CONTEXT/1->[0]->[0],platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//FamilyRegister^U->[0])->[0],BidirectionalJoinOperation@6(BidirectionalJoinOperation@5(platform:/resource/FamiliesToPersonsDeterministic/model/Persons.ecore#//PersonRegister^U->[1],IgnoreFamily_CORR_CONTEXT/3->[1]->[0,1,2])->[2],platform:/resource/FamiliesToPersonsDeterministic/model/FamiliesToPersonsDeterministic.ecore#//RegisterToRegister^U->[2])->[0])->[1],IgnoreFamily_TRG_CONTEXT/1->[0]->[1])

In case you need this information, these are the other remaining patterns that should and do match as required (note that they do not have any pattern invocations so this is probably what's not working):

SonToMaleRule_TRG_CONTEXT/1 @ 0: platform:/resource/FamiliesToPersonsDeterministic/model/Persons.ecore#//PersonRegister

IgnoreFamily_CORR_CONTEXT/3 @ 0: BidirectionalJoinOperation@4(platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//FamilyRegister^U->[0],BidirectionalJoinOperation@3(platform:/resource/FamiliesToPersonsDeterministic/model/FamiliesToPersonsDeterministic.ecore#//RegisterToRegister/source^BF->[2,0],BidirectionalJoinOperation@2(platform:/resource/FamiliesToPersonsDeterministic/model/Persons.ecore#//PersonRegister^U->[1],BidirectionalJoinOperation@1(platform:/resource/FamiliesToPersonsDeterministic/model/FamiliesToPersonsDeterministic.ecore#//RegisterToRegister/target^BF->[2,1],platform:/resource/FamiliesToPersonsDeterministic/model/FamiliesToPersonsDeterministic.ecore#//RegisterToRegister^U->[2])->[1])->[2])->[0])

IgnoreFamily_SRC_CONTEXT/1 @ 0: platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//FamilyRegister

SonToMaleRule_SRC_CONTEXT/1 @ 0: UnidirectionalJoinOperation@4(BidirectionalJoinOperation@3(UnidirectionalJoinOperation@2(BidirectionalJoinOperation@1(platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//Family/name^BF->[0,2],platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//Family^U->[0]),Equal^BB->[2,4])->[0],platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//Family/postalNumber^BF->[0,1]),Equal^BB->[1,3])

IgnoreFamily_TRG_CONTEXT/1 @ 0: platform:/resource/FamiliesToPersonsDeterministic/model/Persons.ecore#//PersonRegister

Just say if I can provide anything else.

Cheers and thanks, Tony

[anthonyanjorin]

Hi Greg,

Can I help or support in any way?

Thanks, Tony

[gervarro]

Hi Tony,

Sorry for the delay, but I was busy last week. I analyzed the situation today on the train, and it seems to be a trivial bug again. If it is really the case, then I will upload the fix either today or tomorrow.

Bests, Greg

[anthonyanjorin]

Great! Thanks :)

BTW: My student is now working full time so things should really start moving soon. Hopefully we can have a meaningful Ibex@Democles release in just a few months.

[gervarro]

Hi Tony,

the fix is now available on the update site. The match set of the IgnoreFamily_MODELGEN/3 pattern was affected by the bug, and it should be OK now. Please re-check the result set or the specification of the SonToMaleRule_MODELGEN/2 pattern. I do not know how you wanted to specify the SonToMaleRule_SRC_CONTEXT/1 pattern, but it requires a Family and not a FamilyRegister. There was only an object of type FamilyRegister in my model, consequently, the pattern matcher did not provide any matches for SonToMaleRule_MODELGEN/2. What I want to say is that the SonToMaleRule_MODELGEN/2 pattern should not have any matches if a model only consists of a FamilyRegister object, but no Families.

Bests, Greg

[anthonyanjorin]

yep - all seems to be working. Great!

[gervarro]

Could you please provide me the latest set of patterns (and models) if they changed?

[anthonyanjorin]

Hi Greg,

Please find a minimal example attached. I have reduced things to a single pattern and a set of models where it should match but doesn't:

IgnoreFamily__CONTEXT_CORR/3 @ 0: BidirectionalJoinOperation@4(BidirectionalJoinOperation@3(BidirectionalJoinOperation@2(platform:/resource/FamiliesToPersonsDeterministic/model/FamiliesToPersonsDeterministic.ecore#//RegisterToRegister/source^BF->[2,0],BidirectionalJoinOperation@1(platform:/resource/FamiliesToPersonsDeterministic/model/FamiliesToPersonsDeterministic.ecore#//RegisterToRegister/target^BF->[2,1],platform:/resource/FamiliesToPersonsDeterministic/model/FamiliesToPersonsDeterministic.ecore#//RegisterToRegister^U->[2])->[2])->[1],platform:/resource/FamiliesToPersonsDeterministic/model/Persons.ecore#//PersonRegister^U->[1])->[0],platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//FamilyRegister^U->[0])
IgnoreFamily__CONTEXT_TRG/1 @ 0: platform:/resource/FamiliesToPersonsDeterministic/model/Persons.ecore#//PersonRegister
IgnoreFamily__CONTEXT_SRC/1 @ 0: platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//FamilyRegister
IgnoreFamily__MODELGEN/3 @ 0: BidirectionalJoinOperation@9(IgnoreFamily__CONTEXT_TRG/1->[0]->[1],BidirectionalJoinOperation@8(BidirectionalJoinOperation@7(BidirectionalJoinOperation@6(platform:/resource/FamiliesToPersonsDeterministic/model/FamiliesToPersonsDeterministic.ecore#//RegisterToRegister^U->[2],BidirectionalJoinOperation@5(platform:/resource/FamiliesToPersonsDeterministic/model/Persons.ecore#//PersonRegister^U->[1],IgnoreFamily__CONTEXT_CORR/3->[1]->[0,1,2])->[2])->[0],platform:/resource/FamiliesToPersonsDeterministic/model/Families.ecore#//FamilyRegister^U->[0])->[0],IgnoreFamily__CONTEXT_SRC/1->[0]->[0])->[1])

patternsAndModels.zip

[anthonyanjorin]

Hi Greg,

Just wondering if you can reproduce or if I messed up the update.

Cheers and thanks, Tony

[gervarro]

Hi Tony,

a new version is available online. Download updates from both sites!

Bests, Greg

[anthonyanjorin]

@FStolte

[anthonyanjorin]

Hi Greg,

unfortunately we still have some cases where matches are missing for certain patterns. @FStolte (doing his master's thesis) will soon post a minimal example with patterns and models here.

Cheers, Tony

[gervarro]

Hi Tony,

sorry, but these complex pattern invocations are first tested with the TGG scenario. I am waiting for the example. Btw. do you still experience the JoinFailedException at runtime?

Bests, Greg

[FStolte]

Hello Greg,

Unfortunately, I was not able to reproduce the problematic cases Tony mentioned since they last occurred. Hopefully they were just caused by some bug in my runtime-eclipse that is now gone...

Anyway, the "Join failed" Exception is still occurring for me. In the attached example it is thrown rather consistently (but not always). However, when i remove the patterns for MotherToFemaleRule, it works without fault every time. Although the patterns for FatherToMaleRule and MotherToFemaleRule are completely similar, the exception only occurs if both are present.

PatternsAndModels.zip

Thanks, Florian

[FStolte]

Hi Greg,

It seems that we were able to resolve our issues. While we still do not know why the "Join failed" Exception was thrown, it does not occur anymore. It looks like all matches are now handled properly.

Thanks, Florian