Default Response to UserInteraction during Propagation

[NeumannDirk]

The classes TestUserInteraction provides different options to define and store responses to different user interactions. The conditions of these responses are tried to be matched if an interaction occurs and if the response matches it is consumed. This allows you to check whether all predefined responses were used and therefore if all interactions occurred as expected.

However, it does not allow defining default-responses. If an interaction occurs multiple times (let's say n times) then one also must add the same reaction n times, even if the answer is every time the same. Concerning the development towards the benchmarx-framework (but also in general) it might be helpful to be able to do so. Since one does not know how many of the same interaction will occur, the current solution to realize default-responses is either:

1. Adding the same response multiple times (say 50 times) and if the default answer should be changed, clear all the remaining answers and add 50 copies of the new response. This requires an estimation of how many interactions will occur and has the disadvantage, that it will also clear all the other responses which then need to be re-added

2. During execution of a response build and add the very same response again if a "default"-flag is set which then needs to exist for every different response once.

My suggestion would be to add a second version of:

• confirmations
• notificationReactions
• textInputs
• multipleChoices
• multipleChoiceMultipleSelections in TestUserInteraction with the prefix "default_" which contain the default answers. Then, at first the program tries to find a matching response in the default list which is not consumed, and if unsuccessful it tries to find one response in the normal list which will be consumed. A problem here is to check if multiple default responses might be triggered on the same interaction and which should be chosen in this case, but a solution might be to just take the first matching and let the user be responsible for what responses he added, much like it is currently with the normal responses.

[HeikoKlare]

I have to admit that I do not understand the necessity for such a kind of default interaction response implementation. Maybe there is some issue I do not understand yet. You say:

Since one does not know how many of the same interaction will occur

But I do not understand why that can be the case. Execution of test cases should be deterministic, so there will be the same number of the same interactions in each execution of the test case. If there is let's say 20 times the same interaction, you have to add one interaction response 20 times, as this is exactly what you expect. Can you explain why that does not work?

Adding default responses is highly error-prone. If you define a default response for some kind of interaction, you will never know whether unexpected interactions occur and there may even be test cases that succeed by accident because default responses are defined, although they should actually fail because unexpected interactions ocurred that are handled by this kind of "catch all" implementation.

So except that there is some good reason for having default interaction responses, which I do not understand yet, I would not be in favor of having such an implementation.

[NeumannDirk]

So the driving factor of this issue is that in the benchmarx test cases, first the preferences are set (Partent or Child, Existing or New Family) and then a consumer with all the model operations is provided. This consumer contains no information how many operations are included or which kind of operation. So there the default response would be really helpful. Otherwise the solution would be a response which re-adds itself to the list of responses or the described way to overestimate the number of interactions. See for example here: https://github.com/NeumannDirk/benchmarx/blob/master/examples/familiestopersons/BenchmarxFamiliesToPersons/src/org/benchmarx/examples/familiestopersons/testsuite/batch/bwd/BatchBwdNotEAndP.java#L35-L58

From a broader perspective, there are also good reasons to implement default responses as well. If one takes the typical Copy-Paste-Interaction for example. There the user is prompted something like "The file XY already exists in the destination directory. Would you like to replace the file XY?" and a typical option is "Yes, to all". This "Yes, to all" is a default response and in terms of Vitruvius it might occur as well. If for example in the beginning there is only a persons-model and no families-model and the user wants to establish consistency preservation, the user might traverse the persons model and build the families model from it automatically. Here, the user will select a default behavior. If for example the persons are ordered by birthday one combination from default responses could lead to the behaviour "The first male per last name is the father, the first female the mother and everybody else is sorted as son or daughter in the existing family".

In my opinion, the error-proneness stays more or less at the same level. The current functionality of responses already requires some caution during the development of test cases concerning the correct order, correct execution triggers and possible overlaps between these triggers. Furthermore, using one of the workarounds I stated is more error-prone and less self-documenting in terms of showing the developer's intention behind the code. If I missed the point you made about the error-proneness and you were aiming at a different aspect, please let me know so I can reconsider.

[HeikoKlare]

Thank you for the clarification. I see the point that having some kind of system tests with larger example models the proper level of abstraction for thinking about the responses to interactions might be to define a specific behavior for specific kinds of interactions rather than to expect a determined sequence of interactions. This would also apply to the benchmarx cases. I would, however, still discourage doing so for low-level tests, where you want to know that the individual Reactions behave properly. Then it is essential to exactly match the expected interactions.

Regarding the error pronness: What I meant is that defining default behavior for responses to interactions can easily hide bugs or lead to unexpected failures. It is about how to most easily achieve a test case that succeeds (which is what your comment is about, if I understand correctly), but about how to have a test case that identifies bugs, in particular in case of modifications to the existing code / the existing Reactions. This is where it is important to explicitly match each interaction, as otherwise test failures (or even unexpected non-failures) give less information than they could or should. This is particularly important for the tests that target the single Reactions, so that identifying bugs in case of code changes is as simple as possible.

I will take a look at the interaction response builder and would propose to add some always functionality for default behavior, so that you may write something like onTextInput[title.equals("model name")].always.respondWith["model"]. That should keep the readability of builder statements.

[HeikoKlare]

@NeumannDirk Could you please have a look at PR #491 and tell us whether the proposed enhancement of the TestUserInteraction fulfills your need for specifying default responses (either here or in that PR)? Thanks!

[NeumannDirk]

Yes. I think this solves the issue and provides everything one might need. I like the simplicity and conciseness of the .always.respondWith solution. Thank you.