Problem with expired groups in sequential match making

[ctreffe]

Let's have a look at the following minimal experiment, in which one of two match makers is created based on an exp.condition determined by a ListRandomizer. In Addition to the basic randomization and match making code, I added some print statements for clarification:

import alfred3 as al
import alfred3_interact as ali

exp = al.Experiment()

@exp.setup
def setup(exp):

    # For an experiment with match making, we want to reduce the default timeout

    exp.session_timeout = 40  # 60 * 60 * 24 is the default value

    # Proof of concept: Combining the ListRandomizer with different MatchMaker specs

    randomizer = al.ListRandomizer.balanced("natural", "sanitized", n=2, exp=exp)
    exp.condition = randomizer.get_condition()

    print(exp.condition)
    print(randomizer.allfinished)

    if exp.condition == "natural":
        spec = ali.SequentialSpec("aggregate", "single", nslots=1, name="natural_spec")
        exp.plugins.mm = ali.MatchMaker(spec, name="mm_natural", exp=exp)
        print(spec.quota.npending)
        print(spec.quota.full)

        group = exp.plugins.mm.match()
        exp.plugins.group = group

    elif exp.condition == "sanitized":
        spec = ali.SequentialSpec("aggregate", "single", nslots=1, name="sanitized_spec")
        exp.plugins.mm = ali.MatchMaker(spec, name="mm_sanitized", exp=exp)
        print(spec.quota.npending)
        print(spec.quota.full)

        group = exp.plugins.mm.match()
        exp.plugins.group = group

@exp.member
class Success(al.Page):

    def on_first_show(self):
        group = self.exp.plugins.group
        role = group.me.role

        self += al.Text(f"Successfully matched to role: {role} in condition: {self.exp.condition}")

if __name__ == "__main__":
    exp.run()

When I run this experiment locally, everything works just fine (since I use two SequentialSpecs). There are, alltogether, four slots for participants, two in each match maker. When I finish each session regularly, all four slots are filled depending on the ListRandomization and the associated MatchMaker. Aborted sessions are also filled up as expected (If I abort a session, the same slot will be reassigned immediately, since the slot is directly opened in the ListRandomizer, again, and the MatchMaker also recognizes the aborted session).

However, problems arise, when sessions are not finished or aborted, but expire after the session timeout. As can be seen from the print statements (I think), the randomizer works as expected. The MatchMaker aborts the ongoing session regularly, hence, the ListRandomizer can directly reassign the slot with the aborted session. This is the reason why we never see the ListRandomizer as full.

After the second started session, the MatchMaker always shows a pending group and returns spec.quota.full as true, which is why the experiment is aborted. This happens even after waiting for the 40 second timeout. Interestingly, new sessions are still added to an appropriate group (can be seen on a local machine by inspecting the relevant group's json file. There, the ongoing session is added as member), which I think is an indicator that the matching process works just fine, but somehow the experiment abort is triggered falsely.

I think the expected behaviour should be that the first slot in the relevant sequential match maker (depending on the order of our randomization list) should become available again after the preceding session expires. At least I think this is the correct way this should work. Please, correct me if I'm wrong.

[ctreffe]

Update: I now see that my idea is corroborated by the long entries:

2022-05-17 00:33:36,539 - exp.default_id.match.MatchMaker - INFO - experiment id=default_id - session id=sid-b62b54781e1c4c53b8d6d512dea6235f - Starting stepwise match of session to existing group: Group(roles=['aggregate', 'single'], active members=1, id='f8a7').
2022-05-17 00:33:36,830 - exp.default_id.match.MatchMaker - INFO - experiment id=default_id - session id=sid-b62b54781e1c4c53b8d6d512dea6235f - Session matched to role 'aggregate' in Group(roles=['aggregate', 'single'], active members=1, id='f8a7').
2022-05-17 00:33:36,836 - exp.default_id - INFO - experiment id=default_id - session id=sid-b62b54781e1c4c53b8d6d512dea6235f - ExperimentSession.abort() called. Aborting session. Reason: matchmaker_full

As we can see, the session is aborted even though the matching was successful and match_next_group() was called (where the log entries above the abort stem from).

[jobrachem]

Do I understand correctly?

Let’s say we have only one spec with roles A and B. Let’s also say that A must be finished before B can be assigned. Let’s als say that the spec expects only one group.

1. Participant 1 is matched to role A
2. The matchmaker (correctly) aborts a new Participant 2, as the spec quota is currently full (since A is not finished).
3. Participant 1 runs into the session timeout
4. Role A should now be available again.
5. The matchmaker (correctly) matches Participant 3 to role A.
6. The matchmaker then incorrectly aborts Participant 3, claiming that the quota is full.

[ctreffe]

Yes, exactly! Sorry that I did not provide this much clearer example earlier (it was late yesterday and I was working on my own experimental setup). I just tested it and the issue is as you described, with the following log output:

2022-05-17 07:28:59,460 - exp.default_id.match.MatchMaker - INFO - experiment id=default_id - session id=sid-e3625e84cea6426ca9a325f5a7e9c9d5 - Starting stepwise match of session to existing group: Group(roles=['aggregate', 'single'], active members=1, id='fc11').
2022-05-17 07:28:59,478 - exp.default_id.match.MatchMaker - INFO - experiment id=default_id - session id=sid-e3625e84cea6426ca9a325f5a7e9c9d5 - Session matched to role 'aggregate' in Group(roles=['aggregate', 'single'], active members=1, id='fc11').
2022-05-17 07:28:59,484 - exp.default_id - INFO - experiment id=default_id - session id=sid-e3625e84cea6426ca9a325f5a7e9c9d5 - ExperimentSession.abort() called. Aborting session. Reason: matchmaker_full

[jobrachem]

Interesting. I hope I can have a look at this later this week.

[ctreffe]

Some additional obervations on this issue:

Maybe the issue has existed since early implementations of the SequentialSpec. As far as I can see, with 2 groups per spec everything works fine. I cannot guarantee, that everything works as expected, but in the end I got 4 complete groups with my first example experiment an nslots per spec set to 2.

However, another thing I noticed when testing this extensively with a local experiment, the match maker created 5 groups instead of the configured 4. Hence, I wonder whether maybe this is related to the primary issue I described? I think the 5th group would also have been assigned by the match maker, but the ListRandomizer closed the experiment before this could happen.

[jobrachem]

Fixed in 85d3845599a7c7a20128ce1bf95e4e5e0155372e