allow custom condition assignment

[deargle]

has come up a few times recently

[deargle]

@jacoblee @ttomita would something like this satisfy your needs for hacking the condition-assigning stuff? It tries to import a function named custom_get_condition from custom.py to use for condition determination. If that fails, it uses the status quo balanced-assignment fallback. This is the main part:

try:
    from custom import custom_get_condition as get_random_condcount
except ImportError:
    def get_random_condcount(mode):
           ...

diff --git a/psiturk/experiment.py b/psiturk/experiment.py
index 16dd245..a6a401f 100644
--- a/psiturk/experiment.py
+++ b/psiturk/experiment.py
@@ -152,56 +152,59 @@ def shutdown_session(_=None):
 # Experiment counterbalancing code
 # ================================

-def get_random_condcount(mode):
-    """
-    HITs can be in one of three states:
-        - jobs that are finished
-        - jobs that are started but not finished
-        - jobs that are never going to finish (user decided not to do it)
-    Our count should be based on the first two, so we count any tasks finished
-    or any tasks not finished that were started in the last cutoff_time
-    minutes, as specified in the cutoff_time variable in the config file.
-
-    Returns a tuple: (cond, condition)
-    """
-    cutofftime = datetime.timedelta(minutes=-CONFIG.getint('Server Parameters',
-                                                           'cutoff_time'))
-    starttime = datetime.datetime.now() + cutofftime
+try:
+    from custom import custom_get_condition as get_random_condcount
+except ImportError:
+    def get_random_condcount(mode):
+        """
+        HITs can be in one of three states:
+            - jobs that are finished
+            - jobs that are started but not finished
+            - jobs that are never going to finish (user decided not to do it)
+        Our count should be based on the first two, so we count any tasks finished
+        or any tasks not finished that were started in the last cutoff_time
+        minutes, as specified in the cutoff_time variable in the config file.
+
+        Returns a tuple: (cond, condition)
+        """
+        cutofftime = datetime.timedelta(minutes=-CONFIG.getint('Server Parameters',
+                                                               'cutoff_time'))
+        starttime = datetime.datetime.now() + cutofftime

-    try:
-        conditions = json.load(
-            open(os.path.join(app.root_path, 'conditions.json')))
-        numconds = len(list(conditions.keys()))
-        numcounts = 1
-    except IOError as e:
-        numconds = CONFIG.getint('Task Parameters', 'num_conds')
-        numcounts = CONFIG.getint('Task Parameters', 'num_counters')
-
-    participants = Participant.query.\
-        filter(Participant.codeversion ==
-               CONFIG.get('Task Parameters', 'experiment_code_version')).\
-        filter(Participant.mode == mode).\
-        filter(or_(Participant.status == COMPLETED,
-                   Participant.status == CREDITED,
-                   Participant.status == SUBMITTED,
-                   Participant.status == BONUSED,
-                   Participant.beginhit > starttime)).all()
-    counts = Counter()
-    for cond in range(numconds):
-        for counter in range(numcounts):
-            counts[(cond, counter)] = 0
-    for participant in participants:
-        condcount = (participant.cond, participant.counterbalance)
-        if condcount in counts:
-            counts[condcount] += 1
-    mincount = min(counts.values())
-    minima = [hsh for hsh, count in counts.items() if count == mincount]
-    chosen = choice(minima)
-    #conds += [ 0 for _ in range(1000) ]
-    #conds += [ 1 for _ in range(1000) ]
-    app.logger.info("given %(a)s chose %(b)s" % {'a': counts, 'b': chosen})
-
-    return chosen
+        try:
+            conditions = json.load(
+                open(os.path.join(app.root_path, 'conditions.json')))
+            numconds = len(list(conditions.keys()))
+            numcounts = 1
+        except IOError as e:
+            numconds = CONFIG.getint('Task Parameters', 'num_conds')
+            numcounts = CONFIG.getint('Task Parameters', 'num_counters')
+
+        participants = Participant.query.\
+            filter(Participant.codeversion ==
+                   CONFIG.get('Task Parameters', 'experiment_code_version')).\
+            filter(Participant.mode == mode).\
+            filter(or_(Participant.status == COMPLETED,
+                       Participant.status == CREDITED,
+                       Participant.status == SUBMITTED,
+                       Participant.status == BONUSED,
+                       Participant.beginhit > starttime)).all()
+        counts = Counter()
+        for cond in range(numconds):
+            for counter in range(numcounts):
+                counts[(cond, counter)] = 0
+        for participant in participants:
+            condcount = (participant.cond, participant.counterbalance)
+            if condcount in counts:
+                counts[condcount] += 1
+        mincount = min(counts.values())
+        minima = [hsh for hsh, count in counts.items() if count == mincount]
+        chosen = choice(minima)
+        #conds += [ 0 for _ in range(1000) ]
+        #conds += [ 1 for _ in range(1000) ]
+        app.logger.info("given %(a)s chose %(b)s" % {'a': counts, 'b': chosen})
+
+        return chosen

 # Routes

[deargle]

See https://github.com/NYUCCL/psiTurk/commit/309a62346832da5aa6c02568851b09230df93d96

[deargle]

@jacoblee @ttomita this should be a cleaner way to get the condition assignments that you want without all the db hacking (looking at you ttomita)

[jacob-lee]

I think that would work great.

My only (small) caveat is that the function being called will be be named, get_random_condcount but might not actually be the function with the nameget_random_condcount defined in the same file; that could be confusing when debugging (especially debugging other people's code).

Instead, I would want to have a variable with a generic name like (get_condition) which is assigned a reference to a function, something like:

# in experiment.py
try:
    from custom import custom_get_condition as get_condition
except ImportError:
    get_condition = get_random_condcount

that way, when looking up the definition of get_condition, one is forced to walk through the import.

[deargle]

Hmm yes, i like your approach.

On Wed, Jul 31, 2019, 3:57 PM jacob-lee notifications@github.com wrote:

I think that would work great.

My only (small) caveat is that the function being called will be be named, get_random_condcount but might not actually be the function with the name get_random_condcount defined in the same file; that could be confusing when debugging (especially debugging other people's code).

Instead, I would want to have a variable with a generic name like (get_condition) which is assigned a reference to a function, something like:

in experiment.py

try: from custom import custom_get_condition as get_condition except ImportError: get_condition = get_random_condcount

that way, when looking up the definition of get_condition, one is forced to walk through the import.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/NYUCCL/psiTurk/issues/363?email_source=notifications&email_token=AAI6Y7LYH3FMIDHJQ6YD5VLQCIDFZA5CNFSM4IHAKR32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3IV5JY#issuecomment-517037735, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI6Y7LIZWMXDJ7DC4BITLDQCIDFZANCNFSM4IHAKR3Q .

[deargle]

implemented: https://github.com/NYUCCL/psiTurk/commit/06e8288c04bb895992e41a6a5b9ac1b66c4b40a2