The issue of balance in DB-Efficient designs

[mmardehali]

Consider the following attributes and levels:

1. Cost: $20, $25, $30
2. Brand: Known, Unknown
3. Rating: 3.2 Stars, 4.0 Stars, and 4.8 Stars
4. Usability: High, Low, Unavailable

These attribute-levels were presented in the code as follows: profiles <- cbc_profiles( cost = seq(20, 30, 5), brand = c("known", "unknown"), rating = seq(4.8, 3.2, -0.8), usability = c("high", "low", "unavailable") )

I found that creating a DB-Efficient design using these attributes and levels, results in complete omission of "Rating: 4.0 Stars" from the final design. Using cbc_balance, I would observe that there is no representation from that particular attribute level. At first I thought floats are not allowed in the design and they are being truncated, resulting in an unbalanced design, but that is not the case. After changing the levels to "5, 4, 3" instead, I found that 4 does get representation, but it was still very unbalanced (e.g. 25 appearances for 5 stars, 4 appearances for 4 stars, and 25 appearances for 3 stars).

The issue seems to be that, since both Cost and Rating are presented as ratio data, the optimization algorithm deems the differences between 3.2 to 4.0 to 4.8 to be not as significant as the differences between 20 to 25 to 30. As such, the optimal design just ignores the "medium" value for Rating, and only represents the highest and lowest values. Obviously, in a real world application, there is a significant and perceivable difference between the ratings of a 3.2 stars product and a 4 stars product.

The workaround I found is to introduce the Ratings with the same values as Cost, i.e. rating = seq(30, 20, -5), and then manually replace the values in the final design with 4.8, 4.0, and 3.2 respectively. For manual replacement, I personally write the dataframe of the optimal design to a CSV file, and then find and replace the values accordingly, since I'll be using the CSV file to generate my survey. This resulted in a far more balanced optimal design with almost equal representation for all levels for Rating. The only issue I can anticipate, is that the power analysis simulation using cbc_power may be even further removed from the findings of the study with actual participants, and thus the power analysis simulation becomes more unreliable.

I wonder if this workaround violates the anything else in the optimal design algorithm, however, this is the only way I could find to create a balanced optimal design. Although this is technically an issue with the "idefix" package, I wonder if there is a proper way to address that in cbc_tools?

[jhelvy]

I think a key piece of information missing here is what you're using for priors. With all 0 priors, you get rather poor balance for the continuous features of cost and rating. This makes sense as you're essentially saying that there's no difference in utility when you have a higher cost or higher rating:

profiles <- cbc_profiles(
    cost = seq(20, 30, 5),
    brand = c("known", "unknown"),
    rating = seq(4.8, 3.2, -0.8),
    usability = c("high", "low", "unavailable")
)

design <- cbc_design(
  profiles  = profiles,
  n_resp    = 500, # Number of respondents
  n_alts    = 3, # Number of alternatives per question
  n_q       = 6, # Number of questions per respondent
  priors = list(
    cost      = 0,
    brand     = 0,
    rating    = 0,
    usability = c(0, 0)
  )
)

cbc_balance(design)

cost:

  20   25   30 
3500 5000  500 

brand:

  known unknown 
   3500    5500 

rating:

 3.2    4  4.8 
4500  500 4000 

usability:

       high         low unavailable 
       3500        2500        3000 

If however you use priors where cost and rating both start to really impact utility, you'll get better balance:

design <- cbc_design(
  profiles  = profiles,
  n_resp    = 500, # Number of respondents
  n_alts    = 3, # Number of alternatives per question
  n_q       = 6, # Number of questions per respondent
  priors = list(
    cost      = -1,
    brand     = 0,
    rating    = 2,
    usability = c(0, 0)
  )
)

cbc_balance(design)

cost:

  20   25   30 
2500 4500 2000 

brand:

  known unknown 
   4500    4500 

rating:

 3.2    4  4.8 
3500 3000 2500 

usability:

       high         low unavailable 
       3000        2500        3500 

I try to at least insert priors that determine the expected direction (negative for increased cost, and positive for increased rating), even if I'm not sure what the true parameters might be.

[mmardehali]

That's a fair point, and you are absolutely right! Currently, my priors are set to 0 for all attributes as I'm designing the pilot study. But it seems like once I have my priors from the pilot, this won't be an issue anymore. I was following the recommendations by Traets et al. (https://www.jstatsoft.org/article/view/v096i03) for designing the pilot study (page 4), hence why I specified my priors as 0. For designing the pilot, they recommend using either an orthogonal design or an efficient design with zero parameters.

[jhelvy]

It's a debatable trade off. If you really don't know what the priors are, I tend to just use a randomized design and aim for a larger N. It will ensure that I get good balance, but at the expense of statistical power. The danger is using all 0s is that the design is being optimized around those values, so if the true parameters are different from 0 (which they probably are), you may end up worse off than if you just used a randomized design.

The good thing is that you can simulate how this might play out. If you use all 0s, go simulate choices where the "true" parameters are not 0, then see what happens in your power analysis. Then use the same "true" parameters on a fully randomized design and check the power. Does it look any different?

[jhelvy]

I'm closing this as it's no longer an active issue, but it's a useful example of the trade offs in setting different priors for a Bayesian D-efficient design.