Feature Request: Weight Case Variables

[bazar0ff]

Hell there.

I was wondering if weighting was a feature that is coming out, or if it is even on your radar. I currently use a couple different free stat tools to run frequencies and cross tabs for survey research. Nothing huge, usually just helping friends out or playing around, but I really like what I've seen of JASP. Adding the ability to use a weight case variable, or to set the appropriate weighting criteria within JASP and have the application generate the variable would be awesome.

Thanks

[EJWagenmakers]

Thanks for the suggestion. We allow weights for ANOVA, so I don't see why we would not add it to cross tabs. But I'm not very familiar with the procedure, let me look this up...hmmm what I see is that weights are sometimes used with grouped data, to indicate the frequency with which particular responses are observed. But we already take care of that automatically when you select "counts". Can you perhaps point to a specific example in the literature? Maybe it is used to correct for over- or undersampling a particular subgroup...OK, back to Google...yes, indeed! Oops, this is probably an important analysis for survey research! Yes we ought to implement this. There are probably some good R packages for this too. Let's put this high on the list of priorities. Maybe @AlexanderLyNL can look at this when he is finished writing his thesis, or else @TimKDJ perhaps.

Cheers, E.J.

[bazar0ff]

Thanks for looking into this. Weighting is done to balance the response from groups of participants that are either under or over represented in a population. For example, if we know that a population is 55% female and 45% male, but our sample is 65% female and 35% male, weighting would address this to better reflect the attitudes of the general population. I usually only weight when my sample is off by more than 4%, and it can be done across multiple variables (party, age, gender, geography). It is usually not advised to weight variable that has a lot of features (gender only has two, political party has roughly 3, age can be grouped into 3,4 or 5 different buckets) and I typically only weight on two variables, gender, and age with 4 age groups - this results in 8 different weight cases (one per age/gender possibility).

The two ways to address weighting that i have come across are: 1. to allow the data to include a weight case variable that has to be generated outside the analysis software but is then used to calculate the frequencies and other analyses; 2. with the software to define the ratios that should be reflected in the population based on the variables imported (Question 2 is gender, 1 = male 2 = female, 1 should be 45%, 2 should be 55%...) and the software then calculates the appropriate weight case values. PSPP uses the first option, SPSS uses the second.

Make sense?

[boutinb]

@vandenman We have to decide if we want to do this enhancement.

[ndpegram]

I realise this has languished for a while. I'd like to comment that the lack of the ability to weight data was one of the strongest arguments against our institution adopting JASP rather than SPSS. This is significant for us as most of our research projects are in the social sciences.

[Ed-Houghton]

> I realise this has languished for a while. I'd like to comment that the lack of the ability to weight data was one of the strongest arguments against our institution adopting JASP rather than SPSS. This is significant for us as most of our research projects are in the social sciences.

Agree with the above point. The lack of weighting on JASP is a real barrier to colleagues using this fantastic software. It would be great to see this function included in the near future.

[EJWagenmakers]

Yes. @boutinb , can you bump the priority on this? We should assign this to someone, let's discuss this on Slack.

[djstepz1]

I realise this has languished for a while. I'd like to comment that the lack of the ability to weight data was one of the strongest arguments against our institution adopting JASP rather than SPSS. This is significant for us as most of our research projects are in the social sciences.

Hi all,

Id agree with this.

I can see a big pitch here for a Jasp takeover (particuarly when it comes to SPSS) among universities and college academics and students if the ability to weight data was an added feature.

I too have been detered from using Jasp solely for this very reason as it limits what i can do when working with survey data that requires weighting of the data prior to any form of analyses. Weighting mainly takes account of survey design for things like representativeness and non response within the survey. Complex surveys in general use more intuned approaches than simple random sampling, with participants clustered by area or strata which in the analysis stage needs to be taken into account for accuracy.

Weighting is a really important feature for researchers and I genuinely believe its addition within JASP would be a gamechanger. Though the addition of applying basic weighting variables to the data should be priority (the weights themselves come with the survey and thus we just need a way of applying this variable to the data).

If i could be so bold I would also like to make a suggestion of how in future complex survey design weighting could be intergrated into JASP. I firstly recommend having a look at SPSS's Complex Survey module. Something similar could possibly be added to JASP as an all-in-one module with built in analyses that uses commonly applied techiques and tests: descriptives, crosstabs/Chi-square, t-test/Anova, Regression, Logistic Regression, GLM, Mixed models). Bests.

[StattMatt]

Hi, I want to inquire if there are any developments or if a decision has been made against weights.

Greetings

[EJWagenmakers]

We're hiring someone who can hopefully implement this (but it may take a while). Any other suggestions on R packages for survey design and analysis are welcome. Please feel welcome to join us in helping design this functionality. E.J.

[dschmuecker]

Thank youm @juliuspfadt, for bringing these three threads together. In addition to what @bazar0ff wrote in 2017, I would like to highlight:

1. Weighting is a standard procedure in demoscopic research. It is used to compensate for systematic (non-random) drop-outs in the sampling scheme (redressment) or control for distortions in stratified samples (design weighting).
2. The ability to use a weighting variable would distinguish JASP from other R-based programs (Bkue Sky has it for crosstabs, but not for all other descriptives) and would probably mean a huge step forward for those using large-scale demoscopic datasets, e.g. in sociology, economics, marketing, health science, political science or education research. As opposed to psychology, where the focus is on comparing two or more groups in an experiment, the focus in demoscopics is on describing and analysing a population by way of a carefully constructed and, in the best case, representative sample.
3. Minimum requirement would be to USE a weighting variable in all descriptive processes. A weighting variable is a variable with a mean of 1 over all cases of the sample. A value of 0 in the weighting variable means to exclude the case from analysis, and negative values do not occur (to my knowledge). "Using a weighting variable" means to multiply all values by the value of the weighting variable and present the result in a frequency value list or cross tabulation, rounded to the full number (the counts, not the percentages) or as a weighted mean, standard deviation etc. The xtab and gmodels packages seem to be able to accomplish this, but only for crosstabs afaik.
4. A completely different thing would be to CONSTRUCT a weighting variable. This can be accomplished by a number of different ways, from simple marginal weighting for a very reduced number of active weighting items to complex cell estimates with a larger number of weighting items. A simple implementation of cell weighting is the RAKING procedure in SPSS.

I would love to have JASP being able to USE weighting variables. In SPSS, this is done with a simple syntax line (WEIGHT BY variable) or a switch in one of the menus. Once switched on, SPSS stays in "weighted" mode until the mode is switched off. I could live vey well without JASP being able to CONSTRUCT weights, because the datasets I use come with a predefined weight and many public datasets have the weight variable included.

[EJWagenmakers]

I agree and would love to prioritize this more. With our new data editing in place probably. Something to discuss with the team. @vandenman @Kucharssim @FBartos

[EJWagenmakers]

A quick request for clarification. Suppose you have a column "weights" and a column "values". Then in JASP you can construct a new column that is "weights * values". If you also want cases excluded when they have weights == 0 you can set a filter for that. Is this essentially a request to make this process automatic and apply the weights to multiple relevant variables at the same time?

[EJWagenmakers]

At any rate, including a "weight by" option in descriptives is easy enough. Putting JASP in "weight mode" does seem more challenging (we'd have to reconsider our discrete data analyses). We could add the option to individual analyses and toggle all of them on or off with a setting in the Preferences menu, for instance.

[dschmuecker]

A quick request for clarification. Suppose you have a column "weights" and a column "values". Then in JASP you can construct a new column that is "weights * values". If you also want cases excluded when they have weights == 0 you can set a filter for that. Is this essentially a request to make this process automatic and apply the weights to multiple relevant variables at the same time?

That works for numerical variables, but is not doable for a data matrix with categorical variables: Here you cannot multiply the original data matrix, but need to count (aggregate) first and multiply the results with the aggregated weights in the respective subgroups. You can do that manually, but it is extremely laborious and error-prone. In demoscopic research, you usually switch weighting on at the beginning of the analysis, both for counting categoricals (e.g. sex or age group) AND for aggregating numericals (e.g. mean or median income or age).

[TarandeepKang]

Hi team, I would just like to add my two cents here. I agree that a sensible first step would be to just add a box to each analysis where a weight variable could be inserted. That would be great for people who already know what their weight should be, such as those with large national survey datasets, where the weights are predefined.

A great second step would be to enable the creation of weights in Jasp. There are two ways people usually do this using raking (also known as iterative proportional fitting) and propensity score matching*. These are both useful when we want to make our survey sample more closely aligned to the general population in some way. I might apply weights or try to match for age or gender (alone or in combination) for example. This is great for those of us who don't have predefined weights and need to make our own. There are many packages for raking with notable ones being mipfp and ipfp. The similarities, differences, and use cases are discussed helpfully by Lovelace and Dumont. Propensity score matching can be accomplished using linear regression or random based methods a great one-stop shop for this is matchit, These would cover common ways of creating our own weights and both of these are available also in SPSS, the resulting weights would then be dragged into the above mentioned weight box. The most comprehensive (and difficult) step would be adding a new survey analysis module probably based around Lumley and his survey package. This would allow for analyses with complex designs, and word meet everyone's gold standard needs. I would however argue that my suggestions in one and two above should be made the absolute priority, because even without a full-fledged survey module, we could accomplish most analyses with self collected data and national datasets.

• Propensity score matching is widely used but not always recommended so another approach such as coarsened exact matching showed also be included to give a more robust alternative

Some literature on the approaches I have discussed:

Barthélemy, J., & Suesse, T. (2018). mipfp: An R Package for Multidimensional Array Fitting and Simulating Multivariate Bernoulli Distributions. Journal of Statistical Software, 86, 1–20. https://doi.org/10.18637/jss.v086.c02 Blocker, A. W. (2022). ipfp: Fast Implementation of the Iterative Proportional Fitting Procedure in C (1.0.2) [Computer software]. https://cran.r-project.org/web/packages/ipfp/index.html Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007). Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference. Political Analysis, 15(3), 199–236. https://doi.org/10.1093/pan/mpl013 Ho, D., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt: Nonparametric Preprocessing for Parametric Causal Inference. Journal of Statistical Software, 42, 1–28. https://doi.org/10.18637/jss.v042.i08 Iacus, S. M., King, G., & Porro, G. (2012). Causal Inference without Balance Checking: Coarsened Exact Matching. Political Analysis, 20(1), 1–24. https://doi.org/10.1093/pan/mpr013 King, G., & Nielsen, R. (2019). Why Propensity Scores Should Not Be Used for Matching. Political Analysis, 27(4), 435–454. https://doi.org/10.1017/pan.2019.11 Lovelace, R., & Dumont, M. (2016). Spatial microsimulation with R. CRC Press, Taylor & Francis Group. Lumley, T. (2004). Analysis of Complex Survey Samples. Journal of Statistical Software, 9, 1–19. https://doi.org/10.18637/jss.v009.i08 Lumley, T. (2010). Complex surveys: A guide to analysis using R. John Wiley. Zhao, Q.-Y., Luo, J.-C., Su, Y., Zhang, Y.-J., Tu, G.-W., & Luo, Z. (2021). Propensity score matching with R: Conventional methods and new features. Annals of Translational Medicine, 9(9), Article 9. https://doi.org/10.21037/atm-20-3998

[Ed-Houghton]

Hi team, I would just like to add my two cents here. I agree that a sensible first step would be to just add a box to each analysis where a weight variable could be inserted. That would be great for people who already know what their weight should be, such as those with large national survey datasets, where the weights are predefined.

A great second step would be to enable the creation of weights in Jasp. There are two ways people usually do this using raking (also known as iterative proportional fitting) and propensity score matching. These are both useful when we want to make our survey sample more closely aligned to the general population in some way. I might apply weights or try to match for age or gender (alone or in combination) for example. This is great for those of us who don't have predefined weights and need to make our own. There are many packages for raking with notable ones being mipfp and ipfp. The similarities, differences, and use cases are discussed helpfully by Lovelace and Dumont. Propensity score matching can be accomplished using linear regression or random based methods a great one-stop shop for this is matchit, These would cover common ways of creating our own weights and both of these are available also in SPSS, the resulting weights would then be dragged into the above mentioned weight box. The most comprehensive (and difficult) step would be adding a new survey analysis module probably based around Lumley and his survey package. This would allow for analyses with complex designs, and word meet everyone's gold standard needs. I would however argue that my suggestions in one and two above should be made the absolute priority, because even without a full-fledged survey module, we could accomplish most analyses with self collected data and national datasets.

Some literature on the approaches I have discussed:

Barthélemy, J., & Suesse, T. (2018). mipfp: An R Package for Multidimensional Array Fitting and Simulating Multivariate Bernoulli Distributions. Journal of Statistical Software, 86, 1–20. https://doi.org/10.18637/jss.v086.c02 Blocker, A. W. (2022). ipfp: Fast Implementation of the Iterative Proportional Fitting Procedure in C (1.0.2) [Computer software]. https://cran.r-project.org/web/packages/ipfp/index.html Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007). Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference. Political Analysis, 15(3), 199–236. https://doi.org/10.1093/pan/mpl013 Ho, D., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt: Nonparametric Preprocessing for Parametric Causal Inference. Journal of Statistical Software, 42, 1–28. https://doi.org/10.18637/jss.v042.i08 Lovelace, R., & Dumont, M. (2016). Spatial microsimulation with R. CRC Press, Taylor & Francis Group. Lumley, T. (2004). Analysis of Complex Survey Samples. Journal of Statistical Software, 9, 1–19. https://doi.org/10.18637/jss.v009.i08 Lumley, T. (2010). Complex surveys: A guide to analysis using R. John Wiley. Zhao, Q.-Y., Luo, J.-C., Su, Y., Zhang, Y.-J., Tu, G.-W., & Luo, Z. (2021). Propensity score matching with R: Conventional methods and new features. Annals of Translational Medicine, 9(9), Article 9. https://doi.org/10.21037/atm-20-3998

I very much agree with this comment. Weighting of case variables will open up the potential user-base to very many more fields of social and behavioral researchers.

It would be wonderful if this could be prioritized - this thread has been going for several years without any movement on this issue.

[EJWagenmakers]

Yes, this has been on our list for a long time. I agree it would be a great addition. EJ

[tomtomme]

duplicates (might contain additional information)

45

84

395

2018

[eplutzer]

One more person to indicate that the one and only thing preventing adoption of JASP is the inability to apply survey weights to common procedures such as descriptives (means, proportions, etc), crosstabs, and linear models (OLS regression, ANOVA, logistic/probit regression).

For the design team, an accurate and thorough explanation can be found at: https://stats.oarc.ucla.edu/r/seminars/survey-data-analysis-with-r/

I should add to some of the commentary a few observations:

1. Nearly all government surveys (health, employment, crime, agriculture, etc.) require design weights to account for unequal probability of recruitment. Just today, a student was working with a health surveillance study whose unweighted descriptives implied that the US adult population was 57% female and 20% Black -- both due to deliberate design features of the survey. Descriptives of every other variable in the dataset will be biased if those variables are correlated with sex or race.
2. Accounting for design weights is not simply a matter of removing bias from statistical estimates. The standard errors need to be adjusted as well (the effective sample size goes down when you correct for designed imbalance in the sample.
3. Most polls, academic surveys and government surveys also suffer from non-response. The public use data sets always include a weight variable that adjusts the design weights to also account for differential non-response (e.g., young people declining to participate at a higher rate than older people).
4. There are also adjustments for attrition in longitudinal surveys.

The good news is that there are many available fixes in R. The simplest run an initial command that points to the variables in the data set that are used in the adjustment process. Minimally, this is the final "analysis weight" that occupies a single column. (Note: One commenter noted that these weights typically have a mean of 1.00. In many economic, health and education datasets, the weights are calibrated to sum to the target population -- e.g., the weight would have a value of 50,000 for a respondent with a design-based probability of selection of 1 in 50K. Luckily, it does not matter. All weighing adjustments will yield the same estimates regardless of the scaling of the analysis weight).

The bottom line is that JASP will, 100% of the time, be using a biased parameter estimator for analyses simple and complex whenever survey weights are ignored. The standard errors will be biased as well (typically too small, leading to biased p-values and too many asterisks). Sometimes the bias is small, but there is no way to know without comparing weighted and unweighted analyses. (and its useful to remember that bias is not a feature of the dataset but of the estimator for a specific parameter).

Weighting has been available in SPSS and SAS since the 1970s. That's long before tractable analyses for SEM, network analysis and Bayesian estimation. It's unclear why this issue, first raised in 2017, is not a priority.

[tomtomme]

@vandenman maybe something for 0.20? But at the same time, others ask since years for missing-value handling stuff (FIML, ,MCAR, MAR, MNAR and stuff)....

@eplutzer it was probably not implemented earlier because JASPs goal in the past was not to be something to replace SPSS completely. It was catered towards psychology and its users and also promoted bayesian stats. Now the goal has shifted a bit and it seems viable that JASP at some point in the future could replace SPSS completely. I guess this was just not thought to be possible in the past.

Disclaimer: All that is my outside obervation as a heavy JASP user since 2018. I am not from the core team nor am I a developer. Just helping here to communicate and sort all the issues and requests.

[dschmuecker]

@eplutzer wrote "One commenter noted that these weights typically have a mean of 1.00. In many economic, health and education datasets, the weights are calibrated to sum to the target population -- e.g., the weight would have a value of 50,000 for a respondent with a design-based probability of selection of 1 in 50K [...]" In practical demoscopics we usually use a weighting variable (which usually has a mean of 1 over the whole sample) and a separate projection variable (which multiplies the mean by some fixed value, e.g. 50,000). As you noted, the result is the same as in your suggestion, but it might make programming and handling easier to conceptually separate both aspects. I am dreaming of a master switch letting the user choose between "non-weighted", "weighted" or "projected". However, this can also be easily implemented by having two weighting variables in the dataset, one without projection, one including the projection.

[EJWagenmakers]

So we are about to start implementing something like this (well, @vandenman is), courtesy of support from the Competens Foundation in the Netherlands. One option is to add a weight variable to most analyses. Another option is to add a separate "Survey" module that essentially implements the R package "survey" (https://cran.r-project.org/web/packages/survey/index.html). Any thoughts on this?

[dschmuecker]

@EJWagenmakers, thanks for the good news! As a selfish user, I vote for the first option, "to add a weight variable to most analyses", because (a) it will probably be easier to handle compared to the "survey" package and (b) if you start working on a weighted dataset, you will need weighting in all steps of descriptive analysis and in graphical visualisations.

[djstepz1]

I concur, the first option is preferred, but any attempt at implementing this feature is greatly warranted. As someone who uses survey data, most datasets have a weight variable included, though not all. It would be nice to have the ability to firstly use a weight variable, but also to create/adjust a weight variable from the data - so the user can adjust data to their requirements for use in their analyses.

On Thu, 6 Jun 2024 at 13:33, Dirk Schmücker @.***> wrote:

@EJWagenmakers https://github.com/EJWagenmakers, thanks for the good news! As a selfish user, I vote for the first option, "to add a weight variable to most analyses", because (a) it will probably be easier to handle compared to the "survey" package and (b) if you start working on a weighted dataset, you will need weighting in all steps of descriptive analysis and in graphical visualisations.

— Reply to this email directly, view it on GitHub https://github.com/jasp-stats/jasp-issues/issues/73#issuecomment-2152294596, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIUDOQI5A25CZYE63GKRVUTZGBJLHAVCNFSM4GDQI3X2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJVGIZDSNBVHE3A . You are receiving this because you commented.Message ID: @.***>

[eplutzer]

I guess I am confused as I don't understand the first option. Simply adding a weight column to a dataset does nothing by itself. That variable needs to be called into use in each analysis.

Perhaps what you mean is to produce weighted results by default? That would imply that all datasets have a weight variable, but many would weight by a benign column of 1s.

That would be a big improvement. It assumes, however, that each procedure weights properly and consistently. If not, it might lead to identical statistical estimates but different standard errors, confidence intervals and p-values from one procedure to the next. This happens in Stata., which I know somewhat better than R.

Procedures that treat weights as survey design weights (inverse probability of selection weights) yield the correct SEs:

mean income educ age employed [pw=weight_variable_name] regress income educ age employed [pw=weight_variable_name]

Procedures that treat them as simply "analytical weights" yield biased measures of uncertainty. mean income educ age employed [aw=weight_variable_name] regress income educ age employed [aw=weight_variable_name]

I don't know how weights are implemented in every individual procedure called up by JASP and I suspect that some will treat weights analytically simply to get the coefficient estimates correct, while others will also correctly calculate the SEs and CIs.

The advantage of using the survey package is it ensures that every procedure is calculating measures of uncertainty based on the same model.

I suspect that, the survey package approach, will actually be easier to implement consistently. For any dataset on file, you would need to write the surveydesign command once to run on request or by default. Most sample datasets would have extremely simple designs. But users with more complex data would have access to all the other options. For example, individual users who have clustered data (e.g., random sample of 30 students from each of 30 randomly selected schools) could designate the school id as the primary sampling unit once and be done. And then applications like surveymean or svyglm etc would also get correctly adjusted confidence intervals and significance tests.

But either approach would be welcome!

In any of these scenarios, a user could either designate a new weight variable that they created (as djstepz1 suggests), or modify an existing one. So that flexibility would be present in either option.

Perhaps I've misunderstood. But I

[djstepz1]

Very good point. I guess to address this issue, a simple tutorial video (which developers of Jasp have been very good at creating to date) would help. Having the feature available for me would be the primary concern, how to use it is another matter that could easily be addressed after I’d say.

On Thu, 6 Jun 2024 at 20:07, eplutzer @.***> wrote:

I guess I am confused as I don't understand the first option. Simply adding a weight column to a dataset does nothing by itself. That variable needs to be called into use in each analysis.

Perhaps what you mean is to produce weighted results by default? That would imply that all datasets have a weight variable, but many would weight by a benign column of 1s.

That would be a big improvement. It assumes, however, that each procedure weights properly and consistently. If not, it might lead to identical statistical estimates but different standard errors, confidence intervals and p-values from one procedure to the next. This happens in Stata., which I know somewhat better than R.

Procedures that treat weights as survey design weights (inverse probability of selection weights) yield the correct SEs:

mean income educ age employed [pw=weight_variable_name] regress income educ age employed [pw=weight_variable_name]

Procedures that treat them as simply "analytical weights" yield biased measures of uncertainty. mean income educ age employed [aw=weight_variable_name] regress income educ age employed [aw=weight_variable_name]

I don't know how weights are implemented in every individual procedure called up by JASP and I suspect that some will treat weights analytically simply to get the coefficient estimates correct, while others will also correctly calculate the SEs and CIs.

The advantage of using the survey package is it ensures that every procedure is calculating measures of uncertainty based on the same model.

I suspect that, the survey package approach, will actually be easier to implement consistently. For any dataset on file, you would need to write the surveydesign command once to run on request or by default. Most sample datasets would have extremely simple designs. But users with more complex data would have access to all the other options. For example, individual users who have clustered data (e.g., random sample of 30 students from each of 30 randomly selected schools) could designate the school id as the primary sampling unit once and be done. And then applications like surveymean or svyglm etc would also get correctly adjusted confidence intervals and significance tests.

But either approach would be welcome!

In any of these scenarios, a user could either designate a new weight variable that they created (as djstepz1 suggests), or modify an existing one. So that flexibility would be present in either option.

Perhaps I've misunderstood. But I

— Reply to this email directly, view it on GitHub https://github.com/jasp-stats/jasp-issues/issues/73#issuecomment-2153227719, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIUDOQJIYIRB7USKCZCLNEDZGCXO5AVCNFSM4GDQI3X2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJVGMZDENZXGE4Q . You are receiving this because you commented.Message ID: @.***>