Open ascott opened 3 years ago
ascott
commented
3 years ago there are only 5 options for this selection and they don't seem to share a lot of overlap. would it be common for a method to use multiple recruiting methods? perhaps we should just update the directions to indicate a single selection. what do you think @jesicarson @pscully
plscully
commented
3 years ago @ascott @jesicarson This is a tricky one. It’s probably true that many -- if not most -- methods use only one recruitment method. But we don’t really know, which is why we tried to make this a ranked question. For ex., some methods appoint some participants and use random selection to recruit the balance. (e.g., the Irish Citizens' Assembly use random selection for citizens and also appointed a few national legislators)
Another reason for allowing more than one choice is that many of the currently popular methods use either random selection or a stratified random sample. The project that we’re about to do for the EU requires us to select all cases in our dataset that include one and/or the other. The way this is currently set up doesn’t allow us to do that in a single search, at least I don’t think it does.
BTW, I don't feel strongly about this one way or the other. -- @vhurtadol @Mattygryan - Do you have time to weigh in?
Mattygryan
commented
3 years ago Hey @ascott @jesicarson @plscully @vhurtadol. My intuition is that multiple recruitment happens in cases in practice with compromise but may be less relevant to the abstract ideal version in 'methods'. There are some that definitely use mutliple recruitment methods like Pat says (combination of appointing people; SRS; electing some delegates maybe). Also i think this question betrays a bit our bias towards the minipublic format for PDIs which is premised on finding a descriptively representative subsample and respondents are probably thinking only about the 'participants' in terms of the 'ordinary' citizens. Alanna makes a good point that the first three options are pretty exclusive.
Do we have any record of the reason we chose to ask this question this way?
plscully
commented
3 years ago @Mattygryan @vhurtadol @ascott -- Looking back at notes from when we were building this data model reminds me that questions related to participant recruitment & their demographic profile was one of the most difficult things to work through. Paolo, Robert, Graham and others all pitched in, but even with all that brainpower, we still may not have gotten it right. Pages 13-8 of this document provides a good record of what we grappled with. .... And this provides a messier version of how we worked through these questions.
jesicarson
commented
3 years ago @ascott is this a bug, or has the multi select never been implemented for this field? wondering if we have we collected any responses for multiple recruitment methods (ie: before the bug was introduced), or not? And, if not, would adding the multi-select at this point result in messy data? cc @plscully @Mattygryan @vhurtadol
Mattygryan
commented
3 years ago @plscully that discussion between Paolo and Robert was a good read... I think suggests to me surveys are better written by peer review than committee :). I had lost the context but now better appreciate it is a question that emenates from earlier logic. I guess I'd probably think muti-select is theoretically the right option to have but 'select one option only' is cleaner and I can see why knowing the 'primary' mode of subsample selection could be useful also. Ranking seems like a compromise that would work in theory but probably is overall less useful inormation - quantitative researcher would likely just analyse the top selection anyway and if they needed to explain a method that used multiple selections they could get that info from the long method description. Most of the time there are multiple selection methods its hard to imagine how you could rank them to me anyway. @jesicarson point about continuity is also important. Tricky question then so i think no obvious correct option. I think try and maintain continuity but if we lost the ranking i wouldn't cry.
ascott
commented
3 years ago @ascott is this a bug, or has the multi select never been implemented for this field? wondering if we have we collected any responses for multiple recruitment methods (ie: before the bug was introduced), or not? And, if not, would adding the multi-select at this point result in messy data? cc @plscully @Mattygryan @vhurtadol
@jesicarson the field being a single select is not a bug, it has always been in this format. what is a bug is that the informational text suggests it is a ranked select instead of a single select. to change this field to a multi select we would have to do a db migration to get the data we have into the correct format as well as update the ui to accommodate multi/ranked selection
Mattygryan
commented
3 years ago In that case i think just change the text to reflect single select?
plscully
commented
3 years ago I am OK with changing the text to reflect single select. @vhurtadol Please change this in the codebook. Thank you everyone for helping us sort this out!
plscully
commented
3 years ago @Mattygryan @vhurtadol Now that we have decided to stay with allowing users to select only one value for method of participant recruitment, should we also delete "stratified random sample" option and manually re-code all those responses to "random sample"? or should we keep both values in the form? ... I think this decision hinges on how you feel about the exchange between Paolo and Robert on pages 13-14 of this document. I don't know enough about this to have strong feelings one way or the other.
Mattygryan
commented
3 years ago I'm more with Paolo. The actual instance of true random sampling is rare. More usually its quota sampling - i.e. we have x number of characteristics we want to represent and we will keep inviting people until we get that representation. Or Stratification (so a partial randomisation). Also Paolo made the point that the sampling frame is usually full of error but that is probably by the by. Robert defends juries as an example of true random selection which is a good point so there are exceptions. If we want to collapse it would make sense to have an entry like 'random or quasi-random sample (inlcuding stratified and quota sampling)'
plscully
commented
3 years ago @Mattygryan @ascott @jesicarson OK - I'm sold! I like the idea of combining the two "random" values into a single value of "random or quasi-random sample (inlcuding stratified and quota sampling)'. However, if we do this I highly recommend that we include the detail we originally wanted to provide that gave examples for each value. You can see that detail in the parenthetical text that is part of the list of values at the top of p.7 in this document. If our design can accommodate that, I will take another pass at reviewing the "eg" statements. We would also need to coordinate this with re-entering the values for a bunch of cases. As of today the values are:
Mattygryan
commented
3 years ago @plscully I'm glad i convinced someone of something but that worries me also. given there are 91 RS responses it might be worth 10 mins of my time to go through and check there is not a clear difference between those and SRS (i.e. do a sense check so we are not losing any info. I should be able to do that before Friday
vhurtadol
commented
3 years ago I am OK with changing the text to reflect single select. @vhurtadol Please change this in the codebook. Thank you everyone for helping us sort this out!
We don't have to update the text in the codebook because I only indicated "Method used to recruit a limited subset of the population". But, I am going to include all the values and labels in the codebook over the weekend.
@plscully I think @Mattygryan suggestion makes lots of sense. I doubt we will get any cases of true random sampling, but thanks Matt for checking! Also, if we combine the two randoms into one, we can still get more details on the recruitment in the case narrative (the body).
jesicarson
commented
3 years ago Hi folks - the text has been updated to reflect single select. we're tracking suggestions for data model changes as usersnaps here.
Mattygryan
commented
3 years ago I just checked what the first 10 or 12 cases i could find that have chosen random selection say in their participant selection description and 90% of them are clear they are actually using either SRS or quota sampling so I think that vindicates the move and we are actually cleaning up here. So clearly there was already misreporting.
plscully
commented
3 years ago I just checked what the first 10 or 12 cases i could find that have chosen random selection say in their participant selection description and 90% of them are clear they are actually using either SRS or quota sampling so I think that vindicates the move and we are actually cleaning up here. So clearly there was already misreporting.
@Mattygryan Thank you for looking into this.
@jesicarson When you have time, can you please suggest a workflow & timeline for getting this done? As I noted above, the task is to combine the two "random" values into a single value of "random or quasi-random sample (inlcuding stratified and quota sampling)'. And if possible, we should add the detail we originally wanted to provide that gave examples for each value. You can see that detail in the parenthetical text that is part of the list of values at the top of p.7 in this document. If our design can accommodate that, I will take another pass at reviewing the "eg" statements. Once you have a timeline for that part of the work, we can ask Jaskiran to manually revise the responses for the 91 random sample values, and make them "stratified .." We would next change the value label on what is now "stratified" to the new "random or quasi-random sample (inlcuding stratified and quota sampling)' label.
jesicarson
commented
3 years ago @plscully we should chat about what makes sense for a review of the outstanding suggested model changes; the plan so far has been to collect a batch of them and implement at the same time, rather than one off model changes (for dev efficiency). that said, if this particular field is very urgent, we can re-evaluate and prioritize it.
For now, we could add an info icon and info modal (pops up when you click the little "i" icon) that is effectively a glossary for these values. ie:
Examples of recruitment methods
plscully
commented
3 years ago @jesicarson I'm available to chat any time. In the meantime, I agree that our default approach should be to review and implement batches of changes to the data collection models. However, I think we should make an exception in this instance because this set of values are a key part of the case identification process for the Horizon 2020 project. That is why I started looking so closely at this module. The H2020 research is limited to looking at cases that use random & quasi-random participant selection, so we will be adding a lot of new content related to this feature. Also, there are 11 stages (or “work packages” in Euro research-speak) to the project, and Andre & Mark are leading Work Package 1. Although the project doesn’t officially begin until March, we are trying to make progress ASAP so as not to slow the subsequent stages.
Ideally, the values would look like those we use in the “Approach” module. But if making that change is too inefficient a use of development hours, we could only include the boldface type in the list below, and go with the icon and info modal for the “e.g.” text. BTW, even though the header to the info modal would say “examples,” it will be good to include the “e.g.” within the list items to reinforce that these are only examples.
Given all this, here is what the new text could look like after Jaskiran recodes the current values. Examples of recruitment methods
The new “Random or Quasi-Random Sample” should include all the cases that are currently tagged for the (1) random sample and (2) stratified random sample.
plscully
commented
3 years ago @vhurtadol I'm looking at the codebook for the POLITICIZE data set and would like your opinion on whether the two selection methods they suggest, that is: - Purely random -- Participants are purely randomly drawn from the population. - Based on representative criteria -- At some point in the recruitment process, representative criteria are applied in the selection of participants ... could both be included within the new combined value suggested by @Mattygryan , that is Random or Quasi-Random Sample: e.g. includes stratified and quota sampling
vhurtadol
commented
3 years ago @vhurtadol I'm looking at the codebook for the POLITICIZE data set and would like your opinion on whether the two selection methods they suggest, that is: - Purely random -- Participants are purely randomly drawn from the population. - Based on representative criteria -- At some point in the recruitment process, representative criteria are applied in the selection of participants ... could both be included within the new combined value suggested by @Mattygryan , that is Random or Quasi-Random Sample: e.g. includes stratified and quota sampling
Hi @plscully, I am not sure. I think the purely random clearly is part of our category of "random or quasi-random" but the based on representative criteria is very vague. I found this article that explains their classification:
In a vast majority of deliberative mini-publics (DMPs) 75 per cent, organises applied representation criteria correcting for the underrepresentation of certain social groups. Some country experts explained in their case descriptions (available on the website) that the implementation of these representation criteria vary quite a lot across DMPs. In some cases, the technique is stratified random sampling. In others, there are quotas for some under-represented categories of the population. What also varied is the implementation of these representation criteria. In some cases, organisers of the DMP start with open invitation and apply representation criteria to recruit in a second phase. In other instances, the selection process starts with selecting a representative sample of the population applying representation criteria, and then whoever replies positively can join. Yet, what is clear is that, in line with our definition above, we do not count as cases of DMP bodies open to any citizens who are willing to participate without any selection by organisers
So, I think the based on representative criteria includes diverse recruitment methods and while some are quasi-random, the "invitation" for example is outside of this category.
jesicarson
commented
3 years ago @plscully - @vhurtadol and I were chatting on slack about an interim solution. She can include a note in the codebook with a tip for users to recode the field manually in the CSV, combining the two values. Vero - does that sound right? Pat - what do you think?
vhurtadol
commented
3 years ago @plscully - @vhurtadol and I were chatting on slack about an interim solution. She can include a note in the codebook with a tip for users to recode the field manually in the CSV, combining the two values. Vero - does that sound right? Pat - what do you think?
@plscully Yes! I told @jesicarson that some datasets do this when they have historical data (like us!). We can include a note saying: Coding strategy for this variable varied after (DATE). If using information for cases after this date, we suggest re-coding "random" and "stratified random" samples into one single value named "random or quasi-random sample".
plscully
commented
3 years ago @vhurtadol @jesicarson Thank you -- This is a smart way to deal with this. I also agree that it should be a short-term solution.
Sender: pscully@clearviewconsultingllc.com Comment: The directions allow users to rank up to three techniques, but the interface only allows you to select one Open #1018 in Usersnap Dashboard
Browser: Firefox 84 (Windows 10) Referer: https://participedia.net/method/5507/edit?full=1 Screen size: 1760 x 1100 Browser size: 1760 x 949
Powered by usersnap.com