Same old baffling analysis problem

[janedupuy]

B: "For descriptive purposes it sounds like you’ll have timely and precise data. It strikes me that in order to really understand the causes (and potential consequences) of Stingray use, you will have to more carefully articulate how and what other data you’ll collect. I suspect that the impact of your research would be significantly strengthened with more attention to selectivity in who downloads the app (and who doesn’t) and how that might vary in relation to socio-demographic characteristics and geography. I’m also concerned that no evidence of a Stingray surveillance is a pretty muddled signal."

[janedupuy]

Thoughts on that as an answer? Would we be open to asking people to report race, level of formal education, and city?

[janedupuy]

Me:

The analysis question has been a conundrum, and we’re continuing to think hard about it. It occurred to us that we could ask users to answer a couple of questions about their demographic characteristics and city of residence before downloading the app. We’ll also have all of the non-threat level reports to tell us where the app is and isn’t being used in general, which as you mention is not great, but it may give us some leverage. Can you think of anything else that might help us on this front?

The most concrete form of analysis we’ve thought of - and I should probably clarify this in the proposal - is to compare Stingray detections to public records to check for unwarranted deployments. Here, any findings would have a conservative bias, so selection issues should be less of an issue.

[marvinmarnold]

I'm of the mindset that from a technical perspective, demographics don't matter. Ultimately each time-bucket (better word?) will either be green, red, or unmeasured. The reading doesn't change based on who owns the phone. That said, from acquisition perspective, we need to get people with a wide range of movement patterns to download which will correlate to demographic diversity. From an advocacy perspective, demographic diversity is essential.

[janedupuy]

Okay, now I'm rethinking this. At first I was thinking that since race and class characteristics would be good predictors for neighborhood, that data would help us to track what sort of geographic diversity we're getting. But if we're able to track that based on the regular green threat level reports, what additional information would this give us?

To take a stab at answering that:

On one hand, we won't have the tools (probably) to differentiate neighborhoods within tracts by race and class, even though they'll be varying dramatically. So we won't know anything about the block in Cincinnati where we got a red threat level reading - it could have happened on the one poor Latino block at the edge of a wealthy White Census tract, but we would only record the race and class characteristics of the tract as a whole and end up with an erroneous data point. If we know the race/class demographics of the user who recorded the detection, that could be a more reliable basis on which to draw conclusions.

On the other hand, we could get a bunch of readings from middle class White people who live or spend time in majority working class Black neighborhoods and then the reading would tell us the wrong thing again.

Basically, we are fucked. We're never going to be able to analyze this data. Unless you can think of a good reason to ask users for their demographic information, let's drop the idea.

[janedupuy]

Okay, one more thought. Is there any way to keep individual reports linked to demographic information we collect in the beginning, without saving IP addresses or anything else? If we did have race/class information, we might be able to create our own random/representative sample from WITHIN the data we collect, so that we only analyze a subset of it (for social science purposes - all of it would of course go onto the website).

[marvinmarnold]

Sure, we could ask them for demographic information, store it on the phone, and upload it each time along with the data. I'm not convinced this will be worth the effort though.

[janedupuy]

Okay, I'll just run it by B and see if she thinks it would solve all of our analysis problems. If not we'll drop it again.

[janedupuy]

I'll reopen this when I've heard back, since it's on hold till then.