Sampling, Crowd-Sourcing & Reliability - Salesses...and Hidalgo 2013

[jamesallenevans]

Salesses, Philip, Katja Schechtner, and César Hidalgo. 2013. “The Collaborative Image of The City: Mapping the Inequality of Urban Perception.” PLoS ONE 8(7):e68400. doi:10.1371/journal.pone.0068400

[sunying2018]

I have a question about the data implied in this paper. This paper mainly demonstrate the randomness of the images took from the 4 cities, but for the perception data, it does not talk much on the users participated in this experiment. I am wondering how to ensure the randomness of users since some demographic characteristics or personal preferences mainly influence their selections. Besides, if there are some landmarks presenting in some imagines that can be identified, how to prevent the situation that people make their selections based on their prior knowledge or stereotypes?

[rachel-ker]

I really enjoyed reading this paper; the research method was very interesting! Similar question as @sunying2018 about the people who were voted. The paper did not specify how they were recruited or invited to participate, but presumably the participants are found online. I was wondering for online crowdsourcing of response, have there been any innovative ways of reaching populations that are not typically represented online / do not have access to the internet ?

[heathercchen]

This article is mind-refreshing in many aspects. The mechanism of rating city perception reminds me of the famous Zuckerburg-ranking-girls system. I have two questions regarding the data collection process of this experiment:

1. The spots the authors choose to capture images in two Austrian cities are sparse and not evenly distributed across the whole area (As Figure 1 shows). So is there any possibility that the data itself is biased and cannot represent the overall city perceptions in Linz and Salzburg?
2. I am curious about why the authors include "uniqueness" in their measurement of city perceptions (by asking participants "Which place looks more unique?"). Though in later sections, the uniqueness of city perceptions seems to have relationships with its social class (see p.8). In my opinion, uniqueness is often perceived as to be more related to one's own aesthetical feelings, which is largely depended on an individual's own thoughts. Also, the authors do not include uniqueness in their prediction of homicides. So what it the point of having uniqueness as a scale in this study?

[alakira]

The study is really interesting in terms of the way of using cloud sourcing to find the ambiguous 'Image of the city'. But I have two questions on sampling.

1. Since Google Street View does not capture every single street (e.g. it does not capture some narrow side road), couldn't there be a sampling bias in the set of pictures?
2. Moreover, since 'Image of the city' might depends on people's experiment, isn't it better to weight those pictures with, say, traffic volume?

[bjcliang-uchi]

I don't feel quite convinced by this approach. My questions are: 1) Figure 3 on Page 3 shows examples of how places are identified with different urban perceptions (class, unique, safety, etc.) However, it seems that first, light matters--perceptions, which should be fixed throughout time, is subject to the flexible weather. Second, the graders/ viewers seem to have a preference over the European-style structures and the presence of Church seems particularly increase the perception of safety. In other words, there might be a priori bias that "White Christian communities" are safer. 2) Also, it is hard to tell how this approach can be generalized to more cities.

• First, the beauty of the natural scenery, as well as the major industries in a city/ town may lead to perception bias.
• Second, since we normally have detailed city-level data about crime rates, demographics, and health, why bother using this much more complicated and less informative method? Particularly, As shown in Figure 5, the predictions clearly miss a few homicide clusters and mispredict a few others.
• Third, as mentioned before, the tastes in favor of European style--or, at least, the graders' lack of cultural experience in other countries--make it hard to extend this method to other places.

[ccsuehara]

I really liked reading this paper, it is a great adherence to studies in urban inequality. I was wondering how they selected the parts of each city they wanted to show, furthermore, did they cover all the possible places available at street view at the time? Would it be helpful to do so, given that the classification of images was manual and maybe they had scarce resources to do this task?

Also, it would be interesting to try this method in cities from developing countries. Besides capturing the inequality of a city, it could also capture the dimension of, for example, the total safety.

[iamlaurenbeard]

I appreciated this paper's attempt to gather various individuals' perceptions on the presented images. While the authors did gather a large number of unique participants, they note that those participants only self-reported age and gender. I assume that this is partially due to the inability to procure a great deal of information from online participants (i.e. SES, race, ethnicity, etc.). Are there common ways that researchers successfully gather more robust demographic information from respondents?

[acmelamed]

Similarly to the questions raised by others in this thread, I was also struck by the lack of attention paid to the demographic dimensions of the respondents polled for this study, Specifically, I am curious as to how researchers studying this subject -- assuming they are able to access this information -- might take demographic information such as race or class into account for a consideration of what makes a certain city image appear "safer" (for instance) to one respondent as opposed to another. How would this additional dimensionality be best incorporated into the methodological framework used in this article?

[sanittawan]

It is quite a creative approach to study the perception of cities using images. To me, the paper falls short when using the NYC crime data to help validate the results. If the researchers were to have access to crime data of other cities in the studies, would the results be similar to that of New York City? I am not convinced by the fact that the authors only point to New York City to show that images can capture the perception of inequality.

[Dominiquo]

This paper was published in 2013, and though I know image recognition was pretty good at that time, it has gotten quite a bit better for much less computation. If we were to revisit this paper and attempt to use object recognition, scene segmentation, or honestly just color analysis, what do we think might be indicative of what creates these perceptions? Ultimately, if we really assume people have a sense for this that is built from the image alone, there should be physical indicators from the picture. I think this data should be revisited using more CV techniques.

[ziwnchen]

This paper did a very interesting work in terms of urban perception and social outcomes. Here are some of my questions when reading the paper:

(1) Research Design: why use the specific three questions (safety, class, uniqueness)? Why choose these specific four cities (NYC, Boston, Linz, Salzburg)? Is there any theoretical support for the validity of the questions? Will the city selection reduce the generalization power of this research conclusion? (2) Survey validity: I'm wondering if every participant of the survey has the same level of understandings in terms of the three concepts. Saftey, upper-class and uniqueness sounds to be rather abstract and arbitrary concepts that may be interpreted differently by different people. How could the researcher make sure that what they measure is what they want? Although they look at potential confounders like age, gender and location, there should still be more examinations to guarantee the construct validity

This paper also reminds me of several other works that use google street view data to predict neighborhood demographics composition,, travel patterns or car accident... Is there any theory about what kind of information is contained in the static street photos? Since surprisingly rich information could be retrieved from the street view data, should access to this kind of data be restricted under privacy concern(e.g., high-resolution satellite photo is usually not publicly available)?

[luisesanmartin]

I'd like to build on from the questions posed by @ccsuehara and @heathercchen . The first question that arose to my mind was how did the researchers ensured the representativeness of the sample of images they use -- as I recall, the methods for collecting them are indeed mentioned but not how they were selected. Similarly, I also wondered why these particular cities were selected for this research. Boston and New York (especially New York) seem to be much more well-known cities than Salzburg and Linz, and it's also salient how the sample size of images for the first two was higher, according to Figure 1. Also, I wonder if any image filtering methods were conducted to try to standardize the brightness, pixel density, etc. These image characteristics could also affect how the images are perceived, and it could especially worrisome if they were generated by different means -- recall that in this case the US images were obtained from Google Street View, while the European ones were generated manually onsite, according to the information from the Data section.

[yaoxishi]

This paper is very interesting to read as it develops a new method to evaluate the inequality of a city, and the validation methods show that this method is to some extent reliable. But there is one major thing that I am not very convinced by this paper. The number of cities selected in the study is too small, which would introduce lots of randomness of the results, also, the paper seems doesn't justify the reasons why they use this sample. Also, the cities used in the study are in all developed western countries, which may limit the external validity of the results. I am curious to see whether this method could be applied to a wider range of cities.

[xpw0222]

This is really an interesting paper! This is how I have expected "computational social science" to be! I am wondering, is that possible to train a machine learning model (or whatever more advanced model) based on data from this research, and apply it to Google map images from other cities? In this way, we can draw the inequality map of any city we want. That might be cool.

[luxin-tian]

I think @sunying2018 and @rachel-ker brought forward a really important question concerning the subjective bias introduced by the participants who vote for the street images: if some participants made their choice based on their pre-existing experiences or perceptions, there can be measurement error, which would threaten the reliability of the annotations. Even though it might not be a significant problem in this research, it constitutes a general problem in other research that engages crowdsourcing labeling data.

P.S. I have some experience working with the same data set used by this paper - Place Pulse at MIT - and used a very simple Elo Rating algorithm to calculate the perception scores for 56 cities around the world. This is exactly the rating method mentioned by @heathercchen. I put this simple project on this website, and interactive maps are included. :)