So You Think You Can Data

[karthik]

At the unconf we're going to have a friendly data challenge (previously discarded names include Iron Data and Data With the Stars) between Hadley Wickham (@hadley) and Wes McKinney (@wesm). This session will be moderated by GitHub's own Arfon Smith (@arfon) and broadcast at a later time.

Actual time TBD: Either Thursday night (03/26) or sometime on Friday (03/27).

In this issue we'd like to solicit ideas for potential datasets and challenges.

cc @gvwilson @tracykteal

[wesm]

Sure to embarrass all involved =) looking forward.

[Ironholds]

The Netflix Challenge? ;)

[benmarwick]

How about this for a potential challenge & dataset: https://github.com/ropensci/unconf/issues/13 ?

[brianckeegan]

Something around long vs. wide data?

[jordansread]

How about all the state and federal data for phosphorous (commonly the limiting nutrient for harmful algal blooms)?

[CamDavidsonPilon]

Some interesting datasets, questions TBD:

• 200,000 Jeopardy Questions
• 10 million passwords and usernames
• Chipotle orders. It would be neat to combine this with their nutritional value data.

[rhiever]

Excited to see more R vs. Python competitions popping up!

I've been kicking around the idea of making a 20 questions game bot powered by a decision tree. Acquiring the data set is the tough one there.

[hadley]

Some thoughts:

• The dataset should be relatively clean already. I don't think it should be perfect, but if it's too messy, cleaning it up is all you'll see me and Wes do. (Which might be ok, but I think seeing a little bit of the whole analysis cycle would be better)
• It should only need relatively simple modelling, because I don't think modelling is a particular strength of mine. We don't want it to be a kaggle-like where it's all about a crazy complicated model, it should be more about getting some insight into the data.

[jcheng5]

I suppose "Data Idol" was too on the nose? ;)

[tracykteal]

We didn't actually think of that one! Maybe that will be the spin off.

[tracykteal]

Since it will be hosted at github, how about something with the github archive data https://www.githubarchive.org (although maybe too close to the GitHub Data Challenge) One idea might be to correlate repo tiltle/keywords with number of contributors. Do different domains/topics have more contributors? More contributors from different geographic locations? Diversity, referring back to #13? I'm not sure the archive has information on what programming language the repo is in, but domain/topic could also be correlated with language, and maybe filtered for domains of interest like 'ecology', 'genomics', 'archaeology', 'astronomy', 'economics'.

[tracykteal]

There's also this https://github.com/caesar0301/awesome-public-datasets

[gvwilson]

Arfon and I talked about using GitHub data - problem is, it's already pretty clean. Now, if it was stuff from CVS repositories, we'd have a game...

[dgrtwo]

I'll propose this United Nations voting data, containing all votes from the history of the United Nations General Assembly (since 1946). I've used this to teach dplyr/tidyr to students before and it's gone very well.

Pros:

• It is relatively tidy: organized into two tables, one with one row per country-vote pair with yes/no/abstain, and one with one row per roll call with a description of the vote and its topic.
• It's not completely clean, though, as it has redundant columns, a date variable that has to be parsed, rows to be filtered out, and topics spread across columns, so there are indeed cleaning operations one would have to start with
• Countries in the second half of the 20th century are more likely to be familiar to the audience than a domain-specific topic (e.g. genomics or ecology)
• Many insights are possible (across countries, across time, across topics) even when using only the simplest models.
• By combining with a mapping package, one could make some cool world map visualizations (e.g. cluster countries by voting patterns and see if they cluster geographically)

Cons:

• Conclusions might have a political component (e.g. concerning issues of modern international relations)

[wesm]

Couple notes

• I'm not actually very good at statistical modelling. Exploratory data analysis and visualization I can get by
• Friday 3/27 is better for me

[hadley]

I'm not that great at stat modelling either :)

I like @dgrtwo's idea