search

exercism / discussions

For discussing things like future features, roadmap, priorities, and other things that are not directly action-oriented (yet).
37 stars 5 forks source link

Group similar solutions / heatmap #65

Closed jessf closed 7 years ago

jessf commented 8 years ago

Forgive me if this has already been suggested / discussed.

I find the sheer number of solutions to a given exercise on exercism overwhelming to browse through, especially as many of them are extremely similar to one another. I end up clicking on 2-10 and then wandering off to do something else. What I really want to see is how people have taken different approaches to the same problem, not click on 50+ similar solutions to find the 1 or 2 unique ones.

I'm keenly aware exercism is not a competitve site and there should be no ranking/marking of submissions in a way that would undermine that. What would you think of a heatmap solution, where a static code analyzer would parse the solution and, if it's (say) 80% similar to another solution, group them together on the map. This way, one could read 1-5 examples from each group, making really easy to see a bunch of different approaches. It'd also be a quick way to see trends in how people solve problems.

Just an idea :)

Insti commented 8 years ago

Interesting idea. :+1:

clarkburbidge commented 8 years ago

+1 Maybe allowing community or submitter tags would help

kytrinyx commented 8 years ago

@jessf I love this idea—and would be absolutely delighted if we were able to use a combination of static analysis and machine learning to do this automatically.

In addition to being able to point people to solutions that are meaningfully different to yours, we could also point you to the ones that are similar to yours so that you can read the comments/discussions about them.

massich commented 8 years ago

I'm totally in.

I guess that the most brutal manner of doing so is to clone all the repos (maybe something like this). Then apply some NLTK to find tokens. From the tokens, it would be easy to select functions, etc.. then just do some answer clustering.

what you think?

kytrinyx commented 8 years ago

I think a slightly less brutal approach would be to suck the data into BigQuery.

A caveat: we don't have the solutions in git, unfortunately, just the exercises (test suites). The solutions are not publicly available. Everyone has copyright (by default) on the code that they've written—we should have put something into place to do a creative commons license or something, but I didn't think of it until it was too late. At least too late to fix history :)

massich commented 8 years ago

But if we are able to access the other users answers by web, we should be able to clone the git. Or if its done by exercism, then this could be in some sort of hock when uploading an answer. Anyhow, doing a web scrapper is also fine. I don't know how these snippeds might apply. But I'm sure we can figure out.

Any more thoughts?

Maybe some code analyzing tools can be applied (see quantified code)

kytrinyx commented 8 years ago

But if we are able to access the other users answers by web, we should be able to clone the git.

If the solutions were stored in git that would work. As it is now, it's stored in a blob in the postgresql database.

if its done by exercism, then this could be in some sort of hock when uploading an answer.

Yes, that would be fine. We can give access to whatever system we need to in order to make it happen. We have an API endpoint where, given the UUID of an iteration you can get the actual code.

I don't know how these snippeds might apply.

Yepp, once we know how we want to analyze it, I'm sure we can figure it out. I can also reach out to the author of the article, as well as the author of the GitHub gist if we decide to go down that route.

Maybe some code analyzing tools can be applied

Yes, I think that's probable.

jessf commented 8 years ago

When I suggested this, I didn't realize it would be quite so difficult sweatdrop I've been looking into solutions a bit--mostly programs that analyze code similarity for the purpose of preventing student plagiarism. When I was a computer science student, I think one of my professors ran all our solutions through MOSS (https://theory.stanford.edu/~aiken/moss/), a Stanford tool for doing just that. Unfortunately, it's closed source, but I think that is the right direction.

kytrinyx commented 8 years ago

The plagiarism approach is interesting. I wonder if there are any open source tools that take a similar approach.

massich commented 8 years ago

I've been playing with exercism a little. And think that that something really quick that would allow to explore other people solutions manually is a search bar.

Here's an example (I was working in c++ word-count)

I've used istringstream to break the sentence in words. Without wandering much of what would happen with a word like don't. On the other hand I've found lots of people using either boost::split or std::regex_iterator. To have a better control of what's a word.

Anyhow, if I was able to search any of the keywords: istringstream, regex_iterator, boost or boost::split i would get a grasp of the solution landscape.

jtigger commented 8 years ago

@matthewmorgan and I read through your idea, @jessf, this weekend and it spurred a lot of insights for us. We're pretty fired-up about what you're suggesting.

The goal of Exercism (AFAIK) is to create a space for conversation about code between programmers so that all involved get better.

A quality interaction is a sequence of give/receives... two or more people turning over a particular aspect a number of times.

We often see only one-half of that. Either a submitter is great about sharing their thought process, but no one responds... or a commenter makes an observation/suggestion but the submitter doesn't reciprocate. It's a spark that doesn't quite catch fire. Personally, it makes me a little sad — I feel the loss of what could be: an enriching conversation.

What if the conversations about a specific submission of a given exercise could continue to exist on its own right but also folded into a larger forum? The basis for that larger forum could well be the similarities of submissions (presumably, if two submissions are "similar" the conversations from each are somewhat fungible). In this way, when someone comments on a give submission, that comment is discoverable by anyone looking at a similar submission (including their own).

I see at least two parts:

... each of which can evolve independently.

In whole, this analysis and aggregation feature set would aim to gather the glowing but dispersed sparks of interest.

Imagine the experience of submitting a solution and within a minute coming back to it and seeing a number of suggested conversations almost as if they were commenting on your submission?

jtigger commented 8 years ago

The Great Beer Run

I'm running an experiment on the Java track. Today, there are 13 implementations of Beer Song. I went through each one and attempted to categorize them (in no particular order):

  1. Builder pattern:
  2. Sequence of if statements / switch:
  3. Map from verse number to Verse object:
  4. Verse templates extracted, selected by version number:
  5. Mapping function from verse number to String.
  6. Method per verse variation.
  7. Decomposed along the two lines of a given verse.

It definitely has that "first-timer" feel but something that probably gets better over time...

With these groupings, I thought of a couple of experiments.

Experiment 1: Birds of a Feather

Then, I thought I'd see what happens when I put two "similar" solutions, side-by-side and look at the little differences (I blame @skmetz and @kytrinyx for infecting me with applying flocking rules everywhere — go get their book!).

I went to both lafent's Beer Song and vatbub's Beer Song and posted the differences I saw... seeking a conversation about trade-offs.

Experiment 2: ... and now for something completely different ...

One of the submissions was very different from the others. leetwinski's Beer Song uses a builder pattern to support a rather fluent description of the verse structure.

I went to the other Beer Song solutions that I hadn't posted any comments on and encouraged them to take a look at leetwinski's solution and comment.

Granted, some of these submissions are from July. No telling what will transpire.

kytrinyx commented 8 years ago

This is so interesting, @jtigger—I'd love to hear if something comes of this experiment.

Daniel-at-github commented 7 years ago

Could the grouping be semi manual?

When I see a good solution that I want to use in my next iteration of the exercise a fork or merge (or I will reuse this) button will be handy to:

kytrinyx commented 7 years ago

@Daniel-at-github I hadn't thought of that.

The thing that makes me hesitate is that most of the similar solutions come from people who have never seen each other's code. For example, an exercise might have several thousand solutions, and there might be about ten or twelve different "typical approaches". Each one would usually have many hundreds of submissions that are somewhat similar.

jtigger commented 7 years ago

Well, "The Great Beer Run" poured out flat. :) I got a little bit of a response, but not much.

I hypothesize that there was little movement because the submissions were a) at the end of the track; and b) most submissions were from 3 month prior. The combination of those two likely meant that folks might have moved on.

The Java track is teeming with activity: half a dozen new exercises are coming online and the track is about to reorder their exercises. I'm going to retry this experiment again with one of the new exercises.

kytrinyx commented 7 years ago

The Java track is teeming with activity

That's very exciting!

jtigger commented 7 years ago

Squaring the Differences

(aka "The Great Beer Run, take 2")

In this experiment, I targeted the very small and rather newish-to-Java-track difference-of-squares exercise.

  1. I collected the solutions into three categories: functional, imperative, and computation.
  2. I identified an example of each. I favored the most approachable of each.
  3. I removed my personal submission for the exercise and replaced it with a three-file submission: the three example solutions (citing authorship and linking to them)
  4. I then went to every submission on the track for this exercise (~20) and pointed participants to comment on the three options and explore the trade-offs of each.

The result was a thoughtful discussion:

http://exercism.io/submissions/524dba3333a14097983bea7489916b78

I'm going to continue this experiment by inviting each new submitter to this discussion.

kytrinyx commented 7 years ago

@jtigger this is exciting.

Also, we could make a rule for Rikki to follow to submit a comment that points people to that discussion when they submit iteration 1 of difference-of-squares.

petertseng commented 7 years ago
  1. I collected the solutions into three categories: functional, imperative, and computation.
  2. I identified an example of each. I favored the most approachable of each.

As a problem-solver curious about the various different ways to approach a problem, my Exercism experience would be improved significantly if something like that were done for more exercises.

Instead of repurposing a single individual's submission as a space for this discussion, I envision a space where everyone who has solved a problem can come together to discuss it... which is like https://github.com/exercism/exercism.io/issues/2615 which already has a link to this issue.

Additionally, I would wonder whether such discussions must necessarily be limited to the language * exercise pair, or if there should be one such discussion for the exercise, across all languages. One might think that an abstract approach to solving a problem is language-independent, so one could benefit from hearing ideas from other languages as well.

Interesting question about how we categorise, if not by machine. If the job is in the hands of a few fixed individuals, this becomes hard to scale. It could be crowdsourced: I can self-register my solution under an already-existing category... or create a new category if I truly think my solution is unique. Maybe I can register others' solutions as well, etc. This is of course expanding on previous comments such as https://github.com/exercism/discussions/issues/65#issuecomment-274340803 that suggest a human element in the categorisation.

jtigger commented 7 years ago

I would love to find ways to make such comparisons framed with trade-off thinking. Not at all interestes in the "best" solution, rather the trade-offs of a given set of design choices.

What if Exercism was also a living, evolving "encyclopedia" of the craft of code? In the same way that scientists come together to displace yesterday's theories with different ways of thinking?

jtigger commented 7 years ago

Squaring the Differences (an update)

I've executed two runs since I initiated this experiment.

Run 1: http://exercism.io/submissions/524dba3333a14097983bea7489916b78 Run 2: http://exercism.io/submissions/85467bc28d41452f92f21640d0946e90

Reading the discussion for yourself is the most powerful way to convey the richness of the conversation. Here are some teasers:

@abo64: One advantage (and that is one reason why FP has become more popular these days) is the much easier option of parallel/asynchronous/distributed computation. Think Google's MapReduce - which about started the wave.

@gspitz01: ... Either way, it's clear that the imperative code is not something that programmers should be writing anymore. The important differences between even those two simple functions gets lost in the boilerplate code. Frankly I think the way you've displayed them here is how I at least plan on coding in the future, if I haven't been doing it already. First think about a computational solution. If that's too complex, functional should serve for most purposes. And only if it's absolutely necessary will I even think about doing something imperatively...

Getting Rikki's help

@kytrinyx noted:

Also, we could make a rule for Rikki to follow to submit a comment that points people to that discussion when they submit iteration 1 of difference-of-squares.

At this point, we're getting so few submissions that doing this manually is no biggie (I picked an exercise rather far down the track for this and so that I select on folks who are so invested they made it this far).

That said, this would be a cool feature to just be able to turn on for a given exercise.

Looking forward

What other kinds of conversation can we spur?

kytrinyx commented 7 years ago

This is an area that we are working on in the new prototype: https://github.com/exercism/v2-feedback/blob/master/README.md

One of Thalamus' core competencies is in AI and machine learning, so we anticipate doing some very exciting work in the future to explore the solution space and make that useful to both mentors and learners.

It's going to take us a while to get to this, but it is one of the things we're very excited about tackling once we have the bandwidth.

I'm going to go ahead and close this, as we would like any further discussion about the topic to be based on the new site.