Representers

[iHiD]

This issue outlines a new angle to automated feedback. It has come out of a few weeks of thought and prototyping by a smaller working group, who are happy that it is a sensible approach for us to move forward with. However, we might have missed problems, or missed ideas that could make it 10x better. All feedback is very welcome.

---

Problems with analyzers

Increasingly, it is clear that providing automated feedback is critical to how Exercism is devleoping. We have been working with analyzers to achieve this but there it has become clear that there are a few major issues with our current implementation:

1. All of the power lies with the person creating the analyzer. They determine what things to look for, what comments are given, when things are approved etc. It is both risky and unsubstainable giving this much power to one person. It is quite possible that the person with the technical skill to make an analyzer is not the right person to be making mentoring decisions.
2. It's unscalable. Every new exercise needs a solid chunk of work by the analyzer-maintainer. Every new solution that's not matched involves proper brainy effort to add the right analysis.
3. It's a hard problem in the first place, which limits the amount of contributors willing to flesh out the first cut of it, Which means we have lots of people who want to get started, but never get a version deployed.

In parallel to these issues, the shape of core tracks is looking increasingly likely to be made up of numerous small exercises, often with stubs, generating a smaller amount of uniquely submitted submissions.

An additional/alternative solution

We have been experimenting with an alternative approach to Analyzers, which I've called Representers. The basic idea is that rather than trying to understand a solution and provide feedback, we simply look to see if a previous identical solution has been given canonical feedback (see worked example below), and if so, provide the same feedback. The Representer is a language-specific service that takes the solution code, and is responsible for providing a representation of that code. This representation will be an AST with some simplifications/normalisations, depending on the language. An example is walked through below.

We do not percieve representers replacing analyzers, but working alongside them. Analyzers might still do checks for linting, variable names, whitespace issues, etc, as well as more advanced analysis on exercises with wider problem-spaces, but Representers give us a brute-force method to guarantee that mentors never have to give feedback on the same exercise twice.

How it could work

We have some rough prototypes of the flow below. They seem to work well.

Canonical feedback will be provided by a team of vetted mentors, who have an proven understanding of both mentoring and the goals of the exercises. In this document we call these people "Authors" but the term may change. They will have access to a specific UI for this purpose.

What the Author sees

In a UI, the Author sees the following code:

class TwoFer
  def two_fer
    foo = "foo"
    bar = "foo"
    return foo
  end

  def foo
    TwoFer
  end
end

They give the feedback below. Any class/variable names etc must be written to be interpolated using %{...} in the same way the analyzers currently work with website-copy.

(The comments might be stored directly in the database as text, or might be stored as references in website-copy with some sort of cleverness in mentoring UI. I've not working out the pros/cons of that yet and I don't think it matters for this)

{
  "status": "disapprove",
  "comments": "Return is implicit so you can just finish with `%{foo}`"
}

What happens behind the scenes

Once the Author hits the "Save" button, we kick off a process to a generic Exercism service (let's call it "Processor").

The Processor takes the code and shells out to a language-specific application (a "Representer") that creates a Representation of the code by taking an AST and normalising it. There might be various normalisation abstractions, but as a minimum class/method/variable names must be normalised. If a variable and function name share the same name, they must be normalised to have the same key. This ensures that the comments will be correct, which we'll get to shortly...

The input/output interface will be very similar to that of the analyzers - take code in, write representation.txt out, and also writing out a dictionary of normalised terms.

In this example, the Representer handles Ruby code:

It would create a standard (Ruby) AST, which would look like this for the code at the beginning:

s(:class,
  s(:const, nil, :TwoFer), nil,
  s(:begin,
    s(:def, :two_fer,
      s(:args),
      s(:begin,
        s(:lvasgn, :foo,
          s(:str, "foo")),
        s(:lvasgn, :bar,
          s(:str, "foo")),
        s(:return,
          s(:lvar, :foo)))),
    s(:def, :foo,
      s(:args),
      s(:const, nil, :TwoFer))))

It would then normalise it, for example, the class/method/variable names are normalised below:

s(:class,
  s(:const, nil, :PLACEHOLDER_1), nil,
  s(:begin,
    s(:def, :PLACEHOLDER_2,
      s(:args),
      s(:begin,
        s(:lvasgn, :PLACEHOLDER_3,
          s(:str, "foo")),
        s(:lvasgn, :PLACEHOLDER_4,
          s(:str, "foo")),
        s(:return,
          s(:lvar, :PLACEHOLDER_3)))),
    s(:def, :PLACEHOLDER_3,
      s(:args),
      s(:const, nil, :PLACEHOLDER_1))))

This would be output as a representation.txt along with the following mapping.json

{
  "TwoFer": "PLACEHOLDER_1",
  "two_fer": "PLACEHOLDER_2",
  "foo": "PLACEHOLDER_3",
  "bar" "PLACEHOLDER_4":
}

The Processor uses this mapping to normalize any interpolated values in the code comments. For example:

{
  status: "disapprove",
  comments: "Return is implicit so you can just finish with `%{PLACEHOLDER_3}`"
}

The Representation and feedback are then all persisted in the database.

A new submission

A new solution is submitted.

class TwoFer
  def two_fer
    thing = "foo"
    other = "foo"
    return thing
  end

  def thing
    TwoFer
  end
end

We want to check to see if it's been previously given canonical feedback. We firstly shell out to the Representer, which provides us with the a normalised representation as per above:

s(:class,
  s(:const, nil, :PLACEHOLDER_1), nil,
  s(:begin,
    s(:def, :PLACEHOLDER_2,
      s(:args),
      s(:begin,
        s(:lvasgn, :PLACEHOLDER_3,
          s(:str, "foo")),
        s(:lvasgn, :PLACEHOLDER_4,
          s(:str, "foo")),
        s(:return,
          s(:lvar, :PLACEHOLDER_3)))),
    s(:def, :PLACEHOLDER_3,
      s(:args),
      s(:const, nil, :PLACEHOLDER_1))))

This representation is now looked up and a match is found then:

{
  "status": "disapprove",
  "comments": "Return is implicit so you can just finish with `%{PLACEHOLDER_3}`"
}

This is interpolated as:

{
  "status": "disapprove",
  "comments": "Return is implicit so you can just finish with `thing`"
}

And everything makes sense.

[yawpitch]

I like it. Normalization will be the trick, but if we can get that right in each language this seems like a solid approach. Also opens up the comments to i18n.

[iHiD]

Also opens up the comments to i18n.

Hadn't thought about this. Nice!

[ErikSchierboom]

Normalization will be the trick

Yes, this is the tricky, but also the fun part! If someone is interested, the C# representer already has several normalization tricks up its sleeve: https://github.com/exercism/csharp-representer/tree/master/src/Exercism.Representers.CSharp/Simplification

[sshine]

I also like it. I have some questions:

We want to check to see if it's been previously given canonical feedback.

1. How do you differentiate between one-time feedback and canonical feedback?
2. How do you know that the canonical feedback applies to a solution with slight variations?
3. Have you considered statistical models for recognizing when canonical feedback applies that are independent of syntax tree analysis?

[yawpitch]

Come to think of it, wouldn't the mapping.json be more useful the other way round?

{
  "PLACEHOLDER_1": "TwoFer",
  "PLACEHOLDER_2": "two_fer",
  "PLACEHOLDER_3": "foo",
  "PLACEHOLDER_4": "bar",
}

[iHiD]

How do you differentiate between one-time feedback and canonical feedback?

Different UIs entirely. Only feedback given through a specific UI by "Authors" will go into the analyzer. There are various options we have here, such as presenting a mentor's solution/feedback pair to an Author in the special UI, and letting them approve/alter it, or presenting a solution to an Author and saying "We think this looks like XYZ" and showing that feedback for the to play with (more complex technically, but also likely to provide less work for the Authors and more consistency in the platform).

How do you know that the canonical feedback applies to a solution with slight variations?

Unless the Representations match, the feedback won't be given. So it is important that Representations only diverge from ASTs in ways that will not affect the feedback. We will also restrict feedback on things like variable names (e.g. don't say "we use full words rather than single letter variables in Ruby"), as these things won't be the purposes of the exercises anyway.

To put it another way, we need to guarantee that the "slight variations" offered by AST->Representation transformations are so slight as to be irrelevant to the feedback given.

Erik has also suggested to me that we show an Author both the students solution and a re-coded versions of the Representation (so a pipeline AST -> Representation -> Code), which should allow them to understand the realm of changes that could occur through the Representor. I don't know if this is possible across languages though, and I suspect it's not.

Have you considered statistical models for recognizing when canonical feedback applies that are independent of syntax tree analysis?

Yes. I would love to use some machine learning to be able to provide feedback automatically, and I don't actually think it would be that difficult. But the risk of incorrect feedback would be a lot higher than this approach, so I think the most sensible thing would be to potentially propose the machine-learning-generated feedback to an Author to "sign off". Very few solutions submitted are "new" so once a wad of mentoring/feedback is done for an exercise, this should cover most cases. And those that are new generally require a mentor anyway as they're specifically unusual in some way.

Come to think of it, wouldn't the mapping.json be more useful the other way round?

It is essential that "TwoFer" should only appear once in the mapping, and that all instances of "TwoFer" are mapped to the same placeholder*. I therefore prefer to keep it as a key to ensure this, although it does logically feel nicer to have the PLACEHOLDER_X as the key

* This might not be immediately obviously/clear. It took me prototyping it the other way round to realise that there must only be one "TwoFer", and then a good 30min chat to explain it to someone the other day. I haven't found a good way to explain the reasons more concisely yet, so you'll just have to trust me for now :)

[yawpitch]

It is essential that "TwoFer" should only appear once in the mapping, and that all instances of "TwoFer" are mapped to the same placeholder*. I therefore prefer to keep it as a key to ensure this, although it does logically feel nicer to have the PLACEHOLDER_X as the key

Gotcha, I just assumed that the one-to-one constraint would be enforced elsewhere than the JSON representation.

[iHiD]

would be enforced elsewhere than the JSON representation.

Yeah - that is another option, which I hadn't really considered.

[ErikSchierboom]

Erik has also suggested to me that we show an Author both the students solution and a re-coded versions of the Representation (so a pipeline AST -> Representation -> Code), which should allow them to understand the realm of changes that could occur through the Representor. I don't know if this is possible across languages though, and I suspect it's not.

So my reasoning here is that most AST parsers hopefully also have an un-parse option, where they convert an AST back to actual code. The C# parser has this exact feature, but I was indeed wondering if other languages have this option. @sshine @SleeplessByte (and anyone else) do you think this approach (code -> AST -> normalization -> code) is feasible?

[yawpitch]

Erik has also suggested to me that we show an Author both the students solution and a re-coded versions of the Representation (so a pipeline AST -> Representation -> Code), which should allow them to understand the realm of changes that could occur through the Representor. I don't know if this is possible across languages though, and I suspect it's not.

So my reasoning here is that most AST parsers hopefully also have an un-parse option, where they convert an AST back to actual code. The C# parser has this exact feature, but I was indeed wondering if other languages have this option. @sshine @SleeplessByte (and anyone else) do you think this approach (code -> AST -> normalization -> code) is feasible?

With builtins Python can only parse human readable source code to an AST and then compile a (potentially modified) AST to an executable bytecode object, there’s no builtin path back to human readable (but transformed) source. There is the astor project, which can generate a probably workable human readable form and can also dump a more readable AST, but we’d have to do some effort to verify how well the round trip works in practice.

[ErikSchierboom]

Hmmm, if this is not something that's built-in for a popular language like Python, I then expect other languages to have the same issue. Too bad, but thanks for checking!

[SleeplessByte]

Not built in. Eslint has one walker function that "writes" code.

[yawpitch]

Honestly roundtrip capabilities may well be more common in other languages, especially ones you might write a compiler in... since Python is interpreted you’ve usually got the source code right there in front of you, which somewhat renders code generation from an AST a concern of those trying to decompile “pyc-only” distributions, as you tend to find in commercial applications and hacky attempts at “closed-source” packaging.

That vaguely piratical use case kind of pushes against it being a need of the standard library distribution, but it does exist in the wider ecosystem.

It might be worth doing a formal survey of the tracks, rather than make assumptions based on Python. On Oct 18, 2019, 09:39 +0100, Erik Schierboom notifications@github.com, wrote:

Hmmm, if this is not something that's built-in for a popular language like Python, I then expect other languages to have the same issue. Too bad, but thanks for checking! — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[iHiD]

Ruby doesn't either, but there is a library that apparently does it reasonably well (untested by me though). The reason I think it won't be common is that there is no "normal" usecase for this in most languages. All languages need to parse the human-written to "something", but there will be few exceptions (.net being one) where it's generally useful to go back from AST to code. It's also worth noting that Ruby has only just added any AST extraction into the core language at all - and it's currently marked as experimental and unstable. I'm using 3rd party libraries to do it there.

[sshine]

@iHiD wrote:

We will also restrict feedback on things like variable names (e.g. don't say "we use full words rather than single letter variables in Ruby"), as these things won't be the purposes of the exercises anyway.

Hmm. As a mentor I've consistently given general programming advice in all exercises, e.g.:

• Function and variable naming: E.g. you don't prefix functions with get... in purely declarative languages; since values don't mutate, functions are inherently getters (unless I/O or stateful simulation is involved, but then you're naming monadic actions rather than functions). But also less language-specific advice like y vs. year, as you hint at.
• Version-control friendly indentation: Don't indent your lines in such a way that forces meaningless re-indentation of nearby located lines.
• Expressing problem constraints in a forward-compatible way. (An example of where this failed, and I was the committer of the least forward-compatible solution is exercism/haskell#861, where one example solution needed a fix for a new test case and another one didn't.)

When it is possible to encode some of this advice into an analyzer, why not?

And when not, and a mentor is willing to give this feedback manually, why not?

---

@iHiD wrote:

[On "Representers"] There might be various normalisation abstractions, but as a minimum class/method/variable names must be normalised. If a variable and function name share the same name, they must be normalised to have the same key. [...]

Forgive my ignorance: Why is it that we're aiming for an intermediate representation of a solution to a given problem where foo is replaced with PLACEHOLDER_3? If we depend on a program with a language-specific syntax parser (what we call "an analyzer" in Exercism, right?) for generating this, why not let it spit out { "status": "disapprove", "comments": "Return is implicit so you can just finish with `thing`" } directly?

In short, what does this intermediate representation do?

---

@iHiD wrote:

[...] This ensures that the comments will be correct, which we'll get to shortly...

@ErikSchierboom wrote:

most AST parsers hopefully also have an un-parse option, where they convert an AST back to actual code

Depending on what you mean, un-parse could either mean "print the AST in a way that preserves the lexical structure of what was parsed", or "pretty-print the AST in any way deemed by some pretty-printer".

In combination with what @iHiD says, I understand that you mean the former.

My guess is that it's not super common?

Nowhere near as common as having a pluggable parser.

Might be worth a survey, as @yawpitch suggested.

I think this type of utility occurs around the moment a language obtains an IDE-like linter that will perform automated substitutions. I think a constraint like this might exclude a fair amount of "old but good" languages that don't see this kind of active ecosystem development.

---

Come to think of it, wouldn't the mapping.json be more useful the other way round?

It is essential that "TwoFer" should only appear once in the mapping

it does logically feel nicer to have the PLACEHOLDER_X as the key

Since JSON is used here to express a bijective mapping, and since the JSON spec allows duplicate keys in an object, deferring this constraint until a post-parse validation step should provide the same safety no matter what way you flip the mapping.

[yawpitch]

In short, what does this intermediate representation do?

@sshine the idea is that with the AST sufficiently normalized you can effectively take a hash of the IR and any exercise with the same hash(normalize(to_ast(source)) would have the same feedback. There being a relatively limited number of ways most people would express the solutions to most of the core and easier side exercises (at least) the idea is to get people with similar approaches (ignoring for whitespace, comments, docstrings, variable / class / atom naming, and potentially helper function calls, etc) into similar response streams.

The advantage is that it’s more data driven ... each hash represents something we’ve received, and in theory the advice fo that hash will lead to mutation towards another more optimal (and more common) hash we have received before, ultimately leading to one of the most common hashes, the set of optimal (but normalized) solutions.

An analysis step can then take over and make style and naming and other such recommendations to their pre-normalization, but optimal solution.

Just writing an analyzer that’s aimed at a specific “preferred” optimal is a recipe for founder effect biases ... it’s the same problem that the mentor notes have: whoever writes them first embeds their own habits (including the bad ones) and omits what’s behind their blinkers.

Plus there’s an added benefit ... build up a large enough repository of normalized and tagged intermediate representations and we’ve got a good training set for ML. Though that assumes the normalize algorithm remains stable or at least versioned.

[iHiD]

If we depend on a program with a language-specific syntax parser (what we call "an analyzer" in Exercism, right?) for generating this, why not let it spit out { "status": "disapprove", "comments": "Return is implicit so you can just finish with thing" } directly?

I think yawpitch has covered this very well, but for clarity, an analyzer has to "understand" the solution to see what's going on, whereas the representater is relatively "dumb" and is just producing some normalised AST output. To put this in perspective, the total program size of a representer is like 10 LOC. My Ruby analyzer for Two-Fer is probably 200 LOC. And that would grow per exercise.

We could do this with no representer at all, and just match on whether we've seen the code before. The principle would be the same. However, the representer allows us to give the same advice regardless of whitespace, variable names, comments, etc.

When it is possible to encode some of this advice into an analyzer, why not? And when not, and a mentor is willing to give this feedback manually, why not?

A huge problem we have is variance of mentoring. At the moment some mentors try and guide a student to a "perfect" solution for an exercise, helping them with the algorithm, the idioms, comment style, variable naming, indentation etc. Other mentors simply try and get the student towards a solution that is idiomatic enough to pass.

With the advent of the new "concept exercises" we will be moving towards having more specific exercises trying to teach specific things. For example, we might not bother telling someone about variable names in the "how to use functions" exercise. Instead we might have a "refactoring" exercise where someone makes the variable names and whitespace more normalised (I'm making this up, but hopefully the example gives an idea). Because each exercise will have a specific goal, we don't want mentors to give extraneous advice (even if it's helpful at a general level) and instead want to ensure that there are exercises that cover the things, or that those things are covered in the comments on Practice exercises (which will not have "approval" and where a community-member can nitpick a lot more without it blocking a student's progress).

Does that make any sense?

Depending on what you mean, un-parse could either mean "print the AST in a way that preserves the lexical structure of what was parsed", or "pretty-print the AST in any way deemed by some pretty-printer".

So C# has a built in AST-generator-library, but that library will also go the other way, and turn AST into C# code. From what I understand doing myAst.ToString() outputs C# code, rather than a lisp-like structure (Ruby) or JSON (JS). So Erik's "Representation" for C# is actually C# code (that's gone to the AST level, been tweaked, and then output as C#, which then has whitespace removed for compactness).

I'm pretty certain this is a non-starter. I wouldn't trust the Ruby one to output it, so I don't think this is a path that is worth us going down.

Since JSON is used here to express a bijective mapping, and since the JSON spec allows duplicate keys in an object, deferring this constraint until a post-parse validation step should provide the same safety no matter what way you flip the mapping.

Ok, I don't mind us going down this route. It's just an extra bit of code that needs to check things are valid at my end, and then some error handling around that. But that's fine.

---

I'm only half-way into my first coffee of the day, so I'm not sure what I've written is particularly clear :) Please (as always) feel free to dig deeper or reiterate a question if I haven't made sense.

[sshine]

Thanks for detailed explanations both of you.

the total program size of a representer is like 10 LOC. My Ruby analyzer for Two-Fer is probably 200 LOC. And that would grow per exercise.

So Erik's "Representation" for C# is actually C# code

OK, so is the general idea that you take any input program, "normalize" it, meaning you convert it into an abstract form (that holds the property of being hashable) and isn't necessarily a raw syntax tree, and can be concrete syntax, being called a Representer, and embed/annotate such things as placeholders when a feedback concerns an identifier, thus isolating the approval/disapproval and actual comment?

I suppose, also, that another property of a Representer is that they are composable: Given two hints for an exercise, there exists a Representer that embeds information about both hints. And that for an exercise with n hints, there are 2ⁿ compositions of Representers (for any one hint, it is either enabled or disabled).

My only concern now is how to represent a Representer for hints that do not address a particular identifier or sub-expression, since this is the only example given so far. I'll try and produce some examples that I cannot seem to translate into Representer lingo/representation later.

Here is another example that for two hints in Space Age on Haskell: a common pattern is

getOrbitalPeriod :: Planet -> Seconds
getOrbitalPeriod Mercury = earthYear * 0.2408467
getOrbitalPeriod Venus = earthYear * 0.61519726
...

where orbitalPeriod is a helper function, and earthYear is some constant defined elsewhere. This may be represented as

_PLACEHOLDER_1 :: Planet -> Seconds
_PLACEHOLDER_1 Mercury = _PLACEHOLDER_2 0.2408467
_PLACEHOLDER_1 Venus = _PLACEHOLDER_2 0.61519726
...

and the accompanying mapping.json

{
  "_PLACEHOLDER_1": "getOrbitalPeriod",
  "_PLACEHOLDER_2": "earthYear *",
}

and the composition of the two hints

{
  "status": "approve",
  "comment": "The function `%{_PLACEHOLDER_1}` is prefixed with 'get', but all pure functions are getters, so this prefix is unnecessary."
}

{
  "status": "disapprove",
  "comment": "Your code contains duplicate code: `%{_PLACEHOLDER_2}`. You can avoid this by moving the duplicate code outside of a case-of or into the calling function."
}

[yawpitch]

@sshine if I understand the intent correctly the first of those two comments would make more sense in an analyser pass, not a representer pass.

Think of the representer as a function whose job it is to strip user-controllable noise from a piece of source code to produce a potentially isomorphic representation of the code.

Ignoring all talk of CSTs and ASTs and at the least useful level it could take this:

getOrbitalPeriod :: Planet -> Seconds
getOrbitalPeriod Mercury = earthYear * 0.2408467
getOrbitalPeriod Venus = earthYear * 0.61519726

Or this

orbitalPeriod :: Body -> Period
orbitalPeriod Mercury = year * 0.2408467
orbitalPeriod Venus = year * 0.61519726

And produce:

PLACEHOLDER_1 :: PLACEHOLDER_2 -> PLACEHOLDER_3
PLACEHOLDER_1 PLACEHOLDER_5 = PLACEHOLDER_6 * PLACEHOLDER_7
PLACEHOLDER_1 PLACEHOLDER_8 = PLACEHOLDER_6 * PLACEHOLDER_9

Or, considering you can overload operators in Haskell:

PLACEHOLDER_1 PLACEHOLDER_2 PLACEHOLDER_3 PLACEHOLDER_4 PLACEHOLDER_5
PLACEHOLDER_1 PLACEHOLDER_6 PLACEHOLDER_7 PLACEHOLDER_8 PLACEHOLDER_9 PLACEHOLDER_10
PLACEHOLDER_1 PLACEHOLDER_11 PLACEHOLDER_7 PLACEHOLDER_8 PLACEHOLDER_9 PLACEHOLDER_12

And then you'd have a key that would represent any source that was isomorphic in structure ... advice given for either solution is more likely advice good for both solutions than advice given to a solution that isn't isomorphic with the above.

Of course there's a LOT of ways of writing code that's isomorphic in the essentials, which is which is why incorporating the idea of CSTs/ASTs gets us to much more power in doing this, because you could then do things like:

1. Ignore unimportant details (ie whitespace, comments)
2. Ignore invariants (ie non-overloaded operators and prelude functions, language keywords, etc)
3. Ignore literals, since these will be necessarily the same across solutions ... the only problem here is that they may need to be fed in from the test data somehow.
4. Normalize user-controlled names.
5. Potentially de-sugar language constructs (the "idiomatic" sugar form to use may best be left up to an analyzer).

The idea is to get us into the ballpark, not all the way home.

[tehsphinx]

@iHiD: Not sure if this was mentioned already or if it is done already:

I'd save the original code with the representer in the database. That way when the processor needs updating, all the existing representers can be updated as well and are not lost.

[SleeplessByte]

Ok, I don't mind us going down this route. It's just an extra bit of code that needs to check things are valid at my end, and then some error handling around that. But that's fine.

I agree with placeholder names as keys.

[ErikSchierboom]

@sshine if I understand the intent correctly the first of those two comments would make more sense in an analyser pass, not a representer pass.

Yes, I believe that in this case (comments based on identifiers), it makes sense to do this check in the analyzer, as the representer will indeed have "lost" this information.

Think of the representer as a function whose job it is to strip user-controllable noise from a piece of source code to produce a potentially isomorphic representation of the code.

Yep, this is also how I view the representer. Great write-up. Does that make sense @sshine?

[mpizenberg]

The Representer is a language-specific service that takes the solution code, and is responsible for providing a representation of that code. This representation will be an AST with some simplifications/normalisations, depending on the language.

I don't know about other tracks but I'd guess that tracks for small languages like elm do not have people with the skills to do that. I certainly don't for elm. What should we do for tracks that cannot pull this off? It seems to be the heart of v3 proposal. So finding what consequences unability to do that for a track may have is important I think.

[yawpitch]

There appears to be — via Google — an AST module for Elm, which would be the likely first stop. But also, since IIRC Elm compiles to Javascript the compiled representation could likely serve to feed a JS representer to useful effect.

In general though languages tend to come with the tools necessary to parse their own syntax, so all that may be lacking is sufficient familiarity with those tools; that can be developed, and as more representers roll out it should become easier to see how to utilize those tools to develop a well-tuned representer for more obscure languages.

For languages that completely lack those tools it could be more difficult, but those languages likely also don’t have the kind of throughput that makes an automated representer really useful ... such languages may need to move to V3 only incrementally. On Oct 31, 2019, 16:28 +0000, Matthieu Pizenberg notifications@github.com, wrote:

The Representer is a language-specific service that takes the solution code, and is responsible for providing a representation of that code. This representation will be an AST with some simplifications/normalisations, depending on the language. I don't know about other tracks but I'd guess that tracks for small languages like elm do not have people with the skills to do that. I certainly don't for elm. What should we do for tracks that cannot pull this off? It seems to be the heart of v3 proposal. So finding what consequences unability to do that for a track may have is important I think. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[iHiD]

There's a demo of the Elm AST module here which looks good. I'd be really surprised if there was a language where there was no AST or equivelent support, because as @yawpitch says, languages tend to be good at parsing their syntax. If those cases do exist, at worst, I think we could just take the compiled code and match off that. It wouldn't allow use of the placeholders, etc, but would still do the job.

As everyone seems happy in principle with this, I'm going to close this issue and start to move forward with stuff. Thanks for everyone for the comments!

[workingjubilee]

Please be wary of relying on compilation results as a fallback, as compilation actually across multiple languages runs a dangerous chance of being nondeterministic(!):

https://blog.regehr.org/archives/294 https://gitlab.haskell.org/ghc/ghc/wikis/deterministic-builds https://github.com/mgrang/non-determinism

[iHiD]

@workingjubilee Thanks. Having thought about it more, we'd just go back to code, not compiled-code, as the compiler would optimise away so much detail anyway, that I don't think it would make sense.