Use an expression tree as input

[erikpukinskis]

Expected behavior:

renderCode takes an expression tree as its input

Current behavior:

renderCode takes an array of lines of text as its input

Tasks:

• [ ] Load existing "webPage" demo into an expression tree
• [ ] Get first line to render
• [ ] Get the rest of webPage demo to render
• [ ] Get the "activity" demo to render

[erikpukinskis]

Getting this:

render-code$ node demo.js 
/x/render-code/editor.js:389
        throw new Error("trying to add an expression for a line without an id")
        ^

Error: trying to add an expression for a line without an id
    at Editor.noticeExpressionAt (/x/render-code/editor.js:389:15)
    at Editor.text (/x/render-code/editor.js:453:12)
    at Object.<anonymous> (/x/render-code/demo.js:36:21)

[erikpukinskis]

(I keep wondering if Editor should be its own module, like maybe "editor-model" or something, independent of write-code. Maybe even subsuming javascript-to-ezjs... like, "this takes various inputs and maintains a model of an expression tree")

[erikpukinskis]

So, the error is in noticeExpressionAt. We do call ensureSomethingAt for line 0, if there's no rootFunctionId on the editor.

We don't ensureSomethingAt for the lineNumber being noticed... I guess because the edit model always ensures there's an open line before we add anything.

We do, in syncExpressionToLine do

var nextLineId = this.ensureSomethingAt(lineNumber + 1, lineId)

[erikpukinskis]

I guess maybe when we import we just need to add that first line... How does that get added when we were just in the original write-code editor experience?

[erikpukinskis]

Ah, in the original write-code editor experience, when we were typing in, e.g., an initial string, it was at line 0. I thought the lineIds were starting at 1, and 0 was a virtual root expression.

[erikpukinskis]

OK, if I just index from 0 I can add the library.using call, which adds an empty string for whatever reason. Then I can add the "web-element" string but not "web-site", ostensibly because there's no last line.

I think maybe (A) we need to be ensuring there is always an empty last line in every program?

Or, alternatively (B) we need a new insert function, which can insert text after a line, instead of just typing text into a line.

[erikpukinskis]

~What confuses me about (A) is nesting. If I type library.using( then seemingly that would end up as:~

library.using(
  *)

~but if I then type "web-element" would we get (1)~

library.using(
  "web-element",
  *)

~or (2)~

library.using(
  "web-element")
*

~?~

~I'm having this doubt when I'm formatting the demo code. I haven't designed any sort of indent/outdent UI at all, so I have no intuitions here.~

[erikpukinskis]

Last comment was a bit off topic. I'm choosing

(A) we need to be ensuring there is always an empty last line in every program?

because it's simpler and closer to what happens in the editor.

So that means, the task here is to get the write-code tests running again, and add a test:

"When I type a function call and then a string in an array arg, there should be an empty expression on the next line in the array so I can add another string there"

[erikpukinskis]

Typing that up as an Issue, couldn't quite come up with the acceptance criteria. Made me think maybe a third option (C) would be to "press enter" after each line is added if there is no empty line already (as there would be after pressing ( and getting a new function call arg line).

[erikpukinskis]

OK, Editor doesn't seem to have any kind of "press enter" functionality (makes sense, it doesn't have a cursor at all. That's in write-code I guess.

Closest thing is this:

    Editor.prototype.ensureSomethingAt = function(lineNumber, parentId) {

which... means we need to keep track of the last parent?

[erikpukinskis]

I wonder if I'm sidetracked and the issue is just that we haven't handled array literals at all in the demo.

[erikpukinskis]

Coming back to this after quite some time.

It doesn't seem like this work is in progress anymore. renderCode still just takes an array of strings (lines).

Background

I think my original motivation for wanting to take an anExpression.tree as input to renderCode is that Editor in writeCode takes a tree as input. Although Editor can importLines, all that does is essentially "type" the lines into the editor one by one. The tree can be passed to the Editor constructor.

And most importantly, Editor doesn't use lines for persistence, it uses an expression tree. It calls tree.addExpressionAt, tree.ensureSomethingAt, tree.addToParent, tree.insertExpression, and tree.setAttribute. That is probably the clearest interface where the Editor has completed its work.

However, currently there is another interface. Inside editLoop, we call this function syncLine which calls a number of editor functions: editor.role, editor.getIntroSymbol, editor.getSeparator, editor.getOutroSymbols, editor.getFirstHalf, and editor.getSecondHalf. It then uses the tokens library to actually edit the DOM. tokens.setIntroToken, tokens.setSeparator, tokens.setOutroTokens.

The anExpression.tree is in there because we have to have persistence. That's a given.

The motivation behind using it as the interface to renderCode is that there might be a somewhat clean mapping between those tree functions and the tokens functions.

Also, syncLine isn't a complete interface for handling changes. It shares responsibility with the editor itself. editor.addLineAfter adds lines.

[erikpukinskis]

So I think what I need to do is kind of ruminate on these interfaces and think about how I want to separate concerns.

Current separation of concerns:

• The edit loop notices events from the user, and tells the editor editor.pressEnter, editor.text

• The editor is responsible for understanding actions from the user, mapping them into a line-based representation, and notifying the expression tree

• Expression tree is just there for persistence

• The edit loop then handles updating the cursor/selection

• The edit loop also reads info from the editor and calls a bunch of functions in tokens

• Tokens updates the DOM within a specific line

[erikpukinskis]

I think this separation of concerns is mostly ok.

I think render-code does a great job of rendering well-formed EZJS from text, and that's ok. That's actually a simpler task than rendering from an expression tree, because you can't render a line from an expression, you have to look up the stack to see the closers.

Using an expression tree as an intermediate representation that separates the editor from the renderer.... I don't like it. It feels like it introduces an artificial data structure.

That makes me question whether it's the right structure for persistence at all. Maybe I should persist something like the segments instead, or lines!

Pros of persisting trees

1. They can be easily converted to strings/segments for render-code.

   • We can load an Editor from a tree, and then we have tree.toJavaScript().
   • We can also easily export trees to segments if we need to... the Editor represents things internally essentially as segments (intros, outros, firstHalf, secondHalf, etc)

2. Semantically, trees should yield narrower edits... Adding a new expression to an array, for example, will change the segments of the expression above it, e.g.:

var numbers = [
  1]

... edited to become ...

var numbers = [
  1,
  2]

If we (A) store this as segments, then

• the 1 line has to change (its outros changed)
• we add a new line
• the var numbers = [... line doesn't need updating.

If we (B) store it as expressions,

• the 1 expression stays exactly the same
• we add a new expression
• the numbers assignment expression gets a new child expression

What I don't like about (A) storing the segments is that sibling edits affect each other.

3. We can make large copy/paste style changes with one or two edits. If we wanted to deindent a block, for example, if we're persisting lines we'd potentially have to change/copy every line. With expressions we can just re-home the parent expression.

4. A small piece of one program, if it is equivalent to the entirity of another program, can be fully compressed away.

5. Trees can potentially represent non-ezjs code as well... even other languages. Segments can only represent EZJS or similar.

Cons of persisting trees

1. We store segments in the editor, and we render segments in the renderer. From a purely data-driven perspective it would be more direct to just store the segments, and boot the editor more or less verbatim from the log.

[erikpukinskis]

Decision time.

Maybe I'm being speculative here, but #4 above is pretty compelling to me. Expression trees have been designed to efficiently represent an entire tree of programs, which is what the server will be doing. The segments model is really oriented to representing a single user editing a single program, in a single state.

So, for now let's say "segments are user-oriented" and "trees are server-oriented" and put this question to bed. Render-code can continue to take text or segments (which are text-like) and the editor will continue to persist to a tree.