Glue and Paragraphs

[micabytes]

Back again.

Before I get down to the nuts and bolts here, let me just reiterate that I really love what you guys have done here. I've looked at a lot of IF scripting languages over the past few years while working on my own games scripting language v1 and v2, and Ink is - IMO - the best combination of power and simplicity I've encountered so far. Ink is awesome, and you guys are awesome for sharing it with the rest of us.

That said, nothing is perfect. Most of the nits I have a pretty minor things, but I am really not keen on your "single new line -> paragraph break" convention. I commented on this earlier in the chat, but I'd like to bring this up again now that I've worked some on a bare-bones implementation in Java and have a bit more perspective, because I do feel that this is a weakness in Ink, and perhaps something you should (re)consider.

My initial disquiet with this mechanism was due to my experiences collaborating across various technical platforms. Since I work mostly in Linux and the people I collaborate with in Windows or Max, I'd often have someone writing in one editor, and me then having to edit the text in another. That did cause issues from time to time - esp. if you're working with editors that do hard-wrapping (such as many IDE's). But while annoying, this is not a big deal (just requires some editing discipline).

I think this convention has two other effects, though, which IMO are much more serious:

• It forces Ink to include an unnecessary scripting element: Glue
• It makes the Ink implementation a lot more complicated/fragile than it needs to be

My suggestion would be to go with "hard-wrapping" by default, similar to Markdown syntax and Mediawiki syntax. Basically:

• A single newline does not cause a paragraph break (unless the next line is syntax; e.g., stitch, expression, choice, etc)
• A double newline causes a paragraph break.
• If you want to support something like the request in #8, three or more newlines in the syntax could be used to create extra line breaks (this is standard markdown syntax). I kind of agree with Jon that it shouldn't really be needed, though.

  Glue

Feel free to correct me if I'm off, but as far as I can see, the necessity for "Glue" (<>) is entirely the consequence of following the "single new line -> paragraph break" convention.

Without that convention, the core glue example:

=== hurry_home ===
We hurried home <> 
-> to_savile_row 

=== to_savile_row ===
to Savile Row 
-> as_fast_as_we_could

=== as_fast_as_we_could ===
<> as fast as we could.

Would just be:

=== hurry_home ===
We hurried home 
-> to_savile_row 

=== to_savile_row ===
to Savile Row 
-> as_fast_as_we_could

=== as_fast_as_we_could ===
as fast as we could.

Your glue test case:

"Some <> 
content<>
with glue."

Would just be:

"Some 
content
with glue."

A more complex example:

- I looked at Monsieur Fogg 
*   ... and I could contain myself no longer.
    'What is the purpose of our journey, Monsieur?'
    'A wager,' he replied.
    * *     'A wager!'[] I returned.
            He nodded. 
            * * *   'But surely that is foolishness!'
            * * *  'A most serious matter then!'
            - - -   He nodded again.
            * * *   'But can we win?'
                    'That is what we will endeavour to find out,' he answered.
            * * *   'A modest wager, I trust?'
                    'Twenty thousand pounds,' he replied, quite flatly.
            * * *   I asked nothing further of him then[.], and after a final, polite cough, he offered nothing more to me. <>
    * *     'Ah[.'],' I replied, uncertain what I thought.
    - -     After that, <>
*   ... but I said nothing[] and <> 
- we passed the day in silence.
- -> END

Could be written as follows (note the use of diverts to "glue" together across break syntax, as well as line breaks to keep the code nicely formatted):

- I looked at Monsieur Fogg
*   ... and I could contain myself no longer.

    'What is the purpose of our journey, Monsieur?'

    'A wager,' he replied.
    * *     'A wager!'[] I returned.

            He nodded.
            * * *   'But surely that is foolishness!'
            * * *  'A most serious matter then!'
            - - -   He nodded again.
            * * *   'But can we win?'

                    'That is what we will endeavour to find out,' he answered.
            * * *   'A modest wager, I trust?'

                    'Twenty thousand pounds,' he replied, quite flatly.
            * * *  I asked nothing further of him then[.], and after a final, polite
                    cough, he offered nothing more to me. -> after_that
    * *     'Ah[.'],' I replied, uncertain what I thought.
    - -     (after_that) After that, -> we_passed
* ... but I said nothing[] and -> we_passed 
- (we_passed) we passed the day in silence.
- -> END

Note that in the above, I refer to "break syntax" as those syntax elements that break up the script: knots, stitches, choices, gathers, etc.

TL;DR - my impression at this point is that glue is only required because of the choice of "single new line -> paragraph break". At the moment, I can't really see any obvious or common use-cases that makes it worthwhile to litter the code with a lot of <> syntax, as opposed to simply using the double newline convention (+ the occasional divert - something that already works that way in the current Ink syntax).

Code Complexity

The thing that really cements my unease with the glue mechanic, is that it seems to make the Ink runtime/parser a lot more complex than it needs to be.

Based on my experience implementing a markdown style parser vs the work I've done on a Java Ink runtime so far, the former just seems a lot more straightforward.

In the markdown style, you simply parse text/lines until you hit break syntax or a double newline (which is a break syntax itself). Because paragraph breaks are explicit, you always parse/compute until you hit one. It's pretty simple with no exceptions.

I can't say the same for dealing with this in current Ink script. Every time you process a line, you need to check for glue either at the beginning or the end of the line, requiring alternative back-tracking or look-ahead. The Glue syntax is an exception to the "new line -> paragraph break' rule, and because Ink is as powerful as it is, this becomes a really important exception.

The result is a lot of complicated code to deal with this exception.

I was rather puzzled by Joseph's comments earlier about whitespace handling being a problem. No longer, after implementing glue, and having to compensate for all the potential white-space issues that this syntax adds to the code. I haven't checked the C# code in detail, but I found myself having to add a lot of extra whitespace checking because of glue.

I smiled a bit when I read your comment in the runtime that goes "This code is slightly fragile" when discussing lookahead due to glue. Indeed.

Bottom line

In short, I think that there are good reasons to go with the markdown syntax for paragraph breaks both from the POV of the writer and the POV of the developer.

I'm not interested in forking or creating a variant of Ink, and I do realize that you may have other priorities, so if you want to kill this discussion for now, I'll accept that.

I think you should consider the issue, though. I may be overlooking something, but I feel pretty sure that the markdown style paragraph formatting would result in simpler (and by extension - less brittle) library code and it definitely makes the syntax simpler. For a project that will - hopefully - pick up a lot of fans and live for many years to come, such things are better fixed early than late.

[micabytes]

Edit to reference the right issue.

[joethephish]

Thanks for the proposal - that's really interesting. From an implementation perspective, I can totally see the advantages - the glue code adds a lot of complexity to the runtime, and eliminating all that is very tempting.

But, although it's nice to have a simpler implementation, obviously the design of the language has to come first. And, at first glance, despite the simplicity of the test case examples (always lovely to simply remove syntax), I'm not especially enamoured with your real 80 Days sample. The white-spacing doesn't look particularly "beautiful".

I think it's partly because in our games, we always try to have short "paragraphs". Unlike the markdown I'm looking at at this very moment, we try to avoid runs of long paragraphs. In short snappy dialogue, in the average case, you end up with double spacing almost everywhere.

Having said that, I definitely think your idea is worth considering and discussing. It will largely depend on how @joningold feels, since he's the one here who actually has to write millions of lines of our content with this. Who knows, maybe we can find a third solution that combines the best of both...

[micabytes]

That's the danger of making something open-source - suddenly every person and their cats have an opinion on what you're doing.

I'm not the best judge of this, as I've worked with markup pretty much every day (either wikis for tech documentations, bug handlers like this on github, etc) for way too many years, so markup ends up looking "natural" to me at this point. FWIW, though, I think markup syntax of this type has become pretty ubiquitous on the internet anno 2016.

And while I my preference leans toward short snappy sentences as well for game IF, you should probably expect that a lot of works coming out with Ink will be more "traditional" choice-based stuff with longer paragraphs like what you often see in Choicescript (which also follows double the line-break convention... as I said, pretty ubiquitous).

I realize it's a big ask, though, so take your time to think things through. Hard for the rest of us to really see all the issues yet, since Ink is still new to us.

[joningold]

Well, I'll have a proper think, but my initial reaction is that this would mean, in practice, several hundred thousand white space lines added to a full game script; and 50% less text per screen. Branching content is already highly redundant and one thing ink tries to do is give you maximum visible context so you can read what you've written.

The other thing that's strong about glue IMO is that it's declarative; glue marks serve the function of saying "this sentence is intentionally only half written"; so when scanning old code you can tell what you intended at the time you're writing.

I suspect with the whitespace approach you'd get a lot of bugs where lines where broken unexpectedly; easy to fix but harder to prevent. Ink tries to create a syntax where you don't actually have to test every path to be able to see it'll fit together correctly. I think that's clearer with glue than whitespace (and we've used the same argument when discussing white spacing bullet points in weave.)

[joethephish]

Yeah... for me it boils down to: joining inline text together should be the exception rather than the rule.

Goddamn I'd love to find a way to improve the glue code though...!

Please do continue to push and test our ideas though, it's great to get a different perspective.

[Fahrengeit]

I think that, throwing away complexity of code, it's just another point of view, that not necessary make thing simplier. What if in that example:

"Some content with glue."

breaking lines was intentional? Then we need to make extra space line each time we want to break the line, or make operator for breaking line. In the end it would be the same thing, but from the different angle. Yes, when we are talking about standart choose-your-own adventure books, where is small need for breaking lines, it definitely will look strange with all the glue. So, maybe for such cases with big amount of text (that will be anyway in some games), there should be another way of "gluing" things? Something like:

"<"Bunch of text and it's all on one line">" If it's not so hard to catch in parser, of course.

[joningold]

I don't think I follow your example: aren't you just using ">" to mean glue there anyway?

Tbh, in the standard case, the issue doesn't arise and the line breaks fall as expected. (As you say ignoring code complexity) On Sun, 20 Mar 2016 at 4:53 pm, Fahrengeit notifications@github.com wrote:

I think that, throwing away complexity of code, it's just another point of view, that not necessary make thing simplier. What if in that example:

"Some content with glue."

breaking lines was intentional? Then we need to make extra space line each time we want to break the line, or make operator for breaking line. In the end it would be the same thing, but from the different angle. Yes, when we are talking about standart choose-your-own adventure books, where is small need for breaking lines, it definitely will look strange with all the glue. So, maybe for such cases with big amount of text (that will be anyway in some games), there should be another way of "gluing" things? Something like:

and it's all on one line>

If it's not so hard to catch in parser, of course.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/inkle/ink/issues/28#issuecomment-198962098

[Fahrengeit]

Oh, sorry. Noticed, that quote system deleted the first part of example. It's (without "")

"<"Bunch of text and it's all on one line">"

Using "<" for start, ">" for end and between - As much as needed amount of lines that will combine into one. It's for the cases, about which @micabytes said before - text with long paragraphs, where you need to put glue on every line to prevent breaking.

[joningold]

Ok. For long paragraphs, though, we'd always recommend using an editor with word wrap rather than breaking over lines and using glue; it makes editing easier as there are fewer lines knocking around. On Sun, 20 Mar 2016 at 6:21 pm, Fahrengeit notifications@github.com wrote:

Oh, sorry. Noticed, that quote system deleted the first part of example. It's

" and it's all on one line>"

Using "<" for start, ">" for end and between - As much as needed amount of lines that will combine into one. It's for the cases, about which @micabytes https://github.com/micabytes said before - text with long paragraphs, where you need to put glue on every line to prevent breaking.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/inkle/ink/issues/28#issuecomment-198979101

[micabytes]

I'm not interested in dragging out the discussion too much; as I said, if you're against the idea, I'm Ok with that.

Do note that there is no way (unless you have very weird scripts) that you get 50% less text per screen - that's not how the markdown syntax works. For example:

-   "Tell us a tale, Captain!"
    *   "Very well, you sea-dogs. Here's a tale..."
        * *     "It was a dark and stormy night..." 
                * * *   "...and the crew were restless..." 
                        * * * *  "... and they said to their Captain..." 
                                * * * * *       "...Tell us a tale Captain!"
    *   "No, it's past your bed-time."
-   To a man, the crew began to yawn.

Would look no different in markdown syntax than in the current syntax (markup breaks lines by default).

Wrt to ease of use of glue vs line breaks, I suspect that may be more a matter of what one is used to. I know I certainly was not quite sure what glue was doing at first in the non-trivial examples I've gone through. This is particularly a problem because you can have glue both at the start and the end of a sentence, so unless you track forward to whereever the text flow actually ends, you can't know whether something will suddenly get glued on (same reason why the parser becomes a little tricky to do).

I might have suggested disallowing the addition of glue to the beginning of lines, and only allowing it at the end of lines, but I assume that you had a good reason/some need for that particular bit of syntax (given that it's one of your more complex elements - both to read and parse).