Feature parity with YarnSpinner

[ephread]

First off, I want to really thank you for the project and the work you've done! I was looking for a way to preview my Yarn files, without firing up Unity or using the outdated webplayer.

Since the project seems to be silent these days, I want to help make it shine.

As per the requirements outlined on #15 with DigitalMachinist, I've implemented a lexer based on the original C♯ lexer, but with a couple of tweaks to make it interoperable with the Jison parser. I do agree that, given the small scope of the yarn syntax, it's better to stick to BNF definitions than to roll out an hand-made parser in Javascript. (I'd actually like to study the feasibility of also replacing the hand-coded lexer by a generated one.)

The lexer is missing one major feature, to be on par with the original C♯ one, it's the lookahead ability used to match full-text commands. I'll implement it later.

The results can be seen in the branch lexer-update. I've drafted a test command to check if the tokens are emitted properly.

$ bin/bondage_runner.js test -t <file>

I've tested it against the example files provided by bondage.js, YarnSpinner and some private ones; but I'm not sure all tricky files will be lexed correctly, it's a work in progress.

I plan to quickly update the BNF rules in the parser to match the current capabilities of YarnSpinner. I should have a working prototype supporting the shortcut options, the variables and the conditional statement quite soon, but that's depending on how much work I can put in. I'll make a pull request when the engine will be able to execute (again) the example files.

Before I carry on with the development, I wanted to get you input on the topic and maybe exchange about stuff like the coding style I should follow (I've tried to infer it from the pre-existing linter rules), whether or not something like Flow or Typescript is advisable (it find it easier to write the lexer if the types are enforceable), what issues you may have encountered (so I can take advantage of your experience on the project and avoid pitfalls), etc.

[hylyh]

Oh wow, thank you! Yeah, it has been a little bit since I've worked on this, these contributions will help immensely.

To answer your questions, for coding style as long as you stay within the linter rules you should be fine, looking at your code I don't have any complaints about your style.

I'd rather not change over to typescript but integrating flow is a good idea, I was trying to avoid adding extra steps in needing to compile the library, but since that's already happening for the web build I don't see any problem in needing to compile it to a lib/ directory.

Honestly, I can't remember the pitfalls I had since it's been nearly a year since I've worked on this haha. The only thing I will mention (and judging by how you're splitting out the files I don't think this will be too much of an issue) is that I've been recently considering making this library capable of supporting multiple formats (given a new parser maybe? I haven't put too much thought into it, I've been thinking about a precompiled json format that doesn't need to be parsed on the runner's side. this is just me thinking a bit too far into the future though, something to consider when implementing though I think. mainly finding a good way to add new kinds of results to be returned by the runner)

Thanks again for working on this! I've been meaning to get this implemented but, ya know. I'm excited to see this come together and maybe be some actual use to someone!

[ephread]

I'd rather not change over to typescript but integrating flow is a good idea, I was trying to avoid adding extra steps in needing to compile the library, but since that's already happening for the web build I don't see any problem in needing to compile it to a lib/ directory.

Great, I'll add Flow through Babel, then!

The only thing I will mention (and judging by how you're splitting out the files I don't think this will be too much of an issue) is that I've been recently considering making this library capable of supporting multiple formats (given a new parser maybe? I haven't put too much thought into it, I've been thinking about a precompiled json format that doesn't need to be parsed on the runner's side. this is just me thinking a bit too far into the future though, something to consider when implementing though I think. mainly finding a good way to add new kinds of results to be returned by the runner)

Actually, I plan on making the parser output a JSON AST. This tree would also be the intermediate representation for the runner (something along the lines of jsparser; a bit similar to what you drafted). The runner would traverse the tree, recreate the assignments, boolean expressions, method calls, etc., as Javascript, evaluate them and call the appropriate methods to display the text and ask for input.

Since that all the formats I've encountered so far are inspired by Twine, I guess they would mostly offer the same set of functionalities, albeit through different grammars. Adding new formats would only require a new set lexer/parser and few tweaks to the IR and the runner. Do you have specific formats in mind?

Apparently, the guys behind YarnSpinner created a custom VM to execute the bytecode outputed by their parser. I haven't really dived into that part of the code, but this design is probably overkill for us (and totally beyond my current abilities). I don't think we'll run into performance issues anytime soon.

Ultimately, I'd like to add some stuff to the Yarn syntax —most likely metadata—, such as ways to specify who's talking. Using "Joe:" is far from ideal. The ability to plug in different renderers would be nice also (I'd like to preview my dialogs as if they were unrolling on screen). But for now, we'll stick to the web-based-text-adventure style.

Thanks again for working on this! I've been meaning to get this implemented but, ya know. I'm excited to see this come together and maybe be some actual use to someone!

You're welcome 😉

---

I'm currently working on the parser. One of the biggest challenge is to actually figure out what the grammar of Yarn is supposed to be (particularly inside the command chevrons). I haven't found any document detailing the language in a clear way. The main documentation is a bit outdated and the files used to test the C♯ engine aren't thorough enough to infer the proper behavior. I'm certainly going to poke the guys from the Secret Lab once I get a minimal tree output (which might come at the end of the week).

[hylyh]

Having it output a json ast would be great!

The runner would traverse the tree, recreate the assignments, boolean expressions, method calls, etc., as Javascript, evaluate them and call the appropriate methods to display the text and ask for input.

This sounds great

As for specific formats, I don't have anything specific yet, but I've been discussing a bit with @konistehrad what an ideal format for visual novels would be (which is a bit separate from what bondage is currently but I've been interested in bringing it in that direction for a while). You mentioned adding metadata and renderers which is along similar lines I think, basically I would imagine a visual novel would just be metadata passed into some sort of renderer that can process the visuals.

I'm not a huge fan of yarn's format, partially because of the lack of documentation but also because of things like the metadata you suggested, but if we can compile it into some ast that's generic enough that we could also have, say, a ren'py-like format compile to it as well, then that would be a really neat and flexible, since each format has its applications in different places (yarn/twine formats are more at home being displayed in web browsers over implementing their syntax into game dialogs imo).

As I see it, and I think we're on the same page, there are two separate layers to bondage right now: parsing the yarn format into a json ast, and the actual runner itself. Then in some imaginary future there would be renderers that plug into the runner to display them a la the command line runner, or twine, or visual novels, or someone's own game. As well as separate parsers that compile to the same ast. After the yarn format parser moves along it may be worth moving it to a separate npm module all together.

Sorry for the rant-y thought dump aha! I've been thinking about this a lot recently but you came in at the right time to give me an excuse to think about it a lot more. Does this make sense? It's been a while since I worked on this so I may be off on a few things

[ephread]

Sorry for the lack of updates!

Parser stuff

I've implemented a draft of the parser, which, combined with the lexer output a pretty decent tree for simple files. It doesn't support commands yet (e. g. <<move stuff 33>>), but any yarn file without commands should be properly parsed. The grammar still have some ambiguities and some rules could probably be refactored, but that's a starting point. The work is available in the update-parser branch, here.

I've also replaced bondage_runner.js by bondage with two associated subcommands (run and compile), as I've dropped the test command:

• bin/bondage compile --tokens will print the tokens emitted by the lexer;
• bin/bondage compile --ast will print the json tree;
• bin/bondage run doesn't work anymore (obviously 😛)

Once I'll tackle the runner part, I'll know whether a semantic analyzer would be a good idea or not. I assume (perhaps wrongly) that Javascript will be quite forgiving and make the appropriate type conversions all the time.

It also seems that YarnSpinner automatically create undeclared variables and set them to 0. That would make a symbol table rather useless.

I throw big words about compiler stuff, but, actually, I have no idea what I'm doing, so feel free to object!

About Yarn

I'm not a huge fan of yarn's format

I've actually picked Yarn because of the Yarn editor. I wanted a tool which would let me write dialogues with simple variable-based conditionals in a visual way, preview them quickly and export them in whichever format I needed (I would certainly have written the converter). The Twine editor was super nice, but not really adapted to what I wanted to do; Harlowes really missed the ability to create global macros, or perhaps I failed to see the feature?

Yarn itself, like Twine, missed the ability to attach a character ID to each dialogue, but otherwise, it mostly fit the requirements. I'm not particularly attached to Yarn.

About the visual novel thing

If y'all, more experienced with the visual novel thing, could come up with a language simple for basic uses, but with neat advanced features, that would be awesome. Maybe you should create another repository, bring in some folks, and we could throw ideas around and draft a grammar. Or maybe this should be the repository, because this language should be called Bondage, dunno.

The super package for me, would be:

• as straightforward as Yarn for writers;
• with the ability to simply define who says what (so that a custom runner would be able to color the text, depending on the character, kinda like Ren'py, I guess);
• with the ability to simply define conditional dialogue branches which should be made accessible once in a dialogue loop (rather than wrapping conditionals around the link manually) — maybe a bit beyond the scope, for visual novels, but cool for adventure games;
• bundled with a cool, mindmap-like, visual editor, including a runner for preview;

Finally…

Sorry for the rant-y thought dump aha!

No worries, didn't seemed rant-y at all.

I've been thinking about this a lot recently but you came in at the right time to give me an excuse to think about it a lot more. Does this make sense? It's been a while since I worked on this so I may be off on a few things

Yup!

The only thing we should do, is to try and keep the scope as tight as possible, until we get something usable. Otherwise unless you're on the (TJ Holowaychuk)-level of productivity (I'm really not), we'll try too many things at once and achieve nothing!

[hylyh]

Yeah I'd say definitely move forward with the yarn stuff and if it gets to the point where we'd want to break things out into separate modules we can do that in \~the future~ but that's way out of scope for what is happening now, and having the ast is already a really handy step in that direction anyway.

I just checked out your branch and this is looking really good!

One EXTREMELY minor thing is that in linknodes without identifiers it's probably worth just automatically putting the name in the identifier also, given that that's the expected behavior of yarn anyway and that's one less thing to worry about on the implementation side.

And re: semantic analysis. I think it's ok to lean on javascript to just roll with the punches in this case, I think doing that on our end is a bit overkill and I think having javascript's type quirks in there is a fair compromise. Though I'm looking at this more from a "using this in a javascript game engine" rather than "using this as an out-of-unity yarn tester" so if this direction would have behavioral differences with yarnspinner (I'm not sure if there would be any?) that could be an issue yeah.

[hylyh]

Hey! As an update from my end:

After checking out the basic ast you had set up I had some ideas about what a vn renderer would look like, so I've sketched one out (in a private repo right now). It's based entirely around the concept of basically providing it a "type" of event and all associated metadata (which can be anything) and letting it handle them as it would like, and reporting any input (selecting options) back to the runner. This is essentially what your ast already has, and I really like the idea of basing the structure of it around this: having the runner (bondage) going through the ast and sending the relevant pieces to the renderer one by one as needed, and handling all of the actual Logic of going through a dialog tree on its own. My little sketch is working out a lot better than i expected, and was pretty easy to do! So that feels pretty promising. I may try to write a twine-like renderer (actually I may just re-implement the commandline runner now that I think about it) and tie it to the current bondage master branch to see how it feels with an actual runner.

To keep this back on the ticket topic: I like the ast format youve started building lets keep doing that :P

[ephread]

One EXTREMELY minor thing is that in linknodes without identifiers it's probably worth just automatically putting the name in the identifier also, given that that's the expected behavior of yarn anyway and that's one less thing to worry about on the implementation side.

Good point, you're right, I've fixed it in the branch. I've also fixed the grammar ambiguities (those were the results of a very sleepy brain, I'm afraid) and slightly tweaked the node definition.

I think I'm going to lay down the structure and write a document describing what the AST could/should look like, to keep a clear head (e. g. foo nodes can only contains bar nodes). Maybe that's also a job for Flow, which I have yet to implement.

And re: semantic analysis. I think it's ok to lean on javascript to just roll with the punches in this case, I think doing that on our end is a bit overkill and I think having javascript's type quirks in there is a fair compromise. Though I'm looking at this more from a "using this in a javascript game engine" rather than "using this as an out-of-unity yarn tester" so if this direction would have behavioral differences with yarnspinner (I'm not sure if there would be any?) that could be an issue yeah.

Good for me! Honestly, as long as Yarn doesn't have a well defined grammar written somewhere, we're practically bound to have some behavioral differences. I'm totally fine with that, as I don't want to spend too much time trying to reverse-engineer how YarnSpinner manages the type. We may in the future create a “strict” mode, but you're right, let's lean on javascript for now.

I've worked a bit on the runner, and written a condition solver, which traverse the children of a “condition” nodes and return which “statement” node should be used. It's not pushed yet, but it does let javascript do its magic.

After checking out the basic ast you had set up I had some ideas about what a vn renderer would look like, so I've sketched one out (in a private repo right now).

I'm glad you feel remotivated to work on the project!

This is essentially what your ast already has, and I really like the idea of basing the structure of it around this: having the runner (bondage) going through the ast and sending the relevant pieces to the renderer one by one as needed, and handling all of the actual Logic of going through a dialog tree on its own.

That's how I envisioned it too.

My little sketch is working out a lot better than i expected, and was pretty easy to do! So that feels pretty promising. I may try to write a twine-like renderer (actually I may just re-implement the commandline runner now that I think about it) and tie it to the current bondage master branch to see how it feels with an actual runner.

If you work on a renderer while I work on the runner, we should probably discuss in details how they will interact together, in an another issue maybe?

Should I make a PR for the updates I made, which you would merge into a wip branch on the main repo?

To keep this back on the ticket topic: I like the ast format youve started building lets keep doing that :P

Thanks, I'm planning to!

[hylyh]

I've went ahead and made a branch for you to put a PR up for (wip-yarnspinner-parity)

And right now I'll make a separate ticket for discussing what the output of the runner should look like in the context of making a renderer (this will also include what you mentioned about specifying how the AST should look i think)

[hylyh]

Just wrote a huge amount of stuff in #18, definitely let me know what you think

[hylyh]

So also, for the record, after this thread ive warmed up a lot to the idea of using typescript

[ephread]

Wow, feels like I just took a nap, and 9 days went away, juste like that. Thanks, I'll check out #18!

As you already know, I'm all for TypeScript!

[ephread]

(this will also include what you mentioned about specifying how the AST should look i think)

18 seems to be more about how the runner and the renderer would interact together. We should definitely open another issue dealing with the AST format outputted by the parser.

[hylyh]

Yeah definitely, that ticket is a bit muddled. I'll do that right now

[hylyh]

20

[hylyh]

Added handling of variable assignments, expressions (still in the wip branch). Just conditionals, commands, and functions left to go i believe?

[hylyh]

\o/ thank you so much for the help. sorry this took so long aha