"Extending Twine" proposal suggestions

I've just read the current Extending Twine proposal, and here are the first few major things I would like to alter about it. This is based on my own understanding of what sort of features Harlowe possesses, and opinions of the kind of support I feel they deserve in the story map and editor.

I'm glossing over the requirement that CodeMirror modes use only built-in tokens, as it sounds like custom tokens and appearances are planned in the future. I'm also glossing over the toolbar definition as it doesn't affect Harlowe (which already has a toolbar implemented from scratch with more sophisticated features built-in).

Story formats should be able to define as many bespoke story map reference arrow varieties as they wish, complete with separate parsing conditions, and control the styling of them

The purpose of story map arrows, I feel, is that they should let you discern at a glance the relation between any particular passage and the rest of the story. Beyond links, formats can support many different possible relations.

Consider Harlowe. It currently has the following ways in which passages can refer to other passages, each of which has its own semantics and purpose.

Passage links with literal names already exist, and come in normal and broken varieties.
However, there are Harlowe-specific macro forms of this syntax, such as (link-goto:) and (click-goto:). If a link macro was given something that wasn't a string literal, such as a $variable name, I feel that an outbound link arrow should still be drawn. I imagine it should appear like a broken link, but with a ? in place of an X, to denote that the destination can't be computed statically.
(display:) (which has its own link arrow in Twine 1.4), needs to have a different arrow style from links: the displayed passage is transcluded into the host passage, and no "movement" of the player occurs, but sometimes, it is used in a way to simulate movement.
(goto:) is similar to a link, but triggers with no input. Typically, it's used with timing-based macros like (after:) or (event:). Should it be drawn with the same arrow as a link? Does it have to?
In the future, I plan to add a (redirect:) macro that would behave like (goto:) but, when used, removes the invoking passage's entry in the undo cache, so that one could create a "router" passage that has various branching conditions embedded in it, rather than having to duplicate it for various passages. This would have a meaning closer to (display:) than (goto:).
Other non-link macros make reference to passages by name: (history:) contains "Kitchen" is a common Harlowe idiom. However, in a future version I want to add a special annotation for strings that denotes a string as explicitly being a passage name, something like p"Kitchen", so that in-editor name-checking can occur. This would not only allow the drawing of explicit reference arrows, but also allow broken reference arrows.

So far, only the first kind of reference is represented in Twine 2. I'd like to change that. Being able to control the styling of these arrows in some way is necessary for hinting at the semantics of the arrows - references are much "weaker" than links, UNLESS they're used in something like (link-goto:)

Story formats should also be able to change the manner in which specific passage boxes are drawn, case-by-case, based on their text and tags.

Here's two possible use-cases for this.

What if passages were marked in red if their code contained syntax errors?
(storylet:) is a recent addition to Harlowe, which holds a declarative condition specifying when the passage should be non-linearly visited. Other macros are used to select passages based on these conditions. I've found that the use of storylets as a high-level story-wide routing/redirection tool, above lower-level passage links and branching, can have great results when defining a non-linear story structure. Having storylet passages automatically marked in some way (outside of the tag system, which really only allows four-ish tags to be visible on a passage in the map) would be helpful. I envision some kind of badge (similar to the start passage) or possibly a different border (2 lines?).

Actually, CodeMirror modes should have access to the entire story

The idea that they mustn't just seems ridiculous to me. I mean, what about stuff like:

Colouring broken links in red in the editor? (Something Twine 1.4 already has).
Being able to assume a variable name is misspelled, by cross-referencing it with every other variable name used in the story, and colouring it differently if it has no other uses anywhere?
Adding a message above the textarea like "3 header passages are included" when 3 Harlowe header-tagged passages exist in the story, and whose code will thus affect this passage?

I understand you probably have some performance concerns about allowing full story access to the mode, but the amount of static analysis that this would afford is simply too powerful to pass up.

That's all I've got so far. Thoughts welcome.

...an outbound link arrow should still be drawn. ...appear like a broken link, but with a ? in place of an X ...

If we put aside the performance issues already associated with drawing links (1), I wonder what purpose adding an additional link indicator arrow type like the above mentioned one actually serves.

The broken link indicator arrow servers a useful purpose because it shows the Author that a Passage contains (a) link(s) to a Passage (or Passages) that either doesn't exist yet, or whose name is spelt differently. Thus allowing the Author to correct a lacking in their project.

How will adding a link indicator arrow type that can never be resolved, and that takes up extra real estate in the Story Map, improve an Author's ability to develop their project?

(1) which on a number of occasions has been the main cause of an Author switching their project to being TWEE based instead.

Lots of interesting food for thought here. To avoid any misunderstandings, I want to remind you that the toolbar that's currently in Harlowe will not appear in Twine 2.4 unless it uses the toolbar API, as I mentioned when this first came up. I should add a note to that the doc that story formats can implement their own UI connected to the toolbar (I could see lighterweight formats wanting to add tiny modals), but it'll need to be independent from Twine, as opposed to injecting stuff into Twine-managed DOM. I'm not sure if that changes your feelings about the API.

Story formats should be able to define as many bespoke story map reference arrow varieties as they wish, complete with separate parsing conditions, and control the styling of them

References are very much a first step into the waters and may very well see added functionality. I want to see how formats use them, and where the friction points are, before making any decisions on expanding them.

References are intentionally weak right now; any stronger and format authors will not only need to parse passage text for references, but also update those references when a passage name changes, and do it in a bulletproof way. Twine users are used to renaming passages and having links update for them. In order to do this properly, I think format authors would need to embed their entire parser engine into editor extensions, which some people may feel comfortable with; I imagine others might find that daunting, or just not something they want to spend time on.

Story formats should also be able to change the manner in which specific passage boxes are drawn, case-by-case, based on their text and tags.

Right now, I'm against this because I feel pretty strongly that format extensions should extend the core Twine experience and not replace it. If a format has complete control over the appearance of the passage cards, then one user's story map could easily become incomprehensible to someone who doesn't use the same format.

Badges instead? Maybe. BTW the rocketship badge is going away in 2.4 in favor of a larger indicator that sits outside the passage.

Actually, CodeMirror modes should have access to the entire story

I considered this, but I think--right now--that doing so would actually make life harder for format authors rather than easier. If the entire story structure is exposed to formats, then any time its schema changes, format authors will need to update their formats, and in the meantime users will lose functionality. We'd also need to look at perf as the mode would need to receive a deep clone of the story, as anything changing it (intentionally or not) outside of the React app flow would cause real problems... and we would need to be careful that modes don't retain stale copies of the story object, etc.

To me, the advantage of only exposing the passage text is that it keeps the API extremely stable.

Those are my feelings tonight. I'm going to also share this issue with #format-development on the Twine Discord to see what other folks think.

The broken link indicator arrow servers a useful purpose because it shows the Author that a Passage contains (a) link(s) to a Passage (or Passages) that either doesn't exist yet, or whose name is spelt differently. Thus allowing the Author to correct a lacking in their project.

How will adding a link indicator arrow type that can never be resolved, and that takes up extra real estate in the Story Map, improve an Author's ability to develop their project?

Well, I mean, the point of arrows is to be able to grok the purpose of a passage, to determine its role in the story. Consider this passage, "Work", in my newest game: 2021-10-19 16_53_51-You are SpamZapper 3 1 "Work" is the only hub point for the first part of the game, which dynamically routes to any of the 100+ storylet passages that comprise that part's main loop. However, the only exit shown is the static link out to the second hub, "Hub", which only becomes available much later. Thus, the passage falsely appears as a dead-end, when in truth it has an indeterminate number of exits. Having a ---? arrow representing these exits would better convey this passage's purpose.

And regarding the notion that the arrow is not that meaningful: there already is a much less meaningful arrow in Twine right now, the ↻ self-link arrow, which, while necessary, doesn't really express anything about how the player navigates to and from the passage.

I considered this, but I think--right now--that doing so would actually make life harder for format authors rather than easier. If the entire story structure is exposed to formats, then any time its schema changes, format authors will need to update their formats, and in the meantime users will lose functionality. We'd also need to look at perf as the mode would need to receive a deep clone of the story, as anything changing it (intentionally or not) outside of the React app flow would cause real problems... and we would need to be careful that modes don't retain stale copies of the story object, etc.

To me, the advantage of only exposing the passage text is that it keeps the API extremely stable.

Well… let me bargain down. Rather than "the entire story" (including parts like the tag colours and the zoom level), how about just "every passage's text and tag names"? I don't think it's particularly likely that a future story schema will ever remove or change the notion of a "passage", or at least not that it has a string of text and a list of tag names. As for whether or not

Right now, I'm against this because I feel pretty strongly that format extensions should extend the core Twine experience and not replace it. If a format has complete control over the appearance of the passage cards, then one user's story map could easily become incomprehensible to someone who doesn't use the same format.

Badges instead? Maybe. BTW the rocketship badge is going away in 2.4 in favor of a larger indicator that sits outside the passage.

Let me again bargain down: I don't necessarily expect complete control over their appearance (to the point of turning them into vectorised palm-trees or something) but maybe just having some dominion over a few things, like the box's internal colour, the number (1,2,3), colour and shape of the box's lines, the ability to round the corners, and/or what kind of passage text preview is drawn (other than just the first few dozen characters). (Line colours would replace the "neutral" defautl and still be overridden by Find highlighting etc.). (You may not want to allow specific colours to be specified because of Dark Mode incompatibility, but maybe the colours could be auto-converted to darker/lighter equivalents by converting them to CIELAB and swinging the L around, i.e. 30 becomes 70.)

You may also be wary of, for instance, one passage format defining "double-line passages" to mean one thing that greatly diverges from another format's use of the same. To me, this phenomenon already exists in the sense that CodeMirror tokens can mean completely different things in any given format, and also exists on a story-by-story basis in that any one tag colour can mean any number of things.

I like the idea of displaying various para-links within the Twine UI (references). Making them all with dotted lines seems an unnecessary restriction when you could accept, say, CSS border line properties and/or codemirror color tokens for them, making them much more useful without a lot of overhead. Excluding targetless references also seems counterproductive, especially considering that the UI won’t be updating their text when passage names are changed yet the user would really like to know about lost references.

Another direction to go in is labeling reference arrows. It’s not as visually appealing but could be helpful in some situations.

I’d love to see the reference functions in other people’s story formats in order to parse them for graphing with DotGraph. I can’t say I’m as enthusiastic about writing the functions for my own formats, having done some similar parsing already. It always sounds like it’s going to be simple, but…. One hairy case that comes to mind is a story format author trying to write a parser to handle user-defined reference types.

Actually, CodeMirror modes should have access to the entire story

The idea that they mustn't just seems ridiculous to me. I mean, what about stuff like:
* Colouring broken links in red in the editor? (Something Twine 1.4 already has).

* Being able to assume a variable name is misspelled, by cross-referencing it with every other variable name used in the story, and colouring it differently if it has no other uses anywhere?

* Adding a message above the textarea like "3 header passages are included" when 3 Harlowe header-tagged passages exist in the story, and whose code will thus affect this passage?
I understand you probably have some performance concerns about allowing full story access to the mode, but the amount of static analysis that this would afford is simply too powerful to pass up.

That's all I've got so far. Thoughts welcome.

I'm interested in this feature, I want to implement a story format with Typescript (because I really can't live without type checking apparently), and for proper support the linter would need access to the story JS and maybe some setup passages, so the declared globals can be checked.

With regards to the internal story representation changing: First, there's versioning support, so you could support multiple versions of Twine. Secondly, you could define a more stable representation: A passage is a JS object containing the tags, metadata and content. Story formats could then query passages by name, or get a list of passages. If you're worried about performance and interoperability, instead the story format could register event handlers for things like passageAdded, passageRemoved, passageTagsChanged etc. That way story formats can keep an internal representation of the story if they need, and can keep it in sync with Twine's representation.

klembot / twinejs