As discussed in #150 here is my draft Fountain RFC...

A problem encountered by many Fountain developers is how to store additional data within a source file while adhering to the Fountain 1.1 specification. This brings about the competing goals of adding features to one’s own app while maintaining a position of app agnosticism within the Fountain ecosystem.

Proposal

My proposal for a Fountain 1.2 specification is to include the following example format in the metadata/title page data:

Title: My Script
X-Header: Value

The addition to the spec is that a Fountain app ignores and does not print on the title page any metadata line beginning with:

X-

An app is then free to implement metadata key-values specific to their app.

Any metadata beginning with this prefix is considered an “extension” and will never become an official part of the spec (that is, the full header including the prefix X- will never become an official part of the spec).

There is no other restriction on how a Fountain app implements this as a key-value format:

Title: My Script
Author: Paul W. Rankin
X-Fountain-Mode-Locked-Pages: yes
X-Highland-2-Goal: 750 words
x-revision: 4
X-app-multiline:
    All work and no play makes Jack a dull boy.
    All work and noplay makes JACK a dull boy.

Further:

Headers are case-insensitive, i.e. X-HEADER is the same as x-header
Headers for different Fountain apps may coexist
Headers may contain multiline data consistent with the 1.1 spec
There is no restriction on how headers are named after the X-
Including the app name is not required and only encouraged to prevent naming collisions
Multiple apps may use the same header
There is no restriction on header values

This implementation follows the precedence of extensions to email headers as detailed in RFC822. It is clear to the user that this metadata belongs to the functioning of an app, or upon printing the script it is clear that these values are not printed on the title page.

Also, in my opinion is more aesthetically pleasing to storing a bunch of data in a boneyard comment.

What do I need to do?

In order for this format to become supported all is required from any Fountain developer is that their app ignore any metadata header beginning with X- from printing on the title page:

X-Anything: this line ignored

If your app already limits what is printed on the title page to a select few metadata header values there is nothing to do.

Some edge cases to consider, from my own perspective:

Parsing

How are the meta fields displayed after parsing? Are they hidden from the user or not? If not, it can make documents very messy. I think the data should not be visible to the user while editing, especially as Beat uses JSON objects to store certain stuff.

While the boneyard might be a bit confusing or unappealing, having every metadata field in the title page is problematic for GUI apps as well. For GUI editors, there should probably be a consistent data model in the parser which stores both compatible and incompatible metadata fields. Parser handles compatible ones appropriately, and leaves incompatible fields untouched, putting them back when saving the file.

Then we are not editing pure Fountain anymore and apps need to have specific code in place to store that data, but that's how most of them work already, to be honest.

On-the-fly parsing

What should happen if the user manually adds a X- prefixed field? Does it get parsed immediately after leaving the line, and then gathered into the behind-the-scenes data model?

Data types

Because each app has introduced their own way to store metadata, we should be extra careful when dealing with incompatible metadata. The spec should probably define that unlike normal title page fields, metadata shouldn't contain line breaks to make it sure that we are not breaking metadata coming from another app.

Usability and portability

Consider this metadata block from a random document I found:

/* If you're seeing this, you can remove the following stuff - BEAT: {"FTOutliner: notepanel":"false","Tags":[],"FTOutliner: documentSubName":"\"\"","FTOutliner: indexCardMode":"false","FTOutliner: showLengths":"inline","Revision Color":"green","OLDFTOutliner: oneColumnOutline":"0","FTOutliner: documentAuthor":"\"\"","OLDFTOutliner: currentZoomLevel":"1","FTOutliner: yposition":31,"FTOutliner: trackingIncludesSynopsis":"true","FTOutliner: showVerticals":"false","OLDFTOutliner: cardStateForSections":"true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true","Caret Position":40226,"Window Height":918,"FTOutliner: beatsAreDisplayed":"false","FTOutliner: notepanelWidth":"237","OLDFTOutliner: showPages":"none","OLDFTOutliner: showLengths":"none","FTOutliner: boneyardIsClosed":"true","FTOutliner: showPageDividers":"false","FTOutliner: noteButtonPurple":"true","FTOutliner: showMultipleSceneLines":"false","FTOutliner: ftoutlinerObject":"{\"showNumbers\":\"inline\",\"currentZoomLevel\":1,\"notepanelWidth\":200,\"noteButtonGreen\":true,\"noteButtonYellow\":true,\"noteButtonRed\":true,\"noteButtonMagenta\":true,\"noteButtonPurple\":true,\"noteButtonReview\":false,\"cardState\":[],\"notesAreDisplayed\":false,\"markersAreDisplayed\":false,\"beatsAreDisplayed\":false,\"reviewsAreDisplayed\":false,\"greyonly\":false,\"notepanel\":false,\"showSynopsisInScenes\":false,\"showMultipleSceneLines\":false,\"showMultipleSectionLines\":false,\"allowLowerCaseSections\":false,\"showMultiplewLowerCaseSections\":false,\"showPageDividers\":false,\"boneyardIsClosed\":false,\"scenesAreProportional\":true,\"showSceneNumbers\":\"inline\",\"allowMarkerTextInAllViews\":false,\"colorByPercentage\":false,\"trackingIncludesSynopsis\":true,\"showLengthsInSections\":false,\"showSelectionInfo\":false,\"showRevisions\":false,\"showLengths\":\"inline\",\"showPages\":\"inline\",\"oneColumnOutline\":false,\"indexCardHeight\":5,\"indexCardMode\":false,\"showVerticals\":false,\"autoWiden\":false,\"widenedColumnsGetAutoHeight\":false,\"outdoor\":0,\"saturation\":50,\"windowSize\":{\"x\":0,\"y\":0,\"width\":0,\"height\":0}}","FTOutliner: thePanelWidth":943,"FTOutliner: showSynopsisInScenes":"false","FTOutliner: cardStateForScenes":"false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false","FTOutliner: y-position":0,"Sidebar Visible":true,"OLDFTOutliner: y-position":0,"FTOutliner: xposition":112,"Revision Mode":false,"FTOutliner: thePanelHeight":1041,"Active Plugins":[],"FTOutliner: greyonly":"false","Revision":{"Removed":[],"RemovalSuggestion":[],"Addition":[[72575,84,"green"]]},"OLDFTOutliner: x-position":0,"Window Width":1336,"OLDFTOutliner: cardStateForScenes":"true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true","FTOutliner: documentName":"THIS AND THAT","FTOutliner: oneColumnOutline":"0","FTOutliner: markersAreDisplayed":"false","Page Size":0,"OLDFTOutliner: thePanelHeight":800,"Review Ranges":[{"string":"I this really consistent with the rest of the story or is it just a joke?","range":[4570,171]}],"OLDFTOutliner: indexCardHeight":"3","FTOutliner: showRevisions":"false","FTOutliner: currentZoomLevel":"1","OLDFTOutliner: showNumbers":"none","FTOutliner: showProportional":"1","OLDFTOutliner: indexCardMode":"false","Sidebar Width":332,"FTOutliner: showNumbers":"none","FTOutliner: notesAreDisplayed":"false","FTOutliner: noteButtonReview":"false","FTOutliner: cardStateForSections":"false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false","OLDFTOutliner: thePanelWidth":800,"FTOutliner: noteButtonMagenta":"true","FTOutliner: reviewsAreDisplayed":"false","TagDefinitions":[],"FTOutliner: noteButtonYellow":"true","FTOutliner: noteButtonRed":"true","FTOutliner: showMultipleSectionLines":"false","FTOutliner: showPageDividersAlways":"false","CharacterGenders":{},"OLDFTOutliner: showProportional":0,"FTOutliner: noteButtonGreen":"true","FTOutliner: showPages":"none","FTOutliner: x-position":0,"FTOutliner: allowLowerCaseSections":"false","FTOutliner: indexCardHeight":"10","Changed Indices":{"2388":"green"},"FTOutliner: windowSize":{"y":0,"x":48,"width":841,"height":1095}} END_BEAT */

If that was broken down to X- fields, it could look something like this:

X-FTOutliner-notepanel: false
X-Tags: []
X-FTOutliner-documentSubName: 
X-FTOutliner-indexCardMode: false
X-FTOutliner-showLengths: inline
X-Revision Color: green
X-OLDFTOutliner: oneColumnOutline: 0
X-FTOutliner-documentAuthor: 
X-OLDFTOutliner: currentZoomLevel: 1
X-FTOutliner-yposition: 31
X-FTOutliner-trackingIncludesSynopsis: true
X-FTOutliner-showVerticals: false
X-OLDFTOutliner: cardStateForSections: true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true
X-Caret-Position: 40226
X-Window-Height: 918
X-FTOutliner-beatsAreDisplayed: false
X-FTOutliner-notepanelWidth: 237
X-OLDFTOutliner: showPages: none
X-OLDFTOutliner: showLengths: none
X-FTOutliner-boneyardIsClosed: true
X-FTOutliner-showPageDividers: false
X-FTOutliner-noteButtonPurple: true
X-FTOutliner-showMultipleSceneLines: false
X-FTOutliner-ftoutlinerObject: {"showNumbers:\"inline\",\"currentZoomLevel:1,\"notepanelWidth\":200,\"noteButtonGreen\":true,\"noteButtonYellow\":true,\"noteButtonRed\":true,\"noteButtonMagenta\":true,\"noteButtonPurple\":true,\"noteButtonReview\":false,\"cardState\":[],\"notesAreDisplayed\":false,\"markersAreDisplayed\":false,\"beatsAreDisplayed\":false,\"reviewsAreDisplayed\":false,\"greyonly\":false,\"notepanel\":false,\"showSynopsisInScenes\":false,\"showMultipleSceneLines\":false,\"showMultipleSectionLines\":false,\"allowLowerCaseSections\":false,\"showMultiplewLowerCaseSections\":false,\"showPageDividers\":false,\"boneyardIsClosed\":false,\"scenesAreProportional\":true,\"showSceneNumbers\":\"inline\",\"allowMarkerTextInAllViews\":false,\"colorByPercentage\":false,\"trackingIncludesSynopsis\":true,\"showLengthsInSections\":false,\"showSelectionInfo\":false,\"showRevisions\":false,\"showLengths\":\"inline\",\"showPages\":\"inline\",\"oneColumnOutline\":false,\"indexCardHeight\":5,\"indexCardMode\":false,\"showVerticals\":false,\"autoWiden\":false,\"widenedColumnsGetAutoHeight\":false,\"outdoor\":0,\"saturation\":50,\"windowSize\":{\"x\":0,\"y\":0,\"width\":0,\"height\":0}}
X-FTOutliner-thePanelWidth: 943
X-FTOutliner-showSynopsisInScenes: false
X-FTOutliner-cardStateForScenes: false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false
X-FTOutliner-y-position: 0
X-Sidebar Visible: true
X-OLDFTOutliner: y-position: 0
X-FTOutliner-xposition: 112
X-Revision Mode: false
X-FTOutliner-thePanelHeight: 1041
X-Active Plugins: []
X-FTOutliner-greyonly: false
X-Revision: { X-Removed: [], X-RemovalSuggestion: [], X-Addition: [[72575, 84, "green"]] }
X-OLDFTOutliner-x-position: 0
X-Window-Width: 1336
X-OLDFTOutliner-cardStateForScenes: true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true,true
X-FTOutliner-documentName: THIS AND THAT
X-FTOutliner-oneColumnOutline: 0
X-FTOutliner-markersAreDisplayed: false
X-Page-Size: 0
X-OLDFTOutliner: thePanelHeight: 800
X-Review-Ranges: [{ "string": "I this really consistent with the rest of the story or is it just a joke?", "range": [4570, 171] }]
X-OLDFTOutliner-indexCardHeight: 3
X-FTOutliner-showRevisions: false
X-FTOutliner-currentZoomLevel: 1
X-OLDFTOutliner-showNumbers: none
X-FTOutliner-showProportional: 1
X-OLDFTOutliner-indexCardMode: false
X-Sidebar-Width: 332
X-FTOutliner-showNumbers: none
X-FTOutliner-notesAreDisplayed: false
X-FTOutliner-noteButtonReview: false
X-FTOutliner-cardStateForSections: false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false
X-OLDFTOutliner-thePanelWidth: 800
X-FTOutliner-noteButtonMagenta: true
X-FTOutliner-reviewsAreDisplayed: false
X-TagDefinitions: []
X-FTOutliner-noteButtonYellow: true
X-FTOutliner-noteButtonRed: true
X-FTOutliner-showMultipleSectionLines: false
X-FTOutliner-showPageDividersAlways: false
X-OLDFTOutliner-showProportional: 0
X-FTOutliner-noteButtonGreen: true
X-FTOutliner-showPages: none
X-FTOutliner-x-position: 0
X-FTOutliner-allowLowerCaseSections: false
X-FTOutliner-indexCardHeight: 10

That's 70 lines of header data at the beginning of a screenplay, which looks a bit scary. The power of JSON data model in Beat is that plugins can store document-specific metadata in it. In The example above, it looks like three different versions of FTOutliner were used, the last of which started storing the data in a single JSON block inside the JSON, which is more maintainable, but still not very appealing.

Current version of Beat also stores a UUID for each scene heading, which can amount to hundreds of UUIDs, and seeing them listed in the header might not be feasible either.

Here's the data block from Tutorial alone:

/* If you're seeing this, you can remove the following stuff - BEAT: {"Review Ranges":[{"range":[4059,7],"string":"Hey there! This is a review note. Press ⌘R to review any selected text."}],"Window Width":1123,"Caret Position":0,"Heading UUIDs":[{"string":"INT. SCHOOL ROOM - DAY","uuid":"95009A7F-3A9C-4F36-B76F-8DB5157718A1"},{"string":"EXT. PARK - DAY","uuid":"C68537B9-5C6A-4D95-B0A3-D58834011F11"},{"string":"# This is a new section","uuid":"B3390140-8562-4D00-9C27-9C54912BCAC7"},{"string":"INT. TUNNEL - NIGHT ","uuid":"6AEE0A4C-F3A1-4B34-B0EF-6AAA2355AB76"},{"string":"INT.\/EXT. BEACH HOUSE - NIGHT","uuid":"2CC2D8D7-9C4E-4E78-B04F-7C6FF6E9F609"},{"string":"EXT. STREET - NIGHT [[BLUE]]","uuid":"4948A232-966C-481D-A2D3-A93CD8CBF230"},{"string":"INT. NIGHT CLUB - NIGHT [[RED]]","uuid":"E57DCD8C-2D53-42C1-9E75-D4D9F04E0A06"},{"string":"EXT. UNIVERSITY CLASSROOM - MORNING","uuid":"592FAAAE-C6F1-440F-93E0-973FD11537CE"}],"Changed Indices":{"115":"blue"},"Revision":{"Removed":[],"RemovalSuggestion":[],"Addition":[[4119,9,0]]},"Active Plugins":[],"Window Height":1017,"Revision Level":0} END_BEAT */

Turned into a title page with X-fields:

Title: Beat Tutorial
Credit: Written by
Author: Lauri-Matti Parppei
Draft date: Version 3, 6.8.2023
Contact:
beat@kapitan.fi
www.beat-app.fi
X-Review-Ranges: [{"range": [4071, 7], "string": "Hey there! This is a review note. Press ⌘R to review any selected text." }]
X-Window-Width: 1123
X-Caret-Position: 567
X-Heading-UUIDs: [{"string": "INT. SCHOOL ROOM - DAY", "uuid": "36786649-3AEC-4D8F-ACDB-19C34A1BE2E8" }, { "string": "EXT. PARK - DAY", "uuid": "BCEE3DCA-5BBC-40DC-B0B2-82DC10696C3E" }, { "string": "# This is a new section", "uuid": "A4FA0202-519D-48F4-8B86-B7533F3AB016" }, { "string": "INT. TUNNEL - NIGHT ", "uuid": "D88EA76E-1BD2-4613-91A9-482000DE870E" }, { "string": "INT./EXT. BEACH HOUSE - NIGHT", "uuid": "C97AD9BD-CEF1-4B82-AABC-AAC42842C915" }, { "string": "EXT. STREET - NIGHT [[BLUE]]", "uuid": "1C366CEF-6173-4DAE-8CF3-5BA2CE447BE7" }, { "string": "INT. NIGHT CLUB - NIGHT [[RED]]", "uuid": "2061FF70-97F4-4535-BB13-0A3A5FABB177" }, { "string": "EXT. UNIVERSITY CLASSROOM - MORNING", "uuid": "6AF8ABAE-3EA9-4EFA-BDF0-6C2636681B29"}]
X-Changed-Indices: {"115": "blue"}
X-FTOutliner-windowSize: {"y": 0,"x": 0,"width": 800,"height": 800 }
X-FTOutliner-ftoutlinerObject: "{\"scenesAreProportional\":true,\"showSynopsisInScenes\":false,\"showPageDividers\":false,\"notesAreDisplayed\":false,\"markersAreDisplayed\":false,\"beatsAreDisplayed\":false,\"reviewsAreDisplayed\":false,\"allowMarkerTextInAllViews\":false,\"colorByPercentage\":false,\"trackingIncludesSynopsis\":true,\"showMultipleSceneLines\":false,\"showMultipleSectionLines\":false,\"allowLowerCaseSections\":false,\"showLengthsInSections\":false,\"showSelectionInfo\":false,\"showRevisions\":false,\"showLengths\":\"inline\",\"showNumbers\":\"inline\",\"showPages\":\"inline\",\"oneColumnOutline\":false,\"currentZoomLevel\":1,\"indexCardHeight\":5,\"cardState\":[{\"isExpanded\":true,\"collapsedSection\":false},{\"uuid\":\"A4FA0202-519D-48F4-8B86-B7533F3AB016\",\"isExpanded\":true,\"collapsedSection\":false}],\"indexCardMode\":false,\"showVerticals\":false,\"autoWiden\":false,\"widenedColumnsGetAutoHeight\":false,\"greyonly\":false,\"boneyardIsClosed\":false,\"notepanel\":false,\"notepanelWidth\":200,\"outdoor\":0,\"saturation\":50,\"windowSize\":{\"x\":0,\"y\":0,\"width\":0,\"height\":0},\"noteButtonGreen\":true,\"noteButtonYellow\":true,\"noteButtonRed\":true,\"noteButtonMagenta\":true,\"noteButtonPurple\":true,\"noteButtonReview\":false,\"showMultiplewLowerCaseSections\":false}","Revision": {"Removed": [],"RemovalSuggestion": [],"Addition": [[4131, 9, 0]]}
X-Active-Plugins: []
X-Window-Height: 1017
X-Revision-Level: 0

Any of this is not crucial for the actual content, though revision markers and reviews might still be important, of course, but those would be broken anyway when editing in another app. (I'm working on an alert about this when opening a file with revisions/reviews that was edited outside Beat in between.)

I consider my metadata to be somewhat optional, so it's not "header-worthy" in that sense.

EOF metadata

So, how about taking a cue from both approaches and storing these fields in an EOF boneyard? That way, apps that don't support any metadata tricks, shouldn't become too unusable because of that data mass at the beginning, and the ones that do, can easily strip out that stuff, never exposing it to the user.

Metadata would be parsed only when loading the document, and put in when saving it, removing the risk that the user would try to insert it manually when not allowed to. The EOF block might still grow pretty big though, especially when using multiple GUI apps to edit the file.

Thanks for considering this so thoroughly! Given there's a lot to cover I hope you don't mind if I reply inline.

How are the meta fields displayed after parsing? Are they hidden from the user or not? If not, it can make documents very messy. I think the data should not be visible to the user while editing, especially as Beat uses JSON objects to store certain stuff.

This is interesting because in my Emacs mode I don't think there is any parsing in the same sense. There is syntax highlighting, which is basically on-the-fly regular expression matching (not parsing in the strict sense) and then there is exporting, which only happens in response to the user action. There is a "matcher" function for the syntax highlighting, which then defers to Emacs's internal syntax highlighting mechanism. For exporting there is a function that just returns a list of couplets:

((title . "My Script") (credit . "written by") ...)

But I don't think this is the kind of parsing to which you're referring; from what I imagine a macOS text editing application parses the Fountain source into some internal rich text data before displaying it...?

I think hiding the fields should be left to the app to choose, so I would add the following to the draft:

How does the user interact with X- headers?

There is no restriction on whether a X- header is added/managed by an app or a user. Like all Fountain specifications, this format is designed to be user-editable. For example, a simple header may only be added by a user:
X-Locked: yes
An app may choose to show or hide any X- header from the user for possible aesthetics/usability reasons. This is left to the developer’s preference. X- headers demand no special treatment.

Here I think also the technical abilities of our respective users would come into play, i.e. it would make more sense to hide X- headers for Beat users (maybe under some disclosure button?) but not for Fountain Mode users. That said, in Emacs it is customary to add options for literally everything.

What should happen if the user manually adds a X- prefixed field? Does it get parsed immediately after leaving the line, and then gathered into the behind-the-scenes data model?

I think again this is a difference in the way parsing works in a macOS app vs Emacs. For syntax highlighting, Emacs would apply the highlighting straight away, but this does not constitute parsing, i.e. there would not be any internal variable updated until the user manually calls the export command.

Consider this metadata block from a random document I found:

Yes we're getting quite messy now. In this case I don't think it's great UX for Beat to shift data from the JSON within the EOF boneyard to the header of the file. It benefits no one. And given data within a boneyard is effectively ignored by other apps, is there any benefit for you to switch from using JSON to using X- headers for this? Especially since the Beat data is not intended to be user-editable?

I don't think this precludes the addition to the spec, i.e. that Beat should ignore X- headers from printing on the title page (for theoretical cross-app compatibility) but I do think there is little upside (and considerable downside) to Beat switching from the existing JSON block inside a EOF boneyard.

p.s. I just realised the best analogue to parsing in Emacs. This would be when the "mode" (Fountain Mode) is activated, i.e. when opening a script. Here if we have this header:

X-Format: stageplay

Then this would set a document-local variable that would immediately affect how the program displays and operates on the script.

I've also created a discussion thread on the Fountain Mode repo about this idea, because I'll be directing other developers to that so you aren't burdened with maintenance if this gains traction. I am of course still subscribed here so feel free to discuss on either thread.

The example above, it looks like three different versions of FTOutliner were used, the last of which started storing the data in a single JSON block inside the JSON, which is more maintainable, but still not very appealing.

You sum up not only my data storing methods but also my progress as a coder (and probably as a human being in general). More maintainable but still not very appealing. 😊

Seriously though, just wanted to chime in here, not least because besides being a Beat user (and a terrible coder) I too am playing around with creating a fountain editor.

And just like you, I have had lots of thoughts about how to best store settings and app data with the document. For what it's worth, I find the current Beat implementation (which, if I understood the conversation on google gropus correctly used to be what Highland used too?) a very simple yet effective solution. I understand why one would like to de-bloat the document file and have metadata in its own file, very much the way Highland does it now, but IMHO (where H equals "humble") that sort of negates the main idea behind Fountain to begin with: app-agnostic text-only readability.

Personally, I also wouldn't be too happy about adding matadata tags to the top of the document; I'm already no fan of the title page tags, and have yet to implement them in my app because inconsistent with the rest of the parsing. So to me, tags at the top would further complicate the access to the screenplay material if opening the file in a non-suppporting app (a causual user might not even understand they were looking at a screenplay and not an unreadable data file) plus would create some parsing issues that I'd be vary wary about.

Some arguments for the settings-in-a-comment-block approach, from the top of my mind:

Sits at end of file, so does not get in the way of content if opened in non-supporting apps
Also by sitting at the end of the file, does not introduce confusion to indices (If my app is supposed to hide the proposed x-tags from the user, should they be treated as still-present, so that indices before and after are non-continous? Or stripped out just like we currently strip out the long hidden block at the end?)
From a developer perspective: is unambiguos as far as where to put it, as it sits at the end after everything else. It is stripped off from behind at open, and appended at the end on save.
Is enclosed in already-parsable /* ... */ so will never print regardless of which app it is opened in, as long as it follows Fountain specs
Can contain literal information to the user (if you're seeing this...) for clarification if opened in such app

Some immediate worries about x- tags:

Takes a full iteration of the script at open in order to read and apply all settings
Makes it possible to willfully or mistakenly add conflicting tags (x-paperSize: A4... x-paperSize: USLetter) a. if we assume later overrides former, this opens to CSS-type confusion of the but I did set it to A4 goddamit! kind b. if we assume setting A is valid until we encounter setting B, app will have to parsed differently in different places, and will also need to constantly cycle up and down to watch enclosing tags that may or may not affect any current line
Demands a full iteration of the script if the user types their own x- tag, in order to adjust the entire (or following-only?) material to the typed tag (and/or, in the case of 2a, to check if overridden by a later x-tag, in which case the full iteration only results in the just-added tag being ignored). All in all, a potentially expensive process to iterate and evaluate repeatedly instead of only on open.

But all these arguments aside, what I think should be fundamental to the use of fountain, is that it is primarily a simple, text-only way to write a screenplay. All technical and non-straight-forward stuff should be exceptions to that very tightly kept norm, and also fully possible to not ever encounter for a user who is not savvy about them. This holds true even to current over-technical add-ons like curly-brace metadata or other potentially intimidating stuff, which will never appear without the user explicitly putting it there. Contrarily, app metadata (arguably, the app talking to itself) will be inserted without user consent and thus would confuse rather then help.

Those are my most immediate two cents over morning coffee!

I'll reply to the both of you before eating lunch!

But I don't think this is the kind of parsing to which you're referring; from what I imagine a macOS text editing application parses the Fountain source into some internal rich text data before displaying it...?

Beat parses Fountain to a set of Line objects, and assigns a type for each of them, similar to DOM. The lines store their own parsed content from note ranges to formatting, and you can also request metadata from them. This way, you can gather all sorts of data from the screenplay and interpret the content much more deeply, and all assisting editor views (outline, timeline, etc.) and plugins get that data for free with no additional parsing.

That data is also used to create the rich text representation in editor, so in that sense, Beat doesn't use traditional syntax highlighting, but continuously parses any changes to the text. (Thanks to Hendrik Noeller for the basis of that code!) Parser is actually completely disconnected from the editor representation, and every change to the text storage has to be carefully tracked. During the early days I managed to get it out of sync now and then, which caused data loss and all sorts of trouble.

Takes a full iteration of the script at open in order to read and apply all settings

This is true if they can sit anywhere in the document, but not if it's included either in the title page or at the end, because you can detect it reading just the first or last lines in the screenplay. If it's formatted like Compatible key: Value and an empty line is required to terminate the metadata block, you'll get all the needed stuff in a matter of milliseconds.

But yes, on the other hand, is the title page actually the best place for that data? Fountain is mostly very semantic, and it's pretty clear what is invisible (by default) and what isn't, except for the title page, which needs some English-language keywords, typed correctly. Adding metadata would be even more confusing for most non-technical users, and that also arguably isn't really a part of the title page itself.

The larger question probably would be about what metadata is actually feasible to transport between apps?

Paper size could be one, but because Fountain isn't WYSIWYG, it's entirely up to the apps to implement the output, and it will be different from application to application. Maybe document type as well (stageplay, screenplay, novel) but different formats are not really widely supported, as Fountain is strictly aimed at writing traditional fiction films/TV. Novel mode on both Beat and Highland 2 is mainly a hack, which has nothing to do with Fountain, and Highland doesn't even change how the editor (AFAIK) when outputting to novel format.

Formatting is actually the most feasible metadata field, because it defines a scope for the whole output. Adding a standard metadata for formatting would still require at least a couple of the more popular applications work in unison the way they do with screenplays.

Hm, okay — maybe I misread part of the suggestion — I took it as though any x- line at any point in the app should be interpreted as an app-readable setting. So the suggestion is only to have it between the title page and the screenplay content?

Then some of my arguments are moot (especially the iteration part) but the most important one still stands: Why would we want to take the storage format, whose purpose is to make it as human readable as possible, and add techy stuff to the beginning?

I know nothing about emacs, but I assume that it too can read files from behind — so if anything, it would make a lot more sense to add the x- lines at the end rather than the beginning. In fact, then, having a section at the end, exactly like today, would work for everyone:

/* if you can read this... disclaimer text
x-paperSize: USLetter
x-someOtherKey: someValue
*/

But regardless, a UI-based app like Beat or Highland or Slugline or even my own little embryo of an app would want to keep any storable settings (paper size, app behavior, whatnot) or document self-references that aren't part of the fountain data (line ids, alts, revisions) as part of internal arrays, reachable either through dialogs or just implied in the displayed text, and converted to and from data only at the time of saving/opening.

What am I missing here, that would make user-readable/writeable tags useful for the screenwriter?

What am I missing here, that would make user-readable/writeable tags useful for the screenwriter?

I'm thinking this is more about having certain shared metadata headers, so it wouldn't be such a "wild west". (Is this a proper English idiom?)

But maybe some readable fields at EOF with app-specific blocks could make sense:

/* Fountain metadata
X-Paper-Size: A4
X-Format: Novel
X-Beat-Data: { truckloads of JSON }
X-FTWriter-Data: { more data } 
X-Best-Fountain-App-Data: { even more data }
*/

Welcome @ftolsson. It's great to hear another Fountain app will be joining the ecosystem!

Just quickly I think there is some confusion about the limits of the X- suggestion. To be clear, I am not suggesting that apps currently using (or planning to use) a boneyard block of data within the source file should replace this with X- headers. From the examples @lmparppei gave above I think this clearly shows how quickly this would become unwieldy and a bad user experience.

The suggestion is really limited to: Fountain apps should ignore (i.e. not print) metadata/title page lines beginning with X-. I think it's important to keep this limit in mind through further discussion.

Hm, okay — maybe I misread part of the suggestion — I took it as though any x- line at any point in the app should be interpreted as an app-readable setting. So the suggestion is only to have it between the title page and the screenplay content?

Good observation. I should make clear that X- headers are just part of the BOF metadata/title page block. They are really just additional metadata/title page lines, not new elements. They don't get any additional syntax rules, except that they are not printed. This:

Title: My Script
Credit: written by
Author: Paul W. Rankin
X-Locked: true

FADE IN:...

is equivalent to this:

X-Locked: true
Title: My Script
Author: Paul W. Rankin
Credit: written by

FADE IN:...

I know nothing about emacs, but I assume that it too can read files from behind — so if anything, it would make a lot more sense to add the x- lines at the end rather than the beginning. In fact, then, having a section at the end, exactly like today, would work for everyone:

To be clear, I'm not suggesting here a universal metadata storage and exchange format. I'm not suggesting that different apps need to learn how to read metadata from other apps. If you wish to put a bunch of XML inside a boneyard at EOF, that is perfect. If you wish to use X- lines inside note elements throughout the script, that is also fine. I don't think it would be possible to establish the kind of consensus throughout the Fountain community to establish this kind of metadata exchange. The suggestion here is much more humble — all that is required is that apps ignore header lines starting with X-.

Then, if an app chooses, it can use any X- header extensions that it recognises:

X-Beat-Tags: foo bar baz
X-Fountain-Mode-Scene-Heading-Format: bold underlined
X-Ftolsson-Papersize: A4

Here Beat might read the tags foo bar baz, Fountain Mode reads the scene heading format as bold underlined and Ftolsson app reads A4 papersize. There is no interpolation expected.

But yes, on the other hand, is the title page actually the best place for that data? Fountain is mostly very semantic, and it's pretty clear what is invisible (by default) and what isn't, except for the title page, which needs some English-language keywords, typed correctly. Adding metadata would be even more confusing for most non-technical users, and that also arguably isn't really a part of the title page itself.

This really is the question at hand. @lmparppei you've demonstrated that committing all metadata to X- headers is undesirable. So Beat will likely continue to use your JSON data in an EOF boneyard approach. The question then becomes, are there any key-values you would want to take from the EOF boneyard and put in the BOF metadata/title page as X- headers?

If not, then it's a simpler question of, if other apps begin to use X- headers in the way I'm suggesting, would Beat agree to ignore these and not print on the title page?

My desire for this approach is that Fountain Mode users have asked over the years to be able to set various metadata values at BOF this way. Fountain Mode already uses one of these headers:

format: stageplay

I never sought community consensus on this. I've never tested to see what other apps print this on the title page. I'd like to deprecate it in favour of an X- header because then it would be clear that it should not be printed.

The larger question probably would be about what metadata is actually feasible to transport between apps?

I guess this is where the fun starts. And this is partly why I think some key-values ought to be stored in the BOF metadata. I don't think we're going to see different apps reading JSON/XML/YAML blocks within boneyards or anything. But I think there may be space for a few apps to agree on some little things, like:

X-Format: stageplay
X-Papersize: a4
X-Scene-Heading-Format: bold doublespaced

Maybe. Each of these would be like a treaty established between countries. I think it would be fun.

I never sought community consensus on this. I've never tested to see what other apps print this on the title page. I'd like to deprecate it in favour of an X- header because then it would be clear that it should not be printed.

That would be printed by Beat.

Maybe. Each of these would be like a treaty established between countries. I think it would be fun.

I'd be interested in implementing X- fields, even if just as a proof of concept with no actual use for now. The prefix would differentiate them from any other fields. I'm in the process of rewriting/reimplementing my cross-platform document object, so it would be interesting to see how this would affect it. It requires some changes to loading and initially parsing documents, but that's something that I'd have to do anyway.

Maybe. Each of these would be like a treaty established between countries. I think it would be fun.

Some of these would no doubt make sense to have travel between apps — sceneheading formats, paper sizes etc are perfect examples. It would take some agreeing/negotiating, yes, but if done correctly it might be useful.

Making a line that starts with x- (caps? both?) AND has a colon (or else the test fails?) its own unprintable format does sound super simple if everyone agrees. By "everyone" I guess ideally we're talking more than three people one of whom only has half an app on his hard drive. But yeah. :)

Counter question though: Must the x- lines go at BOF to be valid? I would still prefer not to put non-printing material at the top myself, but would much rather have them in the end block. Is Fountain Mode unable to parse lines from behind, or would it be able to respect an X-Scene-Heading-Format: Bold line that my app tucked away in the EOF boneyard?

Counter question though: Must the x- lines go at BOF to be valid? I would still prefer not to put non-printing material at the top myself, but would much rather have them in the end block. Is Fountain Mode unable to parse lines from behind, or would it be able to respect an X-Scene-Heading-Format: Bold line that my app tucked away in the EOF boneyard?

This is something I'm wondering about too. I think the title page block should only contain title page information. Otherwise we could be opening a can of worms here regarding cross-app-compatibility. I'd personally want Beat files to open as-is in other apps, and I'm pretty sure the likes of Highland, Logline and Slugline won't be implementing this now or ever. Some people also use 'afterwriting to convert pure text files to Fountain, so all of them should support those fields as well.

Especially if they've been invisible to the user until then, it can be confusing to see A4 printed on the title page for no reason.

Counter question though: Must the x- lines go at BOF to be valid? I would still prefer not to put non-printing material at the top myself, but would much rather have them in the end block. Is Fountain Mode unable to parse lines from behind, or would it be able to respect an X-Scene-Heading-Format: Bold line that my app tucked away in the EOF boneyard?

This is something I'm wondering about too. I think the title page block should only contain title page information. Otherwise we could be opening a can of worms here regarding cross-app-compatibility. I'd personally want Beat files to open as-is in other apps, and I'm pretty sure the likes of Highland, Logline and Slugline won't be implementing this now or ever. Some people also use 'afterwriting to convert pure text files to Fountain, so all of them should support those fields as well.

If the suggestion is to add X- headers outside of the BOF title page/metadata block there are only two options, either add a new element to the spec or partition it within a boneyard.

I think it's best not to attempt to add a new element to the spec. This will be very difficult.

There are a few reasons I personally dislike the EOF boneyard approach...

I don't like the aesthetic; while you're writing your words are going between your previous writing and this lump of code. Yes it can be hidden but I don't want to hide things from my users unless they specifically hide them. (This is probably more relevant to the culture of Emacs than anything else.)
It doesn't sit right with me to have meaningful data in a comment. I'm just weird about this. I feel like a user should be able to strip all comments without causing anything to break.
If we already have apps that store meaningful data in an EOF boneyard block, then the assumption is that this block is owned and managed by that app, and does not and should not contain meaningful data from another app. I would hesitate to add data to a boneyard that another app had created because I don't know how it will persist or be managed.
I think this approach disempowers users. I want my users to feel they can edit this data. If the data is not intended as user-editable I think the appropriate place is in an EOF boneyard.

I think the can of worms has already been opened; TextPlay already uses the title page block as configuration (and as mentioned, Fountain Mode has used the format key for a while now).

But a lot of Fountain apps are already compatible with the BOF X- header approach. I've started a small table to track what I've tested here: https://github.com/rnkn/fountain-mode/discussions/144

App	Compatibility
Fountain Mode	No
Highland 2	Yes
Beat	No
afterwriting	No
Screenplain	Yes
Wrap	Yes
TextPlay	Yes
BetterFountain	Yes

(These are just apps where I've had interactions with the authors; I know there are many more on https://fountain.io.)

p.s. I knew there was another Fountain app that already used metadata key-values for configuration: BetterFountain (VSCode). I think we can probably consider this an established convention.

Well, most importantly, this is what the specs say (screenshot from fountain.io, hadn't ever registered this until yesterday)

So I think we should find it pretty much established that any key:value pair (x- or not) that our app doesn't have a direct knowledge of should default to non-printing.

The next step would be discussing if this is something we want to share between us and if so how, but regardless:

The suggestion is really limited to: Fountain apps should ignore (i.e. not print) metadata/title page lines beginning with X-. I think it's important to keep this limit in mind through further discussion.

This part is more or less something we should do already per the specs, unless I'm completely misreading them. (Which, by the way, makes Slugline — not present in your list, but a NO — actually not adhering to standards there. Huh.)

Excellent catch @ftolsson! So much discussion when we could have just made a closer read of the 1.1 spec! In this case Fountain Mode does not adhere to the spec and I have some work to do today :)

I personally like the idea of using X- for my own metadata extensions because it makes it clear to the user that these are extensions, and it avoids naming collisions when round-tripping between apps.

But as far as this extension to the spec goes, it should be limited even further to just the following:

Any metadata key beginning with X- is considered an “extension” and will never become an official part of the spec.

Not that I think it's likely that there will be a Fountain 1.2 that introduces any new metadata keys...

Part two is the fun part and I think will be much more organic — how much do we want to share configuration between apps?

Well, most importantly, this is what the specs say (screenshot from fountain.io, hadn't ever registered this until yesterday)

Ha ha, I also had missed that completely. Many users asked that Beat would display any page elements they had, so I followed that idea way back.

Title page fields prefixed with X- will now be ignored in export. My whole title page parsing code needs a rewrite, so I'll look into that deeper later, but at least it won't produce any weird output on the title page if the screenplay originates from an app that uses title page metadata.

Given this revelation that additional keys should be ignored anyway, I wonder if there's any point to any special consideration for X- headers? Sorry @lmparppei I see you've already spent time on this.

Given this revelation that additional keys should be ignored anyway, I wonder if there's any point to any special consideration for X- headers? Sorry @lmparppei I see you've already spent time on this.

The X- prefix now hides the those fields, but the non-recognized fields are still printed. I'll also start following the spec, so that those fields won't be printed out, but I'll have a grace period to add some aliases for fields (like Notes and Comments, Contacts/Contact etc.)

Cool! Okay given X- headers are ignored either way I should close this.

lmparppei / Beat

RFC: Fountain X-Header #152

Proposal

What do I need to do?

Parsing

On-the-fly parsing

Data types

Usability and portability

EOF metadata

How does the user interact with X- headers?