Yarn file spec: add file header object for storing file metadata, which encompasses all nodes

blurymind commented 4 years ago

@desplesda @hylyh @daviddq This has been bothering me for a while, but I was wondering if we can add an extra object to yarn files where we can give the file a bit more information about its identity and capabilities:

tagStyle: "html" | "bbcode" (used to automatically detect the tag style of the document)
textColourPalette: Array<hexString> (used to store the colour picker's accumulated coloursin the palette) - it will allow us to keep its palette history in the file
userVariables: Array<string> (used to store all variables that are used in the file, we can use it for autocompletion and better debugging when playtesting)
userCommands - same as userVariables but for commands
other metadata - such as author, language, last edited user etc

Right now a yarn json file looks like this:

[
    {
        "title": "Start",
        "tags": "err erg",
        "body": "A: Hey, I'm a character in a script!\n<<test>>\nB: And I am too! You are talking to me!\nB: What would you prefer to do next?\n\n[[Leave|Leave]]\n[[Learn more|LearnMore]]",
        "position": {
            "x": 283,
            "y": 207
        },
        "colorID": 0
    },
    {
        "title": "Leave",
        "tags": "ad ber",
        "body": "A: Oh, goodbye!\nB: You'll be back soon!",
        "position": {
            "x": 190,
            "y": 539
        },
        "colorID": 0
    },
]

a Yarn text file:

title: Start
tags: 
colorID: 0
position: 108,232
---
<b>Bold</b> text
<u>Underlined</u> text
Text in <i>italics</i> 
Text in <color=#ff0000>red color</color> and <color=#0000ff>blue color</color>
<color=#00ff00><b>Green and bold</b></color> text
<b><i>bold and italics</i></b> 
<b><u>bold and underlined</u></b> 
<i><u>italics and underlined</u></i>
<i><u><color=#e900ff>italics, underlined and pink</color></u></i> 
<img>image</img>
<<command>>
[[This link brings you to a new node:|link]]
===
title: link
tags: 
colorID: 0
position: 526,227
---
Empty Text
===

My proposal is to add a header to the file, maybe something like this:

[
    {
        "title": "YARN_HEADER",
        "tagStyle": "html",
                "textColourPalette": ["#E9967A", "#F08080"],
                "userVariables" : ["talkedToBird", "foundClock", "apples", "etc"],
                "userCommands": ["avatar", "speed",  "setTime"],
                "fileInfo": {
                       "language": "english",
                          ....
                  }
    },
    {
        "title": "Start",
        "tags": "err erg",
        "body": "A: Hey, I'm a character in a script!\n<<test>>\nB: And I am too! You are talking to me!\nB: What would you prefer to do next?\n\n[[Leave|Leave]]\n[[Learn more|LearnMore]]",
        "position": {
            "x": 283,
            "y": 207
        },
        "colorID": 0
    },
    {
        "title": "Leave",
        "tags": "ad ber",
        "body": "A: Oh, goodbye!\nB: You'll be back soon!",
        "position": {
            "x": 190,
            "y": 539
        },
        "colorID": 0
    },
]

a Yarn text file (not sure where to put the metadata yet):

title: YARN_HEADER
tagStyle: html,
textColourPalette: ["#E9967A", "#F08080"]
userVariables : "talkedToBird", "foundClock", "apples", "etc"
userCommands: "avatar", "speed",  "setTime"
language: english
 ...
===
title: Start
tags: 
colorID: 0
position: 108,232
---
<b>Bold</b> text
<u>Underlined</u> text
Text in <i>italics</i> 
Text in <color=#ff0000>red color</color> and <color=#0000ff>blue color</color>
<color=#00ff00><b>Green and bold</b></color> text
<b><i>bold and italics</i></b> 
<b><u>bold and underlined</u></b> 
<i><u>italics and underlined</u></i>
<i><u><color=#e900ff>italics, underlined and pink</color></u></i> 
<img>image</img>
<<command>>
[[This link brings you to a new node:|link]]
===
title: link
tags: 
colorID: 0
position: 526,227
---
Empty Text
===

blurymind commented 4 years ago

of course the header data can be top level and all yarn nodes can be somehow nested under nodes or something, but a change such as this would require changing the parsers- ideally with keeping backwards compatibility

desplesda commented 4 years ago

Given that we removed JSON support from the language in Yarn Spinner v1.0, should this project file be in JSON?

blurymind commented 4 years ago

Bondagejs uses json files and so does gdevelop and ctjs because of that. We need to keep it for those and other html5 engines

On Mon, May 18, 2020, 11:19 PM Jon Manning notifications@github.com wrote:

Given that we removed JSON support from the language in Yarn Spinner v1.0, should this project file be in JSON?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/YarnSpinnerTool/YarnEditor/issues/183#issuecomment-630465791, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRRWVNYJ5UWRF2Z4NXRDWLRSGYAVANCNFSM4NEL5PGA .

daviddq commented 4 years ago

Regarding the JSON file: adding the header as a node with different structure is dangerous. Loaders that are not prepared will try to read it as a normal node and they will explode trying to access members that don't exist. We could use a meta-node, identified by the "title" and put all metadata inside the body. That way loaders that don't know about the metadata will load an additional "messy" unconnected (and unreachable) node, but won't explode loading the file. On the editor we can recognize the meta-node, read and hide it.

Regarding the YARN file: safely modifying this format is a challenge. If this file already supports comments we could put all the meta as comments or put the meta inside the body.

Long term proposal:

1) Start versioning the files:

YARN

version: 2
===
tagStyle: html,
textColourPalette: ["#E9967A", "#F08080"]
userVariables : "talkedToBird", "foundClock", "apples", "etc"
userCommands: "avatar", "speed",  "setTime"
language: english
 ...
===
title: Start
tags: 
colorID: 0
position: 108,232
---
<b>Bold</b> text
<u>Underlined</u> text
...

JSON

{
  version: '2',
  meta: {
    "tagStyle": "html",
    "textColourPalette": ["#E9967A", "#F08080"],
    "userVariables" : ["talkedToBird", "foundClock", "apples", "etc"],
    "userCommands": ["avatar", "speed",  "setTime"],
    ...
  },
  nodes: [
    {
        "title": "Start",
        "tags": "err erg",
        "body": "A: Hey, I'm a character in a script!\n<<test>>\nB: And I am too! You are talking to me!\nB: What would you prefer to do next?\n\n[[Leave|Leave]]\n[[Learn more|LearnMore]]",
        "position": {
            "x": 283,
            "y": 207
        },
        "colorID": 0
    },,
...
}

2) Be able to import/export all versions from the editor. That way, authors can keep using old versions for their old productions and switch to the new version whenever they want. Note that they will be able to load a "version 1" file and save it as "version 2".

blurymind commented 4 years ago

Given that we removed JSON support from the language in Yarn Spinner v1.0, should this project file be in JSON?

Json is the parsing standard for html5/javascript game engines. Why did you decide to drop it :) That could be a problem for swapping out bondagejs for yarnSpinner on gdevelop if yarnspinner can one day run in a js app

desplesda commented 4 years ago

We’re approaching a point where we need to have a conversation about the structure of the Yarn project structure, and how this editor relates to the other Yarn Spinner project structures. @blurymind and @daviddq, we should probably have a chat to make sure that we don’t accidentally misalign the projects soon - what’s the best place to talk, Twitter, Slack, etc?

daviddq commented 4 years ago

Slack, Discord, Hangouts, IRC, ... 🤷

desplesda commented 4 years ago

Probably the best way would be for you both to join the Yarn Spinner slack? (It’s currently 1AM for me so right now‘s not the best time, but it’s a great place to start the conversation from!) https://lab.to/narrativegamedev

daviddq commented 4 years ago

I joined. Any preferences:

blurymind commented 4 years ago

I'm in the UK and can join after work (18:00 uk time)

daviddq commented 4 years ago

This has been bothering me for a while, but I was wondering if we can add an extra object to yarn files

Additionally to the solution proposals I provided before, we can have additional files that hold the meta data (user/workspace/project related metadata)

desplesda commented 4 years ago

A simple text file format that encodes key-value pairs, and doesn’t introduce any package dependencies (either in Yarn Editor, bondage, or Yarn Spinner) would be ideal. Something easy to parse like ini files, maybe.

I’d suggest that this be opened as an issue in the Yarn Spinner repo, as a proposal to enhance the format.

blurymind commented 3 years ago

I added a v2 yarn JSON file type, which looks as follows:

old type (note how the root is an array - huge problem for expand ability):

[
    {
        "title": "Start",
        "tags": "err erg",
        "body": "A: Hey, I'm a character in a script!\n<<test>>\nB: And I am too! You are talking to me!\nB: What would you prefer to do next?\n\n[[Leave|Leave]]\n[[Learn more|LearnMore]]",
        "position": {
            "x": 283,
            "y": 207
        },
        "colorID": 0
    },
    {
        "title": "LearnMore",
        "tags": "rawText",
        "body": "A: HAHAHA\nBlah blah more..",
        "position": {
            "x": 534,
            "y": 534
        },
        "colorID": 0
    }
]

new type (root is an object - suddenly a much more flexible file):

{
    "header": {
        "lastSavedUnix": "2021-06-08T19:13:17.277Z",
        "language": "en-GB",
        "markupLanguage": "bbcode",
        "filetypeVersion": "2"
    },
    "nodes": [
        {
            "title": "Start",
            "tags": "err erg",
            "body": "A: Hey, I'm a character in a script!\n<<test>>\nB: And I am too! You are talking to me!\nB: What would you prefer to do next?\n\n[[Leave|Leave]]\n[[Learn more|LearnMore]]",
            "position": {
                "x": 312,
                "y": 123
            },
            "colorID": 0
        },
        {
            "title": "LearnMore",
            "tags": "rawText",
            "body": "A: HAHAHA\nBlah blah more..",
            "position": {
                "x": 534,
                "y": 534
            },
            "colorID": 0
        }
    ]
}

type is detected by the root being an array (OLD) or an object (NEW) You can also select your preferred version number in settings

I am not sure how to do this for ordinary yarn text file, since that has no nesting like json and xml do - the stucture we save in it will have to be flat, but if we can agree on one - I would be eager to implement it.

blurymind / YarnClassic

Yarn file spec: add file header object for storing file metadata, which encompasses all nodes #183