The Core API Overview

[willnationsdev]

Come chat about the project on the Godot Extended Libraries Discord!

Sources

The ideas that follow stem from a desire to emulate and synchronize features from the following applications into a single, cohesive Godot plugin that may also involve git submodules for the integration of project-specific views, editors, and APIs:

• Articy Draft
  • Nestable story structures
  • Story database
  • User-Defined editors for story content
  • Map editors (able to mark positions/regions, add metadata that points to database content.
  • System automatically tracks cross-references between story content and database content.
  • Micro-Story Editor: Nodes-and-wires editor communicating dialogue sequences with branching paths and hooks for triggering game logic, etc.
    • Note: supports both low-level nodes/wires at the line of dialogue LOD while also scaling up to bundles of dialogue sequences represented by a single node, etc.
  • Macro-Story Editor: Nodes-and-wires editor communicating when scenes are triggered and their relationships to quests, concrete game tasks, and updates to the database when completed/failed/etc.
  • Able to render statistics about database content.
  • Highly configurable toolkit with a robust plugin system, enabling users to manipulate the tools as they see fit.
  • Has tools to export dialogue lines for any given character, facilitating the business requirements of voice actors by meeting industry standards.
• Twine / YarnSpinner
  • User-friendly scripting languages to quickly and easily develop narrative content.
  • An intuitive nodes-and-wires interface (on the Passage level). Users write narrative content in a Passage using a scripting language, but the order and linking of Passages are procedurally generated based on each Passages' content including internal hyperlinks to other Passages.
• Manuskript
  • Dedicated writing tool targeted at writers. Jot down ideas into a central database.
  • Written passages can refer to content in the database, tracking cross-references.
• Ink
  • Extremely powerful and flexible scripting language for data-generation and execution of branching storylines.
  • Nestable story structures
  • Provides hooks to integrate with external game systems in the Ink Runtime engine.
• Fountain
  • A simple, Markdown-like scripting language for that emulates a screenwriting workflow
  • Can render to screenwriting scripts for easier reading by actors / voice actors.
• HackNPlan
  • Can build a Game Design Document containing the core design concepts of the project.
  • Provides Kanban boards, user stories, and other Agile tools for tracking game development tasks.
  • Can render statistics and other metrics on the current project's state related to the necessary tasks.
  • Can render tasks in a variety of views and group them according to subsystem, assigned users, assigned departments, etc.
  • Users can contribute to a centralized database of story concepts and tie them directly to gameplay concepts and development tasks in the Kanban boards due to the shared information.

API Suggestions

Okay, so that was a lot of stuff. What parts of this am I actually suggesting we build into a Godot project? We'll get into it below.

The names of things aren't specific. They're just so that I can refer to them during the course of the proposal.

These are merely idealized concepts / speculated implementations. It's not necessarily true that these are required, but each piece works with the other pieces, so the more you strip away from this API, the less the pieces are able to work together, so they consequently give less benefit. The API is also constructed as it is for maximum performance benefits and re-usability of concepts throughout Godot Engine's codebase.

I have specifically outlined each concept under a single numbered list so that we can reference individual points using their numeric identifiers.

1. A "Core" Facade Singleton must exist which tracks all story information related to a Godot project.
   1. Virtually all of this information must be serializable and easily written to / read from memory by Godot Engine.
   2. Core must be accessible both at editor-time and run-time.
   3. Core exposes an external interface that adopts the Cypher Query Language API. This implies that Core must expose a model of a graph-based database.
   4. Why a graph-based database?
      1. Writers, designers, and programmers will need to be able to quickly query items that match highly specific, interconnected criteria.
      2. The details of these criteria, in Godot, will be built upon a duck-typed scripting API which does not conform well to the strict tabular layouts of any SQL-based structure.
      3. Cypher is powerful, expressive and user-friendly, all on top of being uniquely well-suited to managing story information.
2. Users can define data structures which embody their story content: StoryItems, extending Resource (Cypher:nodes). StoryItems can have properties (Cypher:map, Godot:Dictionary).
3. Users can tag their StoryItems with 0 or more Labels (Cypher:Label, Godot: a user-defined tags system for resources rather than Nodes' groups).
4. Users can define Relationships between StoryItems (connections) which must each have 1 Label. Relationships can have properties (Cypher:map, Godot:Dictionary).
5. Users can view any StoryItem and see...
   1. a list of its properties.
   2. a list of its Labels.
   3. a list of its Relationships to other specific StoryItems.
6. A collection of nodes and connections in this context is called a StoryGraph.
7. Users can use a centralized API, accessible to GDScript, (extensible via plugins for an aribtrary number of supplemental GUI interfaces) to...
   1. build a CypherQuery, extending Resource, which are queries for StoryItems that match certain criteria and have certain connections to each other.
      1. Example: Up to this point in the Timeline, what are all magical items that, at one point or another, were located in a place visited by the protagonist named "John"?
      2. Query: MATCH (item:MAGICAL:ITEM)-[:LOCATED_IN]->(:PLACE)<-[:VISITED]-(:PROTAGONIST {first_name:"John"}) RETURN item;
      3. Users should be able to build StoryItem queries, through a runtime API, via an Inspector, and via nodes-and-wires (for non-programmers).
      4. One can save these queries and their search results separately, allowing to either refresh with a new search result at any point or to cache a particular query result under an assumed name, respectively.
      5. Given that building a Cypher interpreter and runtime engine that is integrated into Godot directly will take quite a bit of time, I suggest starting with a GDScript prototype implementation base off the Builder design pattern.
   2. execute queries against the database.
      1. 1 or more of these alterations is called an Event, extending Resource. It refers to a discrete collection of changes in the story's state (a Cypher transaction). Modifications to the graph can only happen through Events. If a user executes a query against the graph directly that results in side-effects, then it generates an Event wrapping the query.
      2. Each Event gets a unique hash identifier.
      3. A sequence of Events is wrapped into a Timeline.
      4. When submitted, Events buffer themselves to a queue inside the Core. The StoryGraph is protected by a Mutex. The queue spawns a configurable maximum number of Threads for handling the load of the queue. With the Core managing a queue of requests issued to it from arbitrary external clients, the core is established with a Server pattern, effectively making the core a StoryServer (akin to the other major low-level Servers in the engine).
      5. Flesh out the rest of the StoryGraph, Timeline, and Event API so that it mimics, precisely, the same API as git. StoryGraphs are Repositories. Timelines are Branches. Events are Commits. StoryItems/Relationships are Files. We should be able attach commit messages, merge and rebase events, diff Timelines, reset to alternate Timeline points, branch off into alternate Timelines, etc.
         1. Might consider integrating libgit2 (would need to rework things so it has no Qt dependencies) as a dependency just so that we can rely on using git directly as a "database". Don't know how performant that would be though. Would also be a good opportunity to integrate Git directly into Godot via module.
         2. If we created a way of visualizing branching TimeLines, their associated Events, and the changes associated with each Event, that same code would provide a common API for any arbitrary visual node-based tree or graph of data, including a GitKraken-like visualization of Git content within a Godot Engine project, skill trees (dedicated SkillTree Control node?), charts (dedicated Charts plugin for data visualizations?), and probably plenty more that I haven't thought of yet.
8. There are a few natural consequences that comes from StoryItem, Timeline, and StoryGraph all extending Resource:
   1. The Core can arbitrarily re-organize how each resource is distributed in the FileSystem.
      1. Are they embedded in a single Resource file so that all of the data is together? This is great for read-only situations where you just pull in all of the data at once.
      2. Do we believe it is likely that a subset of StoryItems A from Timeline T will be edited frequently in the near future? Then let's extract T:A from the StoryGraph file, set it up as an external dependency rather than a set of embedded Resources, and then write them out to their own Resource files. Now, any edits made to them can be read/written quickly, without having to save and load the entire StoryGraph file with each change. Etc.
   2. Because they are stored as Resources, they are still objects.
      1. This means they will read/write smoothly, even as binary files, rather than us having to write a custom API to re-map data types.
      2. The will all have free access to class-wide constants and static functions without impacting storage or performance of the individual records.
9. Core manages a global, editor-time StoryGraph called Universe which tracks a statically-defined universe of content.
   1. When a Timeline executes against a StoryGraph (i.e. merged), the Timeline is capable of creating content in the StoryGraph. This means that at the beginning, a game could start with a nearly empty StoryGraph (nothing but basic interactions and ideas), an AI can create a copy of the StoryGraph, apply a theoretical Event to develop an alternate Timeline and view the resulting theoretical StoryGraph. From there, it can subsequently reason and analyze about this theoretical StoryGraph to examine their state before pursuing them as a Plan in a Goal-Oriented Action Planning API. Makes it much easier to create AI-driven narrative experiences.
10. Users must be able to build and configure GUI editors for editing the data associated with their game's content.
    1. This is so that the developer for a project can construct custom GUI editors for their writer/designer team members.
    2. The plugin would provide a default set of editor templates which could then be edited by users, ideally with as little effort and/or as much detail as desired.
       1. Have drag-and-drop features for basic CRUD operations related to data AND editors. A writer or designer should be able to feel comfortable doing all the same work a programmer does when designing data structures in a text file.
11. Core's git-like API can be abstracted into a variety of sub-projects that rely on the same basic infrastructure. It wouldn't be that much of a stretch to incorporate a full-blown Game Design Document (including information beyond just story, but also gameplay systems, etc.) and task-managing systems that have direct access to the Game Design Document information, effectively building an integrated, homegrown HackNPlan super-system. And what is a quest system but a sweet RPG wrapper around a task management system?

This is what I envision for godot-journey. It would serve as a basis for any and all future story management plugins as other plugins would merely create a particular visualization - like Rakugo's Phone Mode, or interface - like Rakugo's GDScript API or future RakugoScript, etc. and add to the available ways of interacting with the Core.

[clayheaton]

I really like graph databases and feel that one could work well as the backend to support this proposed system. However, I would not push for Cypher as the query language. Rather, I would adopt an API similar to the NetworkX Python library. Such an API could conform more easily to GDScript and various gdnative implementations without introducing another language.

[willnationsdev]

@clayheaton I'll look into it (haven't worked much with Python at all). Thanks for the feedback!

[willnationsdev]

cc @xDGameStudios

[Jeremi360]

'godot-journey' also shoud implements global variables in like my RakugoVars:

• You can put them in to text using [](Ren'Py like markup) or {}(bbcode)
• They emit right singnal when value of it or it part is changed
• They are wapers for: int,float, Dictionary, Array, String
• They are Rakugo's:Character, Subquest, Quest

[willnationsdev]

@jebedaia360

'godot-journey' also shoud implements global variables in like my RakugoVars:

I would like to provide an avenue for storing information related to the story, but I would like to explicitly define an optimal API that allows for namespacing the information and allow users to dictate how their tools interact with the API. Perhaps we can make it a global object, but just use slash-delimiters in the property names, the same way that Godot does things for its global ProjectSettings and EditorSettings objects. For example, the Core might see a variable as "thing1/thing2/thing3/property" whereas a scripting language might do something like [thing1.thing2.thing3.property], etc. What the public-facing interface does would be completely implementation-specific. So, Rakugo, for example, might only deal with global variables, or you might choose to modify the parsed string before passing it along to the low-level API, etc.

On the other hand, I would really like to offer people the opportunity to have Intellisense support on the stored data. This would mean using a Dictionary or underlying HashMap<StringName, Variant> to track the properties at each stage so that you can easily ask, "okay, what are the keys at this stage?" Either that, or you keep the properties (which provide much faster runtime access since it's only one .get() operation), but then you maintain a Dictionary cache of each subgroups options, maybe with toggling on that feature only at editor-time or something.

You can put them in to text using [](Ren'Py like markup) or {}(bbcode)

As mentioned above, I anticipate other APIs having their own specific way of passing information to the Core.

They emit right singnal when value of it or it part is changed

I'm toying around with the idea of having a signal that fires every time a property is updated and then also potentially having a procedurally-generated GDScript that reloads at runtime as the story state expands with new variables and it would generate a property_changed_<name>(old_value, new_value) signal. Then, once users define that the variable exists, then they would be able to connect to a signal with a custom name for it. This would be a way for objects to track changes to an individual property as opposed to having every object receive a callback when any property is updated.

func _ready():
    Core.connect("property_changed_dogs_owned", self, "_on_property_changed_dogs_owned")

func _on_property_changed_dogs_owned(old_value: int, new_value: int) -> void:
    pass # update GUI

Then the dialogue runtime, also driven by Core, can actually update the variable and suddenly the GUI updates.

They are wapers for: int,float, Dictionary, Array, String

Is there a reason that you only use these wrappers? You can just as easily do str2var(value) to interpret the text and convert it into a Godot runtime value, correct? I believe Godot already has tools for this kind of thing. Godot 3.1 also introduced a new Expression object which can help evaluate expressions, even in the context of an object instance's member variables, etc.

They are Rakugo's:Character, Subquest, Quest

This is where we might have custom serialized queries for the story database for data pertaining to certain Labels, and you might have classes that wrap these query objects to provide a user-friendly interface for accessing the data.

Personally, I would like to implement a Composite pattern approach to any quest or task system. Whether a task is considered to be a project, a goal, an objective, a task, a quest, a subquest, whatever. There is still a finite set of states it can be in, there are conditions upon which it may be considered complete, whether it is complete or not may, or may not, be reversible, and there are things that might happen when it does complete (or fail, whatever the case may be).

I plan to create a single structure in the database that can handle all of these, and then we'll use unidirectional relationships between the nodes to form the hierarchical tree structure that the nodes have. But I don't want to assign arbitrary organizational structures to the templated nodes for the system. If the application chooses to assign additional Labels to the nodes to form certain super-hierarchies, that would be application-dependent.

[Jeremi360]

When and what we do first for it?

[willnationsdev]

Well, the first step is creating the low-level StoryGraph data structure. The fastest and most efficient way to do that is to incorporate a third-party library that already has a fast and powerful API with a variety of built-in algorithms for handling the graph structure (searching in various ways, etc.). Once we've ironed out all of the details and come up with an implementation that meets our criteria, THEN we can store creating the Events, Timelines, and ultimately start on the Core implementation which manages the story graph.

Also, while Git / VCS isn't a necessary part of this project, it would make sense to work on that if someone else who is highly motivated wants to jump in and try to build a generic VCS interface that can work through making libgit2 C++ compatible with that base API.

Edit: I'm thinking of integrating the LEMON C++ tree/graph library as a GDNativeLibrary and then putting it into a subsection of godot-next/references/data_structures or something, so that individual C++ data structures (like MinHeap and SplayTree to name a few) can be expanded there.

[Jeremi360]

I think that my descriptio of RakugoVar was bad here is RakugoDict to show my idea more clear: https://github.com/jebedaia360/Rakugo/blob/resource-saver/addons/Rakugo/types/rakugo_dict.gd

[ShalokShalom]

You might be interested in this one: https://grakn.ai/

[willnationsdev]

For those wondering about the status of this repository, the main roadblock is that I would prefer for there to be a better standard of addon structure and development workflow with regard to addons that make use GDNative libraries. For that, Godot needs integrated support for compiling GDNative libraries with built-in support for compiling C++ specifically as a first-party language (since it's an industry standard).

This means that not only does Godot need resources and tools for building C++ (similar to an IDE), but the Godot Asset Library and Godot community need support and consensus, respectively, on how to organize addons that use C++ source code, dynamic libraries, and any other manner of GDNative codebases.

I would prefer to resume work on this after such features are done because, otherwise, I will need to maintain an unstable and potentially shifting standard on such things. In addition, the workflow of using this addon would not be very user-friendly without such tools being provided by Godot itself. Since the entire purpose of the tool is to provide a great user experience for each of the tasks, I feel that investing in improving the Godot ecosystem before diving into this further is the best use of my time.

The necessary changes are largely summarized by godotengine/godot-proposals#119 for anyone who is interested in tackling the issue themselves. My main Godot task at the moment is resolving some other issues related to script classes.