Design and implement new object store

[borjasotomayor]

Based on the experiences from the last two years, it has become clear that chiventure needs a unified object store where all the various components can store their data. This will largely involve the following:

• Flesh out the WDL++ format developed in Spring 2020
• Create an in-memory object store that has a nearly 1:1 correspondence with the WDL++ file format.
• Design an API for accessing/modifying the data from other components.

[dwahme]

Below is a summary of the existing WDL++/WDZ documentation and some proposed updates/clarifications to how the directory structure is handled such that it can be cleanly coupled with a revamped libobj object store. The following is subject to updates and review

Existing WDL++/WDZ Format Documentation Summary

The following is a compilation of all documentation for WDL++

WDZ Sandbox

Source code from- https://github.com/uchicago-cs/chiventure/tree/dev/src/wdl/sandbox

Characteristics:

• Appears to be a normal single JSON file- just need to support all common JSON functionality
• Had the major categories like "ROOMS", "GAME", etc defined directly in the file

WDL++ Design Doc

Reference- https://github.com/uchicago-cs/chiventure/wiki/What-WDL---will-look-like

Characteristics:

• Appears to be normal single JSON file- just need to support all common JSON functionality

WDL++/WDZ JSON Directory Structure Documentation

Reference- https://github.com/uchicago-cs/chiventure/wiki/WDL:-Looking-into-options-to-store-JSON-objects

Characteristics:

• Use a zip folder to hold multiple JSON files
• Appears to want to support multiple directories
• Wants to store media such as images and audio inside the zip folder
• Needs to be able to support multiple files corresponding to a major category to support game "libraries" (such as plug and play RPG elements like a base monster set, base spell set, etc that can easily be added in to an existing game)
• Wants to store games zipped instead of unzipped

Proposed WDL++/WDZ Requirements (and Rules for Parsing into Object Store)

Characteristics

• WDZ is a zip folder containing the following:
  • Some form of "main" or "default" file (for single-file games or small components like "ITEMS")
  • JSON files for a given component (./ROOMS.json gets translated to the "ROOMS" component in the object store)
  • Folders for further component groupings (./ITEMS/staff.json gets translated to "ITEMS.staff")
  • Media (maybe in a media folder?)- needs to be able to be referenced from any JSON file
• Maybe allow single-file JSON files as WDL++ so that they don't have to be zipped?

For determing how files in WDZ are translated into the object store, the following convention is proposed:

• <filename>.json- all definitions are translated into "<filename>.<definition>"
• ./<folder>/<filename>.json- all definitions are translated into "<folder>.<filename>.<definition>"
• DEFAULT.json- all definitions are translated directly into the object store as is in the context of the file as "<definition>"
• ./<folder>/DEFAULT.json- all definitions are translated directly into the object store as is in the context of the file as "<folder>.<definition>"
• This folder structure can be continued in nested subfolders, etc

As an example, consider the following combination of files/folders

sample-game.wdz zip file structure

- DEFAULT.json
- ROOMS.json
- ITEMS/
    - DEFAULT.json
    - staff.json
- MEDIA/
    - creak.mp4

default.json

{
    "GAME":
        {
            "name": "sample-game"
        }
}

ROOMS.json

{
    "hall":
        {
            "name": "hall",
            "items": "wood-staff"
        }
}

ITEMS/default.json

{
    "orb":
        {
            "name": "big-orb"
        }
}

ITEMS/staff.json

{
    "name": "wood-staff"
    "sound": "creak.mp4"
}

The above WDZ file would then get parse into libobj 3.0 as the following:

GAME.name = sample-game
ROOMS.hall.name = hall
ROOMS.hall.items = wood-staff
ITEMS.orb.name = big-orb
ITEMS.staff.name = wood-staff
ITEMS.staff.sound = creak.mp4

[dwahme]

Libobj 3.0 API sketch

The following is a sample proposed API for libobj 3.0 that is capable of tightly coupling with a folder/file JSON storage format. Key components are:

• a flexible hierarchical object storage using dot notation ("ITEMS.orb.name" is an example retrieval key)
• Supports basic datatypes- none, bool, char, int, str, lua (for custom objects), lists (hash tables with string keys are built in)
• Could be trivially expanded to store media file formats and other datatypes
• Uses a recursive definition, so subobjects can be retrieved and passed into functions, etc while still using the same API as the overarching data store (aka to get the "name" property of an object, use "ITEMS.orb.name" or get the "ITEMS" object and then call "orb.name" on it)

For more examples of dot notation for object retrieval, this is based on the original libobj style. See https://github.com/uchicago-cs/chiventure/blob/libobj/src/libobj/examples/obj.c

// All datatypes that can be encoded in an obj
typedef enum type
{
    TYPE_ERROR = -1,
    TYPE_NONE  = 0,
    TYPE_BOOL  = 1,
    TYPE_CHAR  = 2,
    TYPE_INT   = 3,
    TYPE_STR   = 4,
    TYPE_LUA   = 5,
    TYPE_LIST  = 6,
} type_t;

/* A simple object model using strings as attribute identifiers */
typedef struct obj
{
    // The identifier of the object- must be unique for the parent object
    char id[MAXLEN_ID + 1];

    // The type of the data
    type_t type;

    // The data associated with the object
    union
    {
        bool b;
        char c;
        int  i;
        char *s;
        char *lua;
    } data;

    // The object's attributes (a hash table)
    struct obj *attr;

    // For if the obj is part of a list
    struct obj *next;
    struct obj *prev;

    // Required uthash identifier for making the hash table
    UT_hash_handle hh;

} obj_t;

// Lists are just objects using utlist
typedef obj_t obj_list_t;

// Object creation/deletion
obj_t *obj_new(char *id);
int obj_init(obj_t *obj, char *id);
int obj_free(obj_t *obj);
int obj_free_all(obj_t *obj);

// Getters
obj_t *obj_get_attr(obj_t *obj, char *id, bool create);
type_t obj_get_type(obj_t *obj, char *id);
bool   obj_get_bool(obj_t *obj, char *id);
char   obj_get_char(obj_t *obj, char *id);
int    obj_get_int (obj_t *obj, char *id);
char  *obj_get_str (obj_t *obj, char *id);
obj_list_t *obj_get_list(obj_t *obj, char *id);

// Setters
int obj_add_attr(obj_t *obj, char *id, obj_t *attr);
int obj_set_bool(obj_t *obj, char *id, bool value);
int obj_set_char(obj_t *obj, char *id, char value);
int obj_set_int (obj_t *obj, char *id, int value);
int obj_set_str (obj_t *obj, char *id, char *value);

It might be worthwhile to see if some fancy macro stuff could be done to clean up the function calls (like OBJ_GET(objstore, "ROOMS.hall.name", str))

[dwahme]

Notes from the first review of the above items:

WDL++/WDZ

• Need a better way to designate references to objects- use JSON objects to represent linking (see below for example)

Better way to designate object references:

"location":{
        "__ref":"object",
        "id":"ROOMS.purple room"
}

Libobj 3.0

• Add a custom asset type for representing assets (TYPE_ASSET)
• Add a custom type for representing references (TYPE_REFERENCE)?
• Add an object type (TYPE_OBJECT)
• Move the hash table of attributes into the object's data union
• Drop TYPE_LUA for now; will revisit how it can be incorporated later
• Add obj_list_t to the data union

[dwahme]

A sample wdz file has been produced incorporating the proposed changes. It can be found at: https://github.com/uchicago-cs/chiventure/tree/wdz_revamp/games/test_game

This is a translation of an older test game located at: https://github.com/uchicago-cs/chiventure/blob/dev/games/test_game.wdl