Data Loader RFC

[shawncplus]

Problem

The current way data is loaded into the engine is terrible. The Data class is nothing but static methods with hardcoded paths. This makes it impossible for someone to change where/how entities are stored without directly modifying core.

Goal

• There should be an extensible datasource system in which one can specify which datasource an entity is loaded from.
• A DataSource may be used for more than one entity, e.g., the current default setup of everything being in YAML files would be represented by all entities using the YamlDataSource.
• To specify a datasource for an entity it must be configured and registered in the DataSourceRegistry. I'm envisioning something like this in ranvier.json

{
  // ...
  "dataSources": {
    "Yaml": {
      /* 'require' specifies which file or package to require. It will follow
      the same API as the node require() method, which is to say you could
      either have the class locally or it could be from a node module */
      "require": "./lib/path/to/classFile.js",
      /* An arbitrary config passed to the DataSource constructor.
      Each DataSorce might have a different config: file paths,
      database details, etc. */
      "config": {
        "bundlePath": "bundles",
      }
    },

    "Mysql": {
      "require": "ranvier-mysql-datasource",
      "config": {
        /* A config should also be able to pull from env variables
         to keep things like DB connection details out of source control */
        "hostname": "[ENV(RANVIER_MYSQL_HOSTNAME)]",
        /* FEEDBACK: seanohue: could just make ranvier.json a .js file and use process.env */
      }
    },

    "Json": {
      /* If a node module exports more than one data source you may specify
      which export to use with <module>.<object> */
      "require": "cool-datasources.JsonDataSource",
      "config": {
        "root": "data"
      }
    }
  },
  // ...
}

• Configuring the entities to use a data source would also be done in ranvier.json

{
  // ...
  "entitySources": {
    /* The keys here will be a specific list of game entities which the
    BundleManager will use to load data. However, you should also be able
    to add additional entities if you have a bundle with some custom data
    like a vendor's product list or loot tables */
    "items": {
      /* specifies which registered DataSource to use */ 
      "source": "Yaml",
      /* Additional configuration for the datasource specific to this entity */
      "config": {
        "path": "items.yml",
      },
    },
    "npcs": {
      "source": "Mysql",
      "config": {
        "table": "npc",
      },
    },

    "players": {
      "source": "Json",
      "config": {
        "path": "data/players"
      }
    },
    "accounts": {
      "source": "Json",
      "config": {
        "path": "data/accounts"
      }
    }
  }
}

• The DataSourceRegistry will be populated by instances of the classes defined/configured from the dataSources in ranvier.json. It will be the low-level API for data access, generally used only internally by each EntityLoader class.
• Each configured EntityLoader instance will be registered to the EntityLoaderRegistry
• The EntityLoader should have an async fetchAll() method to retrieve all data. As well as singular async fetch(key) method to fetch a single entity
• For higher level data access there will be an EntityLoader class which will be constructed from the entitySources configuration in ranvier.json. It will be the class used to actually fetch entity data, example:

const itemLoader = state.EntityLoaderRegistry.getLoader('items');
/* EntityLoader will have setBundle/setArea methods to optionally specify
which bundle/area the data is coming from */
itemLoader.setBundle('example-areas');
itemLoader.setArea('limbo');
const items = await itemLoader.fetchAll();

/* player data won't be stored in a bundle, for example */
const playerLoader = state.EntityLoaderRegistry.getLoader('players');
const playerData = await playerLoader.fetch(playerName);
const player = new Player(playerData);

• The default ranviermud setup will be the same as it is today: all game entities loaded from YAML. It will come with the ranvier-yaml-datasource already installed/configured. Additionally player/account data will be loaded from JSON using the preinstalled/configured ranvier-json-datasource

[ratacat]

I think that sounds great Shawn! I don't probably have very nuanced view of it, but I really like the idea of being able to easily adapt to different storages systems. Thankyou!!!!

[shawncplus]

Work is visible ongoing on the https://github.com/RanvierMUD/core/tree/data-refactor branch