Data model

[lidavidm]

Overall

Static data:

• Things learned (hiragana, kanji, phrases, grammar templates)
• Things asked (various question templates or hard-coded multiple choice questions)
• Things that happen (events, probably a superset of questions)

Dynamic data:

• Mastery of things learned
• Collections
• Location
• Friendship?
• Flags
• Maybe store a small buffer of recent events/questions to avoid repeating things too much (e.g. iTunes found that a true random shuffle felt biased to users)

Static Data

Things Learned

Hiragana:

• How to represent diacritics?
  • They seem to only apply to certain hiragana, so we could duplicate them all
  • Or we could introduce them separately, in which case, how do we represent mastery?
    • One question might update the mastery of several concepts
• I find it really trippy that there are small versions of hiragana/katakana. Do we represent these separately as well? (They seem to basically be diacritics too)

Kanji:

• How will we represent multiple pronunciations? Should we consolidate these by the actual kanji, or have a separate entry for each conceptual meaning?

More complicated things:

• For version 2.0

Questions

Template-based:

• Define broad classes of questions (pick the romanization, fill in the blank, choose the word with this meaning) where the answers are chosen at runtime based on some criteria

Fixed:

• Specific questions where all/most of the answers are handwritten

Events

Either asking a specific question, and/or giving the player some resource/updating a flag (see "Frienships")

Dynamic Data

Friendships

Presumably we're not going to model individual students, The Sims style. So we probably want some flags that events can update, and which other events can predicate themselves on.

Actually storing things

React is somewhat agnostic to its data model, but word on the street is that storing data inside of components doesn't scale so well. I think it'd make our lives easier if we have a central data store. Redux+Immutable.js seems pretty popular at the moment, and I've used this combo before elsewhere. (It has some performance caveats if you're trying to update it hundreds of times per second, but I don't think we're doing that.)

Long story short, in Redux you have a single immutable state object, which you do not directly update. Instead, you define a set of actions that can be performed, and provide a function that takes the current state and requested action, and returns a new state object. React has bridges to expose the current state object to our views, and views dispatch actions to the central store to update it.

This fairly nicely separates the view and model, allowing us to do things like logging easily (e.g. just wrap the updater in a function that logs the action). The caveat is that updating the store itself is a pain (immutable.js is a library & JS doesn't let you override most syntax, so it starts looking like using Java collections) and you have to come up with a clearly defined set of actions (this probably won't be so hard for us).

Static data can probably be stored outside Redux; we can have global tables of questions, events, etc. We do need to correlate these somehow (unique IDs for everything?).

One final caveat is that Immutable.js hates deeply nested data structures, which I will explain if we run into it.

@a7c (and mauer too, but I can't tag him yet)

[a7c]

I'm using Redux in my other research project, so I wouldn't mind using Redux/Immutable.js here too. I did find Redux to be a bit annoying because of boilerplate but since managing mutable state is a pain, the tradeoff is probably worth it.

I'm still thinking about the other issues so I'll get back to you on those later.

[lidavidm]

How about

type Store = {
    learned: Map<Id, Learned>, // Not fond of requiring a unique ID, but we need some way to reference things
    collections: List<Collection>,
    resources: Map<Resource, any>,
    location: Location, // Probably also an ID, referencing a location in some database
}

type Learned = {
    Learnable, // Agnostic to whatever this actually is,
    lastReviewed: DateTime,
    score: double, // And some sort of level/EXP to next level?
}

type Collection = List<Id> // Do we need any specific metadata for collections? (No need to store whether they know the word or not - this is part of the store impllicitly)

Actions:

• Learn new item (add new Learned to learned)
• Review existing item (update score)
• Change location

We'd also have helper functions to surface reviewable items and potentially relevant new items.

[lidavidm]

Hiragana:

• Actual Unicode character + romanization Hiragana+diacritic: maybe a separate category Tiny hiragana: also a separate category Combos: also a separate category Small tsu: ??? Double all these for katakana

Kanji: separate from vocab - just learning the character

• Character
• Components? (kind of sort of radicals - based on modern kanji learning techniques; informally standardized. Copyright?)
• Readings: on (Chinese pronunciation), kun (original Japanese reading), potential other readings
• Meanings: may have multiple (we may want to break out subscores, or model things this way and then generate individual learnable items)

Vocab: overlap between readings and vocab

• Would be nice to cross-reference kanji and vocab (vocab containing kanji)
• Forms (different writings)
• Readings: hiragana-only or romaji-only (many-to-many with forms, but unlikely that we'll deal with it)
• Meanings: can be relationship to form/reading, but we might just split into two entries
• Maybe: example sentences from public dataset
• Part-of-speech

Ideally cross-link everything (e.g. denwa: link den and wa to respective on-reading of kanji) Need notes/metadata for each of these (irregular reading, etc) Look @ jisho.org API model

Probably have a high-level representation, convert to individual learnable items

[lidavidm]

Couple things:

• Learnable things should have a type, a subkey, and a parent key. The parent key identifies the overall concept learned, and the subkey lets distinguish what part of that concept is being reviewed (e.g. a particular reading.) (Can this key be something like the romaji? Are these unique?)
• The type is used to determine potential question types. This way, the question knows what data it can pull from the item and what other data it can use to fill in the question. (e.g. a multiple-choice question written for on-readings will know to pull other on-readings from the database to serve as the incorrect choices)
• The parent key is used to make sure we don't drill the same thing too many times in a row (or conversely, to make sure we hit a particular concept harder - I vaguely recall research into what works best, but don't really remember the conclusion)
• A learnable might specify multiple things to update. So we might have an abstract concept 肉 in the database, along with its various readings and meanings. Questions will be asked about the readings and meanings, and answers update both the abstract concept and the particular concept in question. But the collection interface only shows the abstract concept, which only fills once the subconcepts are mastered. (So we need a way to specify the relationship between the parent and children concepts)

[michaelmauer]

To clarify, what do you mean by "are these unique"? There can be multiple readings in romaji for a given character (早々 = sousou or hayabaya). There can also be multiple kanji for one string of romaji (hashi = 橋 or 箸).

[lidavidm]

Well in that case, they aren't unique. I'm just looking for some unique identifier for a particular word-concept so that we can avoid having to assign and manage numeric IDs which would be a pain.

[lidavidm]

anyways so I propose

type learnable = {
    id: string;
    subid: string option;
    type: learnableType;
    (* ...other fields depend on type *)
}

example:

let hiragana = {
    type: "hiragana",
    unicode: "<squiggle>",
    reading: "ko",
    id: "hira-ko",
    subId: null,
}

let kanji = {
    type: "kanji-wo-reading",
    unicode: "电话",
    reading: "<series of japanese squiggles>",
    subId: "reading-wo",
    id: "kanji-denwa",
}

[lidavidm]

and the actual database definition might be something like

let denwa = new Kanji({
    unicode: "电话",
    readings: {
        wo: "...",
        ...
    },
    meanings: ["phone", ...],
});

[michaelmauer]

Overall I think this makes sense to me.

[lidavidm]

thank

[a7c]

@lidavidm

export interface StoreProps {
    readonly learned: immutable.Map<LearnableId, Learned>;
    readonly collections: immutable.List<Collection>;
    readonly resources: immutable.Map<Resource, number>;
    readonly location: Location;
}

I'm currently adding a collection of hiragana to the game (via data file) and I'm wondering where to store it, since learned is only for learned vocab and collections is just a list of lists of ids.

Should I make a new field for the store that just contains all the static data?

[lidavidm]

The store shouldn't contain any static data. You can define it separately, and just correlate it using the ID. For instance, you could have a model that just contains tables of static data, or you could make the reducer a partially applied function where the first parameter is the static data, and the data is loaded from someplace at runtime. I do something similar for the latter at https://github.com/lidavidm/reduct-redux/blob/1b245406dd8c78062a1cefd4ebe3a7cc0f64a65b/src/reducer/reducer.js#L35 (calling reduct(...) returns a function which is the actual reducer)