user-defined contextionary extensions

[etiennedi]

Goals

• at vectorization time
  • [ ] A lookup is performed if the word is custom before checking the real contextionary
  • [ ] If present, the cached vector is used, no lookup to the actual contextionary happens
• when a user adds a custom concept
  • [ ] the input is validated
  • [ ] the definition is vectorized
  • [ ] a new entry is created with both input data (for re-calculation) and the actual vector
  • [ ] the cached copy is updated for every replica

Todos

• [x] build extension Looker Upper
  • [x] add Lookup() method with snake case for compounds
  • [x] returns nil if concept not present
  • [x] returns vector, occurrence if present (fake occurrence, e.g. 1000)
  • [x] async watch for changes and update internal cache
• [x] build extension Storer
  • [x] add Put() method
    • [x] validates correct input
    • [x] vectorizes input using supplied vectorizer
    • [x] stores in repo
• [x] build extender repo
  • [x] add PutConcept with vector, occurrence, input, weight
  • [x] add WatchAll
• [x] build API to
  • [x] upsert new concept
  • [x] list all custom concepts
• [x] extend vectorizer to do custom lookup first
• [x] extend CLI client
  • [ ] to add concept
• [ ] add journey test

Limitations

• Only full replacement/new concepts. No ability to extend (yet), covered in #15