v2.x.x progress

[davalapar]

polylines

• https://github.com/mapbox/polyline
• https://developers.google.com/maps/documentation/utilities/polylinealgorithm

[breaking]

• disk-only persistence instead of memory & disk persistence
• only ids are stored in memory
• query is tricky, do we stream a large file or read small files?

[minor]

• offset slice should not include limit if not needed
• limit should not slice if not finite

[major] finalizing

• remove compression support since we want speed
• focus on clustering to serve high-throughput reads instead
• remove cbor/msgpack serialization plans since we want speed
• https://en.wikipedia.org/wiki/List_of_file_signatures

[minor] fix errors

• errors have inconsistent capitalization
• errors should contain table name

[breaking] versioning

• attach db version in table records

[breaking] snapshots

• snapshots should flush current data into a single *.db file
• we should provide a script to extract *.db into multiple *-meta.tb and *-current.tb
• maybe rename?
  • *.tb table current
  • *.st table stale
  • *.mt table meta (counter, serialization, compression, encryption)

// first 1 byte : serialization : 0=json, 1=cbor
// next 1 byte : compression : 0=none, 1=gzip, 2=brotli
// next 1 byte : hash: 0=sha256, 1=sha512-256
// next 1 byte : encryption : 0=none, 1=xsalsa20-poly1305
// next 1 byte : id type : 0=random, 1=counter
// next 8 bytes : counter value : UInt64
// next n bytes : schema hash
// next n bytes : table items

[minor] rollback, commit, snapshot

// db.rollback()
current -> data // readFileSync

// db.commit()
data -> temp // writeFileSync
current -> previous // renameFileSync
temp -> current // renameFileSync

// db.snapshot()
current -> snapshot // copyFileSync

[breaking] improved id

• new type id
• accepts non-empty string, and integer-number values
• data saved is now counter, schemaHash, and list
• new table parameter idType accepting random|counter

[minor] item property value peek

• returns an item's property value
• id must exist, field must exist
• primitives are returned directly
• arrays and objects are copied
• alternative is to also overload the .get(id) method

SomeTable.peek(id, field) -> value; // preferred
SomeTable.get(id, field) -> item|value;

[breaking] add object type

• a schema-less object property that can represent a circular or non-circular plain javascript object
• lodash has _.isPlainObject which can be used to determine if object is really a plain object
• Set can be used in crawling circular objects
• flatted can be used as alternative for JSON.stringify and JSON.parse in working with circular objects and preserving their states
  • https://www.npmjs.com/package/flatted
• lodash also has methods with deep property path support such as _.get, _.has, and _.set
• query filters can use these deep property path methods when trying to sort or filter these types, except that they might introduce unsafety-ness in introducing undefined values

[breaking] triggers / hooks

• allows us to subscribe to table events
• let's say an item is deleted, and one of its rows is dependent on another table, we can also create side-effects that automatically reflect against those changes
• table events:
  • beforeAdd, afterAdd
  • beforeUpdate, afterUpdate
  • beforeDelete, afterDelete
  • beforeCommit, afterCommit
  • beforeRollback, afterRollback

[breaking] transactions

• two-phase commit protocol
  • https://en.wikipedia.org/wiki/Two-phase_commit_protocol
• saga pattern
  • https://microservices.io/patterns/data/saga.html
  • https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part/
  • https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part-2/
• transaction create
• transaction commit
• transaction rollback
• the idea is that the tables should use dummy arrays when db enters transaction mode
• dummy array could contain whole copy of items, or ideally, just a stack of changes needed to be reflected against the actual items
• i also think the query command can be easily modified by slicing the actual items then applying these stack of changes before applying filtering & sorting mechanisms

[breaking] process clustering

• for high-throughput reads
• affected table methods
  • new, clear, add, update, delete, increment, decrement
• affected database methods
  • new, save
• message format
  • identifier, method, parameters
• message flow
  • W1 -> M1 -> W2, W3, W4
• source worker
  • validate, reflect changes
• forwarding master
  • receive changes, qualify message, reflect changes, forward changes
• recepient workers
  • receive changes, qualify message, reflect message
• if necessary
  • if forwarding master or recepient workers encounter error in reflecting changes, they should refresh their own goddamn copies

[minor] key-value tables

• key-value tables? (boolean, string, number, arrays, objects)
  • does not use schema
  • set, get, & has commands only
  • we accept basic primitives except null and undefined
  • we accept arrays and objects as long as they are not circular
  • detect circular objects? https://stackoverflow.com/questions/14962018/detecting-and-fixing-circular-references-in-javascript

[minor] append-only tables

• ideal for logs
• truncates file by 1 character
• appends new line gracefully
• fields
  • id (counter id)
  • type
  • text
  • timestamp
• reference
  • https://redis.io/topics/persistence

[?] worker threads

• https://nodejs.org/api/worker_threads.html
• https://blog.logrocket.com/use-cases-for-node-workers/
• https://bytearcher.com/articles/not-everything-goes-to-worker-threads/
• method query.toWorkerThread(workerFn) tosses data to worker thread instead of copying it

[?] browser-build

• separate fs adapter and localstorage adapter

[?] readme updates

• forced implicit default property values by default
• not having to deal with undefined & null
• default property value helpers
• document each method
• we should also document the order of execution for queries: filter, sort, paginate, select

[?] more usage

• number-type property as enum index
• string-field as json-stringified objects

const LogTypes = Object.freeze({
  Info: 0,
  Warning: 1,
  Error: 2,
  Success: 3,
});

Logs.add({
  id: Logs.id(), // string
  type: LogTypes.Info, // number
  message: 'X at Y at Z', // string
  context: JSON.stringify({ param1, param2 }), // string
});

[breaking] internal logging

• db.save can do logging for us gracefully
• we use an append-only file
• where each line is [type<String>, message<String>, context<Object>]
• for linux we can use the tail command to fetch these logs
• we can log before saving, and after saving, and after error on saving

[breaking] database initialization

• move to single-instance of database
• since we're using single directory /tables anyways

const db = Database.init({
  tableConfigs: [],
});

[?] reference projects

• https://postgrest.org/en/v6.0/
• https://hasura.io/
• https://graphql.org/learn/schema/
• https://dev.mysql.com/doc/refman/8.0/en/data-types.html

[davalapar]

fixed (nov 26, 5pm)

• misc usage
  • extracting first result: .results()[0];
  • extracting results count: .results().length;
  • extracting if results exist: .results().length < 0;
• pure synchronous
  • remove asynchronous save feature
  • allow the user to call save whenever he wants instead
  • always do graceful save instead
  • add try-catch for graceful save instead to capture errors
  • save errors must delete / unlink current version of table
  • cluster mode ensures all save calls are asynchronous anyways
• fix highlights section

[davalapar]

• msgpack should be used in case our size gets large
• https://www.npmjs.com/package/big-json
• v8 string length

[davalapar]

• use cbor instead of messagepack

[davalapar]

• serialization: json/cbor
• compression: deflate/gzip/brotli
• schema hashing: sha256/sha512-256