Double-entry accounting system for nodejs + mongoose
npm i mediciTo use Medici you will need a working knowledge of JavaScript, Node.js, and Mongoose.
Medici divides itself into "books", each of which store journal entries and their child transactions. The cardinal rule of double-entry accounting is that "for every debit entry, there must be a corresponding credit entry" which means "everything must balance out to zero", and that rule is applied to every journal entry written to the book. If the transactions for a journal entry do not balance out to zero, the system will throw a new error with the message INVALID JOURNAL.
Books simply represent the physical book in which you would record your transactions - on a technical level, the "book" attribute simply is added as a key-value pair to both the Medici_Transactions and Medici_Journals collection to allow you to have multiple books if you want to.
Each transaction in Medici is for one account. Additionally, sub accounts can be created, and are separated by a colon. Transactions to the Assets:Cash account will appear in a query for transactions in the Assets account, but will not appear in a query for transactions in the Assets:Property account. This allows you to query, for example, all expenses, or just "office overhead" expenses (Expenses:Office Overhead).
In theory, the account names are entirely arbitrary, but you will likely want to use traditional accounting sections and subsections like assets, expenses, income, accounts receivable, accounts payable, etc. But, in the end, how you structure the accounts is entirely up to you.
You can set the floating point precision as follows:
const myBook = new Book("MyBook", { precision: 7 });Writing a journal entry is very simple. First you need a book object:
const { Book } = require("medici");
// The first argument is the book name, which is used to determine which book the transactions and journals are queried from.
const myBook = new Book("MyBook");Now write an entry:
// You can specify a Date object as the second argument in the book.entry() method if you want the transaction to be for a different date than today
const journal = await myBook
.entry("Received payment")
.debit("Assets:Cash", 1000)
.credit("Income", 1000, { client: "Joe Blow" })
.commit();You can continue to chain debits and credits to the journal object until you are finished. The entry.debit() and entry.credit() methods both have the same arguments: (account, amount, meta).
You can use the "meta" field which you can use to store any additional information about the transaction that your application needs. In the example above, the client attribute is added to the transaction in the Income account, so you can later use it in a balance or transaction query to limit transactions to those for Joe Blow.
To query account balance, just use the book.balance() method:
const { balance } = await myBook.balance({
account: "Assets:Accounts Receivable",
client: "Joe Blow",
});
console.log("Joe Blow owes me", balance);Note that the meta query parameters are on the same level as the default query parameters (account, _journal, start_date, end_date). Medici parses the query and automatically turns any values that do not match top-level schema properties into meta parameters.
To retrieve transactions, use the book.ledger() method (here I'm using moment.js for dates):
const startDate = moment().subtract("months", 1).toDate(); // One month ago
const endDate = new Date(); // today
const { results, total } = await myBook.ledger({
account: "Income",
start_date: startDate,
end_date: endDate,
});Sometimes you will make an entry that turns out to be inaccurate or that otherwise needs to be voided. Keeping with traditional double-entry accounting, instead of simply deleting that journal entry, Medici instead will mark the entry as "voided", and then add an equal, opposite journal entry to offset the transactions in the original. This gives you a clear picture of all actions taken with your book.
To void a journal entry, you can either call the void(void_reason) method on a Medici_Journal document, or use the book.void(journal_id, void_reason) method if you know the journal document's ID.
await myBook.void("5eadfd84d7d587fb794eaacb", "I made a mistake");If you do not specify a void reason, the system will set the memo of the new journal to the original journal's memo prepended with "[VOID]".
By default, voided journals will have the datetime set to the current date and time (at the time of voiding). Optionally, you can set use_original_date to true to use the original journal's datetime instead (book.void(journal_id, void_reason, {}, true)).
Known Issue: Note that, when using the original date, the cached balances will be out of sync until they are recalculated (within 48 hours at most). Check this discussion for more details.
Sometimes you need to guarantee that an account balance never goes negative. You can employ MongoDB ACID transactions for that. As of 2022 the recommended way is to use special Medici writelock mechanism. See comments in the code example below.
import { Book, mongoTransaction } from "medici";
const mainLedger = new Book("mainLedger");
async function withdraw(walletId: string, amount: number) {
return mongoTransaction(async (session) => {
await mainLedger
.entry("Withdraw by User")
.credit("Assets", amount)
.debit(`Accounts:${walletId}`, amount)
.commit({ session });
// .balance() can be a resource-expensive operation. So we do it after we
// created the journal.
const balanceAfter = await mainLedger.balance(
{
account: `Accounts:${walletId}`,
},
{ session }
);
// Avoid spending more than the wallet has.
// Reject the ACID transaction by throwing this exception.
if (balanceAfter.balance < 0) {
throw new Error("Not enough balance in wallet.");
}
// ISBN: 978-1-4842-6879-7. MongoDB Performance Tuning (2021), p. 217
// Reduce the Chance of Transient Transaction Errors by moving the
// contentious statement to the end of the transaction.
// We writelock only the account of the User/Wallet. If we writelock a very
// often used account, like the fictitious Assets account in this example,
// we would slow down the database extremely as the writelocks would make
// it impossible to concurrently write in the database.
// We only check the balance of the User/Wallet, so only this Account has to
// be writelocked.
await mainLedger.writelockAccounts([`Accounts:${walletId}`], { session });
});
}Journals are schemed in Mongoose as follows:
JournalSchema = {
datetime: Date,
memo: {
type: String,
default: "",
},
_transactions: [
{
type: Schema.Types.ObjectId,
ref: "Medici_Transaction",
},
],
book: String,
voided: {
type: Boolean,
default: false,
},
void_reason: String,
};Transactions are schemed as follows:
TransactionSchema = {
credit: Number,
debit: Number,
meta: Schema.Types.Mixed,
datetime: Date,
account_path: [String],
accounts: String,
book: String,
memo: String,
_journal: {
type: Schema.Types.ObjectId,
ref: "Medici_Journal",
},
timestamp: Date,
voided: {
type: Boolean,
default: false,
},
void_reason: String,
// The journal that this is voiding, if any
_original_journal: Schema.Types.ObjectId,
};Note that the book, datetime, memo, voided, and void_reason attributes are duplicates of their counterparts on the Journal document. These attributes will pretty much be needed on every transaction search, so they are added to the Transaction document to avoid having to populate the associated Journal every time.
If you need to add additional fields to the schema that the meta won't satisfy, you can define your own schema for Medici_Transaction and utilise the setJournalSchema and setTransactionSchema to use those schemas. When you specify meta values when querying or writing transactions, the system will check the Transaction schema to see if those values correspond to actual top-level fields, and if so will set those instead of the corresponding meta field.
For example, if you want transactions to have a related "person" document, you can define the transaction schema like so and use setTransactionSchema to register it:
MyTransactionSchema = {
_person: {
type: Schema.Types.ObjectId,
ref: "Person",
},
credit: Number,
debit: Number,
meta: Schema.Types.Mixed,
datetime: Date,
account_path: [String],
accounts: String,
book: String,
memo: String,
_journal: {
type: Schema.Types.ObjectId,
ref: "Medici_Journal",
},
timestamp: Date,
voided: {
type: Boolean,
default: false,
},
void_reason: String,
};
// add an index to the Schema
MyTransactionSchema.index({ void: 1, void_reason: 1 });
// assign the Schema to the Model
setTransactionSchema(MyTransactionSchema, undefined, { defaultIndexes: true });
// Enforce the index 'void_1_void_reason_1'
await syncIndexes({ background: false });In medici v5 we introduced the so-called "fast balance" feature. Here is the discussion. TL;DR: it caches .balance() call result once a day (customisable) to medici_balances collection.
If a database has millions of records then calculating the balance on half of them would take like 5 seconds. When this result is cached it takes few milliseconds to calculate the balance after that.
There are two hard problems in programming: cache invalidation and naming things. (C) Phil Karlton
Be default, when you call book.blanace(...) for the first time medici will cache its result to medici_balances (aka balance snapshot). By default, every doc there will be auto-removed as they have TTL of 48 hours. Meaning this cache will definitely expire in 2 days. Although, medici will try doing a second balance snapshot every 24 hours (default value). Thus, at any point of time there will be present from zero to two snapshots per balance query.
When you would call the book.balance(...) with the same exact arguments the medici will:
In a rare case you wanted to remove some ledger entries from medici_transactions you would also need to remove all the medici_balances docs. Otherwise, the .balance() would be returning inaccurate data for up to 24 hours.
IMPORTANT!
To make this feature consistent we had to switch from client-generated IDs to MongoDB server generated IDs. See forceServerObjectId.
When creating a book you need to pass the balanceSnapshotSec: 0 option.
const myBook = new Book("MyBook", { balanceSnapshotSec: 0 })Medici adds a few default indexes on the medici_transactions collection:
"_journal": 1 "book": 1,
"accounts": 1,
"datetime": -1, "book": 1,
"account_path.0": 1,
"account_path.1": 1,
"account_path.2": 1,
"datetime": -1,However, if you are doing lots of queries using the meta data you probably would want to add the following index(es):
"book": 1,
"accounts": 1,
"meta.myClientId": 1,
"datetime": -1,and/or
"book": 1,
"meta.myClientId": 1,
"account_path.0": 1,
"account_path.1": 1,
"account_path.2": 1,
"datetime": -1,Here is how to add an index manually via MongoDB CLI or other tool:
db.getSiblingDB("my_db_name").getCollection("medici_transactions").createIndex({
"book": 1,
"accounts": 1,
"meta.myClientId": 1,
"datetime": -1,
}, { background: true })For more information, see Performance Best Practices: Indexing
Unluckily, all the default indexes were suboptimal. The book property always had the lowest cardinality. However, we always query by the book first and then by some other properties. Thus all the default indexes were near useless.
This release fixes the unfortunate mistake.
book property cardinality was moved to the beginning of all default indexes.medici v7.0:
_id_journalbook,accounts,datetimebook,account_path.0,account_path.1,account_path.2,datetimedatetime is the only one to be used in the default indexes. Additional timestamp doesn't make any sense.book.listAccounts() caching which was added in the previous release (v6.3). The default indexes cover this use case now. Moreover, the index works faster than the cache.book.listAccounts() method is now cached same way the book.balance() is cached.mongoose v8 support.mongoose v7 support."MyBook;Account;clientId.$in.0:12345,clientId.$in.1:67890,currency:USD". After: "\u001b\u0004NÞj\u0013rÅ\u001b¼,F_#\u001cÔk Nv". Allows each key to be exactly 40 bytes (20 chars) regardless the actual balance query text length.
rawKey of medici_balances for DX and troubleshooting purposes.The balance snapshots were never recalculated from the beginning of the ledger. They were always based on the most recent snapshot. It gave us speed. Although, if one of the snapshots gets corrupt or an early ledger entry gets manually edited/deleted then we would always get wrong number from the .balance() method. Thus, we have to calculate snapshots from the beginning of the ledger at least once in a while.
BUT! If you have millions of documents in medici_transactions collection a full balance recalculation might take up to 10 seconds. So, we can't afford aggregation of the entire database during the .blance() invocation. Solution: let's aggregate it in the background. Thus, v5.1 was born.
New feature:
balanceSnapshotSec option, we added expireBalanceSnapshotSec.
balanceSnapshotSec tells medici how often you want those snapshots to be made in the background (right after the .balance() call). Default value - 24 hours.expireBalanceSnapshotSec tells medici when to evict those snapshots from the database (TTL). It is recommended to set expireBalanceSnapshotSec higher than balanceSnapshotSec. Default value - twice the balanceSnapshotSec.High level overview.
IOptions type.Major breaking changes:
.balance() method does not support pagination anymore.book -> Book.Step by step migration from v4 to v5.
approved property, you'd need to create similar indexes but without the property.approved property in it.All changes of the release.
mongoTransaction-method, which is a convenience shortcut for mongoose.connection.transaction.initModels, which initializes the underlying transactionModel and journalModel. Use this after you connected to the MongoDB-Server if you want to use transactions. Or else you could get Unable to read from a snapshot due to pending collection catalog changes; please retry the operation. error when acquiring a session because the actual database-collection is still being created by the underlying mongoose-instance.syncIndexes. Warning! This function will erase any custom (non-builtin) indexes you might have added.setJournalSchema and setTransactionSchema to use custom Schemas. It will ensure, that all relevant middlewares and methods are also added when using custom Schemas. Use syncIndexes-method from medici after setTransactionSchema to enforce the defined indexes on the models.maxAccountPath. You can set the maximum amount of account paths via the second parameter of Book. This can improve the performance of .balance() and .ledger() calls as it will then use the accounts attribute of the transactions as a filter.timestamp+datetime index on the transactionModel to improve the performance of paginated ledger queries.lockModel to make it possible to call .balance() and get a reliable result while using a mongo-session. Call .writelockAccounts() with first parameter being an Array of Accounts, which you want to lock. E.g. book.writelockAccounts(["Assets:User:User1"], { session }). For best performance call writelockAccounts as the last operation in the transaction. Also .commit() accepts the option writelockAccounts, where you can provide an array of accounts or a RegExp. It is recommended to use the book.writelockAccounts()..ledger() returns lean Transaction-Objects (POJO) for better performance. To retrieve hydrated mongoose models set lean to false in the third parameter of .ledger(). It is recommended to not hydrate the transactions, as it implies that the transactions could be manipulated and the data integrity of Medici could be risked.precision in the Book constructor as an optional second parameter precision. Simulating medici v4 behaviour: new Book("MyBook", { precision: 7 }).name of Book, maxAccountPath and precision.
name has to be not an empty string or a string containing only whitespace characters.precision has to be an integer bigger or equal 0.maxAccountPath has to be an integer bigger or equal 0.__proto__ can not be used as properties of a Transaction or a Journal or their meta-Field. They will get silently filtered out.book.void() the provided journal_id has to belong to the book. If the journal does not exist within the book, medici will throw a JournalNotFoundError. In medici < 5 you could theoretically void a journal of another book.meta, voided, void_reason, _original_journal won't be stored to the database when have no data. In medici v4 they were {}, false, null, null correspondingly."pre save") installed onto medici transactionModel or journalModel then they won't work anymore. Medici v5 is not using the mongoose to do DB operations. Instead, we execute commands via bare mongodb driver..balance() does not support pagination anymore. To get the balance of a page sum up the values of credit and debit of a paginated .ledger()-call.book anymore. Only Book is supported. require("medici").Book.approved and setApproved()) was removed. It's complicating code, bloating the DB, not used by anyone maintainers know. Please, implement approvals outside the ledger. If you still need it to be part of the ledger then you're out of luck and would have to (re)implement it yourself. Sorry about that.npm i medici@1.See this PR for more details
mongoose to v4. This allows medici to be used with wider mongodb versions.moment, q, underscore.grunt, grunt-exec, grunt-contrib-coffee, grunt-sed, grunt-contrib-watch, semver..coffee any more. Using node.js v6 compatible JavaScript only.