simonmichael / hledger

Robust, fast, intuitive plain text accounting tool with CLI, TUI and web interfaces.
https://hledger.org
GNU General Public License v3.0
3.07k stars 320 forks source link

Better UX around starting and ending files, close command ? #2151

Open simonmichael opened 10 months ago

simonmichael commented 10 months ago

Here are a few thoughts aimed at better user experience around starting new files periodically, eg each year. Discussion welcome.

Some context

We need a short term for "multiple files used to cover different time periods" (as distinct from multiple files split up in some other way, like by account). I'll call them "periodic files" unless we find something better.

We need language to distinguish between

From here I'll try to use these four terms (open, close, start, end) as above, where possible. Though, "open"/"close" are pretty baked in to the close command right now.

Can we simplify the process of starting a new file ?

We want to represent correct asset/liability (and possibly equity) balances at all times:

The current common method, start/end transactions (A)

  1. assume all balances are initially zero
  2. in a special starting balances transaction, post the exact starting balances
  3. and immediately assert those same balances, verifying them (and 1).

If you combine multiple periodic files, all but the earliest starting balances are redundant and excessive. So we also

  1. At the end of each periodic file except the latest, in a special ending balances transaction, post the negative of the ending balances. This cancels out the redundant starting balances transaction which will follow in the next file.

We balance these starting/ending balances with an equity account, call it equity:start. If migrating equity balances along from one file to the next: we must always exclude equity:start, otherwise it would accumulate redundantly and cause an incorrect equity balance.

We also need to

  1. exclude the ending balances transaction when we want to see end of year balances in past years (otherwise they all appear zero)

and may want to

  1. exclude starting/ending transactions from register reports.

An alternate method, balance assignments (B)

  1. In a starting balances transaction, assign (set) the balances to the correct starting balances. This doesn't require initially zero balances; it posts whatever amounts are necessary to reach the correct starting balances.
  2. At the end of each periodic file except the latest, assert the ending balances (without changing them), to provide some error checking. The asserted ending balances and the assigned starting balances in the next file could also be visually compared.

A possible future method, smarter balance assignments (C)

  1. Use a new kind of balance assignment that succeeds only if the existing balance is either exactly zero or exactly the correct balance already. This would provide more error checking than B.

New directives (D)

  1. A new directive, like start-balances, would be more semantically correct (in the real world, there are no transactions when you end a data file and start a new one) and could improve usability.
  2. There could be a corresponding, usable end-balances directive.
  3. These could perhaps detect more errors than the previous methods, eg enforcing "no postings are allowed after a file's end-balances or before a file's start-balances".
  4. (Though, most likely these would mainly be syntactic sugar for transactions, since we understand how those work, and other semantics would require many things be resolved, such as how sequencing works.)
  5. More on this below.

Which method is best ?

My rationale for B or C was that they would be easier to use than A, requiring less busywork when you need to change old balances/accounts/journal entries (to fix mistakes, or improve your bookkeeping, eg)  and propagate the effect of those changes through subsequent files. (This is especially noticeable if you track lots of accounts and gets worse over time as you accumulate more files.)

B/C also have this advantage over A: they let you select non-contiguous periodic files, so eg you could process just 2020 and 2024 and still see correct balances in those years.

But B/C have a problem. Assuming the starting balance assignments are balanced transactions (1.1), the balancing equity:start account accumulates excessively with each new file, because there's no corresponding ending balances transaction to prevent it.

To avoid this, we could use unbalanced postings to assign starting balances (1.2). But this violates the Accounting Equation so we would then no longer be able to achieve A+L+E = 0 in reports.

This could be avoided by having the earliest starting balance transaction be balanced, and the ones in later files unbalanced (1.3). But this does not fit the goal of being able to freely choose one, a subset, or all of the files.

Requirements

To review: We'd like to have

  1. correct balances always
  2. flexibility in choosing one or more files to process
  3. good error checking
  4. preservation of the accounting equation
  5. minimal interference with reports, ideally

Method A (posting starting balances and zeroing out ending balances in each file) achieves 1-4. Is it the only way ? Is there really no other solution ?

What other improvements could be made to close ?

Proposals

simonmichael commented 10 months ago

More on new directives (D)

To meet the above requirements, I think the minimal mechanism/behaviour is something like this:

Ie, have a special kind of event that acts as a transaction when it's first and a balance assertion otherwise.

More details: syntax could be like this (possibly with a S short form in future):

start-balances 2024-01-01
  assets:bank:checking     100
  liabilities:credit card    0

An equity:start balancing account is assumed and need not be written. Or it could be written, last and with no amount, to localise it:

start-balances 2024-01-01
  assets:bank:checking     100
  liabilities:credit card    0
  cothromas:tús

(One could think of combining a directive like this with account declarations somehow, but I'm not proposing that.)

Nothing is needed at file ends, though ordinary balance assertions can be put there if desired. If not, then changes in an old file would change its end balances and would not raise an error. But start-balances assertions in the next file would raise an error, and need to be updated.

Upside ?

Downside ?

Simpler methods (E, F)

Are those good enough, after all ?

E adds busywork: if you want to be able to process files starting from any year, it seems like you'd need to maintain (say) 2020-onward.journal, 2021-onward.journal, 2022-onward.journal, eg with that year's starting balances and including all later files (and all files would need updating each year).

F requires not splitting files, or at least not by time. We shouldn't neglect to reconsider this, as it simplifies a lot. If it causes slow performances on slower machines or as data accumulates over time, is that actually a problem ? Could we get ahead of it with more optimisation and Moore's law ?

It's worth thinking about - we are pretty fast (though memory hungry), and achieving yet more performance/efficiency (because forced to) would be good. But it does sound limiting; in general, if we want to support arbitrary data sizes and data periods, and run quickly and non-wastefully on all diverse machines, it will always be necessary to be able to split by time.

On the other hand, it might not need to be a common operation recommended for all users. It might be something to explain only for "big data" or long-time users.

On the third hand, splitting by year is familiar and intuitive to most people, and has benefits other than performance (limiting default report period, limiting editable entries, limiting the impact of bookkeeping errors).

aimuz commented 10 months ago

I have a need for this, when I forget to keep track of my accounts after a period of time, it's a pain in the arse to try and get back on track, I have to keep track of every single entry in the past.

It would be nice to have a feature similar to https://beancount.github.io/docs/balance_assertions_in_beancount.html.

I will be able to do a good job of recording after today and for the records before today I will be able to reconcile them when I have time.

Something like a snapshot? A snapshot of the current account?

This would also verify that past accounts are correct

It may be a bit messy because my subject is not English

With the existence of this feature, it will be possible to assert the current balance at the beginning of each year, or after starting a new file, and be able to keep score with confidence.

simonmichael commented 10 months ago

@aimuz yes very useful, and we have those of course: https://hledger.org/hledger.html#balance-assertions (and https://hledger.org/hledger.html#balance-assignments)

simonmichael commented 10 months ago

Now in master:

Any testing/review welcome.