(most recent) Basis as valuation if no other data are present

[jblachly]

For periodic stock purchases, the most recent purchase price serves as a relatively current proxy of valuation. For many instruments, including mutual funds etc. it can be inconvenient to set up automatic ticker updates and at a minimum will require the end-user to write custom software.

Instead, I propose that if no more recent ticker data are available, the most recent basis price be exposed as the valuation (-V).

This was actually a "gotcha" when I first started with hledger (i.e., even when using the -V flag I still saw security symbols until entering the P... lines) when I first started with hledger because (a) I believe ledger operates this way and (b) this is how I would have expected the software to operate ("principle of least surprise")

[simonmichael]

@jblachly, thanks for the real-world feedback. It's somewhat persuasive. Here are some objections:

Your least surprise as an experienced user differs from mine when I was learning PTA I think. It took me a long time to understand that Ledger worked this way. That was partly a documentation failing, but nobody reads docs so I think it's still an issue.

Using hledger terminology here for convenience: I feel that having transaction prices also imply market prices adds unnecessary complexity to an already complicated system, making it harder to learn, harder to be certain it's calculating the way you intend, and harder to troubleshoot when things go wrong. Providing a simpler conceptual model, at the modest cost of having to write a P declaration or two, seemed better.

Also I feel that it's an approximation that can potentially be misleading. Transacted price and market price are different things, even if usually similar, and it seems wrong that the tool could show you values based on a "market price" X based on a possibly very non-market-rate transaction.

I do agree that fetching and declaring market prices is a troublesome hurdle; I still don't have a satisfactory workflow for it myself. And I agree that just maybe, Ledger-style behaviour might be more convenient for a user that already knows how it works: without any P directives, they can still get some reasonable numbers from -V.

Perhaps there's something else we could do, that brings the benefits/avoids the pitfalls of both ?

There's one absolute prerequisite for this if we decide we want it: to write down a full specification, including all the corner cases. [PS: This looks quite hard to me.]

[adept]

Just a quick thought : As a potentially simpler thing, some sort of warning could be contemplated, with the semantic "price of this transaction is too far away from P-price", with "far away" configurable...

[simonmichael]

I'd like to understand better the scenario(s) where this behaviour ("market prices inferred from transactions", because I can't think of a shorter precise name just now) might be preferred. I think they'd be, eg:

A. You're a new PTA user. You buy 1 TSLA at $500, and maybe later another 1 TSLA at $800. Now you are testing the investment reporting options, which you've seen in --help, or by skimming the manual, but you don't know about P directives yet. You can see the dollar cost, $1300 with bal -B. You try to see the current dollar value with -V, expecting to see $1600, but the TSLA is not converted to dollars at all. You must resort to studying the manual, or asking for help, or filing a bug.

If -V inferred market prices from transactions, you'd see a probably fairly accurate dollar value right away, and have a better starting experience. When you want to see the change in value due to changing market prices, you would be more motivated to learn about P directives.

B. You're a Ledger user, and you haven't been using P directives; you've been relying on market prices inferred from transactions. Now you're trying out hledger on your data, and -V seems to do nothing, unlike with Ledger.

If -V inferred market prices from transactions (note: in exactly the same way as Ledger), it would be one less hurdle for migrating to or occasionally using hledger.

Where do complications arise ?

1. do market prices inferred from transactions have equal standing to P-declared ones ? So all mixed-commodity transactions would generate a virtual P directive, as good as an explicit P directive. So a TSLA purchase today would set today's market price for -V, overruling an explicit P directive dated yesterday ?

2. if the mixed-commodity transaction and the P directive are on the same day, which takes precedence ? Do we need to pay attention to their positions in the parse stream ?

3. or should market prices (for a given commodity pair) be inferred from transactions only as a fallback, if there are no P directives (for that pair) at all ? So a P directive way back at the start of the year would overrule the price in today's transaction ?

4. will inferred transaction prices (from postings in two commodities with no use of @ notation) be used to infer market prices ?

5. valuation uses the market prices declared by P, but also if necessary infers their inverse ("reverse prices") or synthetic prices by chaining these ("indirect prices"). Will we do the same inverting and chaining of market prices inferred from transaction prices ? Will we combine all of these freely when building chains ?

6. Given the above, will we need to provide some debug output or other way to explain which market prices are being inferred, and why ? Will this need to mention the positions of the transactions involved ?

[devbanana]

I believe that the valuation price should be inferred from the transaction price. I think that this can be assumed in almost any situation, where if this behavior is not desire, then the user can easily add their own P statement.

Here are my personal feelings on the questions you posted:

do market prices inferred from transactions have equal standing to P-declared ones ? So all mixed-commodity transactions would generate a virtual P directive, as good as an explicit P directive. So a TSLA purchase today would set today's market price for -V, overruling an explicit P directive dated yesterday ?

Yes, I think if there is no P directive for today, then the virtual one should be used. It's more up to date and it's easy to override if the user wants different behavior.

if the mixed-commodity transaction and the P directive are on the same day, which takes precedence ? Do we need to pay attention to their positions in the parse stream ?

I think if both are available, then the latest P directive takes precedence. If I'm not wrong this is already kind of how it is: if I have several P directives on the same day, the latest one is the one that's used. Shouldn't be hard to mix in transaction prices, too.

Note that I do think the transaction price should have a slightly weaker precedence than an actual P directive, where if there are both in the same day then the P directive should be used. My fear is in the case that a prices file is included at the top of a journal file, the transaction will always come latest in the parse stream and so always override any P directives in the same day, and I don't think this is correct.

or should market prices (for a given commodity pair) be inferred from transactions only as a fallback, if there are no P directives (for that pair) at all ? So a P directive way back at the start of the year would overrule the price in today's transaction ?

Absolutely not, this would be super confusing.

will inferred transaction prices (from postings in two commodities with no use of @ notation) be used to infer market prices ?

Sure, why not?

valuation uses the market prices declared by P, but also if necessary infers their inverse ("reverse prices") or synthetic prices by chaining these ("indirect prices"). Will we do the same inverting and chaining of market prices inferred from transaction prices ? Will we combine all of these freely when building chains ?

Sure, why not?

Given the above, will we need to provide some debug output or other way to explain which market prices are being inferred, and why ? Will this need to mention the positions of the transactions involved ?

I personally don't think this is necessary.

[jblachly]

This is great discussion, thanks everyone. @simonmichael thank you for considering opening up discussion.

My personal feelings and answers to the specific questions above are in fact the same as @devbanana .

Regarding a comment made in @simonmichael 's original reply:

Also I feel that it's an approximation that can potentially be misleading. Transacted price and market price are different things, even if usually similar, and it seems wrong that the tool could show you values based on a "market price" X based on a possibly very non-market-rate transaction.

This is already the status quo with -V if you have a P entry, is it not?

The last thing I would add is that users who have periodic transactions manually or automatically downloaded from brokerages (thus giving them de facto price entries embedded in the journal) must be 1000x more common than the case of the very lucky (or unlucky) person purchasing commodities at non-market rates.

[simonmichael]

I think this could be explained and implemented in a manageable way. But are there stock-tracking users who don't use P records ? Aside from the common but short-lived category of folks getting started, isn't it a rather unlikely niche ?

[simonmichael]

I suppose if you purchase stock/cryptocurrency monthly, that would give you enough price information and you would never need P records. Someone who purchases infrequently, but wants to see current value or capital gains, needs them.

[simonmichael]

Some hacking time opened up, and #1241 is my WIP. It's untested and may not work yet. Some draft docs:

-- When the valuation commodity is specified, this looks for an
-- exchange rate (market price) calculated in any of the following
-- ways, in order of preference:
--
-- 1. a declared market price (DMP) - a P directive giving the
--    exchange rate from source commodity to valuation commodity
--
-- 2. a transaction-implied market price (TMP) - a market price
--    equivalent to the transaction price used in the latest
--    transaction from source commodity to valuation commodity
--    (on or before the valuation date)
--
-- 3. a reverse declared market price (RDMP) - calculated by inverting
--    a DMP
--
-- 4. a reverse transaction-implied market price (RTMP) - calculated
--    by inverting a TMP
--
-- 5. an indirect market price (IMP) - calculated by combining the
--    shortest chain of market prices (any of the above types) leading
--    from source commodity to valuation commodity.

Any testing/problem reports welcome.

[simonmichael]

All, thanks for your helpful input. I think this can land soon, in time for the June release, if no major problems appear.

Here’s an open question: this will change value report output compared to hledger 1.17, if a recent transaction overrides an older P record. Is this ok ? or does the new behaviour need to be optional, either on or off by default ?

[devbanana]

Awesome! Impressed with the quick development on this one.

Yes personally that's fine with me and I doubt there's any easy way around it.

[simonmichael]

This has landed in master, and updated docs are visible at https://hledger.org/hledger.html#valuation . Another long-resisted Ledger feature adopted/long-held opinion changed. Thanks!

[jblachly]

Wow, incredibly fast work -- thank you!

[simonmichael]

A bit too fast :( ... problems reported at #1253 and on chat. Notes:

...two ways in which 1.18 can cause problems:

1) 1.18 uses a more recent market price inferred from a transaction, instead of the P-declared market price which 1.17 used. Symptom: numbers look different compared to 1.17, due to different conversion rates.

This was anticipated, we assumed the price difference would typically be small, and the more recent transaction price would often be the more accurate and preferred one to use, and the old behaviour can be reasonably easily restored by adding more recent P directives (once you have discovered what's going on at least...).

2) 1.18 selects different default valuation commodities because of the market prices inferred from transactions. Symptom: -V shows different commodities compared to 1.17, though -X DESIREDCOMMODITY works as before.

This was not anticipated, and isn't easy to work around. We'll need a bugfix release.

We could avoid 2 by saying that default valuation commodity is determined only from declared market prices (P directives), never from transaction-implied market prices. Will this always be what we want, and intuitive ? Maybe not ?

In addition, I'm expecting more bug reports about the new behaviour, since most users don't read release notes, mail list or issue tracker. I think we at least need to add an easy flag to turn it off. And I wonder if we should turn it off by default, allowing at least until the next major release version for shakedown and to give more warning. Also I should have marked this experimental in docs.

Thoughts ?

[simonmichael]

I've implemented that change for 2. Next: if there are no P-declared prices at all, then I think we should look at transaction prices, to keep the promise that "-V works even with no P directives". Here's what I'm thinking:

for each source commodity A, it chooses a valuation
commodity B based on, in this order of preference:

1. the latest P directive (on any date) declaring a price for A.

2. otherwise, the commodity with which A is exchanged by the greatest
   number of transactions (on any date).

Note the date independence; I think we need this so that the choice of valuation commodity, which has a big impact on reports, doesn't easily flip-flop just because of one new transaction or a single transaction/posting reordering.

[simonmichael]

(Choosing a default valuation commodity based on latest P directive of any date is a change in behaviour; until now we would only look at P directives before the valuation date. Probably somebody will notice this and see their -V reports change as a result. Let me know if you think it's better not to change this.)

[adept]

Assuming that it is highly unlikely to change price reference via P, it feels to me that picking P directive from any date is fine.

As for the case when there are no P directives... I could construct alternative proposal: "the commodity coming from asset account with which A is exchanged (and the rest is like you said".

Example that I am thinking about: currency used to pay for, lets say, bitcoins on exchange. Then bitcoins are exchanged for ether and back 100s of times. Ether is sold for currency several times. Presumably both bitcoin and ether ought to be priced in currency on the assumption that asset account holding that currency implies that this is the "base" currency for the user.

[simonmichael]

Good point. It's getting more and more "clever" but that does sound useful.

[simonmichael]

After working on it a bit, this ("2. otherwise, the (asset account) commodity with which A is exchanged by the greatest number of transactions") is feeling like too much engineering to only partially solve a small problem. I wonder what we could tell users instead, that's easier to implement and to use.

• what does Ledger do ? It does successfully value a two-commodity transaction with no other declarations.
• it has a default subdirective of the commodity directive, marking it as the default valuation target; we could do this.
• if there's no default, but there are commodity directives, use the first commodity declared ?
• if there's a https://hledger.org/journal.html#default-commodity declared, use that ? Probably too much overloading.
• if nothing is declared, use the first parsed, alphabetically first, most-posted, or most-posted-to-asset-accounts commodity ?

[adept]

Thinking about this a bit more, I want to do a U-turn :) I think that any heuristic could be confusing to end-users, but complex heuristic will also be hard to explain, and has more dark corners for various issues to breed.

How about keeping heuristic simple (even if it will potentially make silly decisions), but increasing the visibility instead? For example:

• there is hard and fast rule that P defines valuation commodity
• there is heuristic that provides a decision in the absence of P (any heuristict will do, even reverting back to the code as it was on 26 of May)
• whenever heuristic ends up being applied, there would be an explanation (below the report?) along the lines of:

  Price of X is set to be Y based on transaction from YYYY-MM-DD
  Price of Z set to be Y based on transaction from YYYY-MM-DD
  If this is incorrect, use P directive to specify different commodity (see http://hledger.org/proper-ref-here)

If this is too verbose, maybe it could be triggered (or suppressed) by a flag. This way:

• there would be an easy way to debug issues
• there is immediate explanation on how to fix it

[simonmichael]

We are thinking alike. Latest thoughts: I think --debug will be enough for troubleshooting (I plan to increase priority of valuation output and decrease some other output). Why don't we simplify and pick just one valuation commodity, not one per source commodity. For 1.18.1 it can be simply the price commodity of the latest declared market price (P directive), otherwise the price commodity of the latest implied market price (from transactions). I think this will do the right thing for typical usage, and if some users notice a change, it should be easy to work around.

[simonmichael]

Here's an example to meditate on, minimised from amitaibu's example in chat. First, the output with hledger 1.17, 1.18, and my latest dev build (which takes default valuation commodity from applicable P directives only, if they exist):

$ hledger-1.17  -f examples/1253-amitaibu-2.journal bal -V -0
           ILS-10.00  ...
--------------------
           ILS-10.00
$ hledger-1.18  -f examples/1253-amitaibu-2.journal bal -V -0
          ILS-700.00     
              USD100     
       USD:BOI 97.14  ...
--------------------
          ILS-700.00
              USD100
       USD:BOI 97.14
$ stack exec -- hledger  -f examples/1253-amitaibu-2.journal bal -V -0
          ILS-360.00     
       USD:BOI 97.14  ...
--------------------
          ILS-360.00
       USD:BOI 97.14

You can see that 1.18 picks three valuation commodities instead of one, and the dev build does a little better but still picks two. Here's the journal with the transactions that trigger this and notes on what I think is happening (eg, from adding --debug=4 or 5):

; 1253-amitaibu-2.journal

; Set exchange rate from other currencies to ILS.
P 2019-01-01 USD ILS3.50
P 2019-01-01 EUR ILS3.85

; We create a commodity called `USD:[BANK NAME]`, thus making sure hledger will
; show us the funds in each bank, without consolidating everything under a
; single USD value.
P 2019-01-01 USD:BOI ILS3.50
P 2019-01-01 USD:BOJ ILS3.50
P 2019-01-01 ILS:BOJ ILS1

; txn 1
2019/02/08 Transfer from US to Israel
    assets:banks:israel:boi:usd    USD:BOI 200.00
    assets:banks:us:boa                   USD-200
; hledger-1.18 infers a USD:BOI -> USD market price,
; and also USD as default valuation commodity for USD:BOI.

; txn 2
2019/02/08 Convert the money that was transfered from USD to ILS
    assets:banks:israel:boi:usd
    assets:banks:israel:boi:ils    ILS 340.00 @@ USD:BOI100
; hledger-1.18 infers a ILS -> USD:BOI market price,
; and also USD:BOI as default valuation commodity for ILS.

The more I mess around with this, the more I feel it's too soon to release any code that picks default valuation commodity based on transactions. I lean towards having 1.18.1 pick default valuation commodity based on P directives only. Which means, for now, that -V will have no effect until you add at least one P directive per commodity. Better ideas welcome.

[simonmichael]

PS at the moment I'm aiming for a minimal fix to 1.18. I think switching to a single valuation commodity will affect reports and is too big a change for a bugfix.

[simonmichael]

Well, valuation functionality and UX is quite the maze. Stepping back a bit, I think we want it to be

• Convenient:

  • it should do something useful by default in common situations
  • it should minimise the amount of configuration, typing, and thinking required

• Intuitive

  • it should be easy to explain and to learn
  • it should be easy to remember
  • it should be easy to troubleshoot

• Flexible

  • it should be adaptable to different situations and needs

After further discussion on #1253, here are new draft docs, with inferring market prices optional and off by default:

Market prices

To convert a commodity A to its market value in another commodity B, hledger looks for a suitable market price (exchange rate) in these ways, in this order of preference:

1. A declared market price - the latest P directive declaring A's price in B, dated on or before the valuation date. By default, to see market value reports you must have some of these.

2. An inferred market price - when the --infer-market-prices flag is used, hledger will treat transaction prices like additional P directives. So a transaction where A is priced in B, on or before the valuation date, can also set a market price. (since hledger 1.18; experimental)

3. A reverse declared market price - if there is a declared market price from B to A, we invert it.

4. A reverse inferred market price - when the --infer-market-prices flag is used, if there is an inferred market price from B to A, we invert it.

5. An indirect market price - we look for the shortest chain of market prices (any of the above types) leading from A to B, and combine them to get an approximate A to B price.

Amounts for which no applicable market price can be found, are not converted.

You can add the --debug or --debug=2 option to any command to see output that's useful for troubleshooting market prices and valuation.

More about inferred market prices

The --infer-market-prices flag is new in hledger 1.18, and experimental. As mentioned, it will set market prices also based on transaction prices. This feature is disabled by default, so that market valuation is fully controlled by (and requires) P directives.

If you add this flag, hledger can show value reports without needing P directives, like Ledger. But, your reports can be affected, sometimes in confusing ways, by the transactions in your journal. If so, you can add --debug or --debug=2 to troubleshoot.

Both explicit transaction prices (@/@@ notation) and implicit ones (two-commodity transactions with no @) are used. Note in the latter case, the order of postings matters. hledger print -x can be useful to show implicit transaction prices when troubleshooting.

Valuation commodity

When showing value reports, which commodity should hledger convert to ?

You can specify it, with -X COMM (or --value TYPE,COMM), and hledger will convert all amounts to COMM, whenever it can find a suitable market price.

Or you can leave it unspecified, with -V (or --value TYPE), and hledger will convert each original commodity to its default valuation commodity. This is chosen automatically based on available market prices:

• By default, for an amount in commodity A: it picks the price commodity from the latest P directive for A, on or before the valuation date.

• If the --infer-market-prices flag is used and there are no P directives at all (for any commodity, any date): it picks the price commodity in the latest transaction price for A, on or before the valuation date.

In summary,

• -X specifies what to convert everything to. Or,
• If you have P directives, those decide what -V converts each commodity to.
• Otherwise, if you use --infer-market-prices, transaction prices will.
• Amounts for which no valuation commodity can be found, won't be converted.

[simonmichael]

I have merged new code which I hope covers all the main use cases with a minimum of hassle:

• by default everything works the same way as in 1.17
• you can add --infer-values to infer market prices from transactions, and get value reports without the need for P directives
• semantics and docs are clearer
• various corner cases and complications are avoided.

I just pushed to master to keep things moving and keep discussion in one place, but any review or input is welcome. The main commits are e3cae4aa (code & tests) and e143ad26 (doc). Rendered docs: https://hledger.org/hledger.html#valuation . Please do test in your setup and let me know of any issues.

[simonmichael]

Correction! --infer-value, with no s.

[simonmichael]

Released as 1.18.1.