search

simonmichael / hledger

Robust, fast, intuitive plain text accounting tool with CLI, TUI and web interfaces.
https://hledger.org
GNU General Public License v3.0
3.01k stars 321 forks source link

-V output changed when updated from 1.17.11 to 1.18 #1253

Closed Gioyik closed 4 years ago

Gioyik commented 4 years ago

Hello,

I have been using hledger with two currency COP, and USD. Because I manage stock to get a nice and organized balance sheet, I use hledger bs -V. This is the output I was getting 3 minutes ago before updating to 1.18:

Balance Sheet 2020-06-08, current value

                          ||                      2020-06-08
==========================++=================================
 Assets                   ||
--------------------------++---------------------------------
 Assets                   ||                  500.200,00 COP
   CO                     ||                  500.200,00 COP
     Cash                 ||                  500.200,00 COP
==========================++=================================
 Liabilities              ||
--------------------------++---------------------------------
 Liabilities              ||        100.269,29 COP, 6.19 USD
   CO                     ||        100.269,29 COP, 6.19 USD
     Bank                 ||        100.269,29 COP, 6.19 USD
       XXX                ||        100.269,29 COP, 6.19 USD
         Credit Card      ||        100.269,29 COP, 6.19 USD
--------------------------++---------------------------------

This is the output I get now:

Balance Sheet 2020-06-08, current value

                          ||       2020-06-08
==========================++==================
 Assets                   ||
--------------------------++------------------
 Assets                   ||    200.61 USD, 0
   CO                     ||       200.61 USD
     Cash                 ||       200.61 USD
==========================++==================
 Liabilities              ||
--------------------------++------------------
 Liabilities              ||     75.16 USD, 0
   CO                     ||     75.16 USD, 0
     Bank                 ||     75.16 USD, 0
       XXX                ||     75.16 USD, 0
         Credit Card      ||     75.16 USD, 0
--------------------------++------------------

For some reason, my COP balance is being converted and accumulated in USD, why? I have no idea, it just happened after I updated. I have a file where I put P values for my share prices, I have never set COP values to make conversions, so I don't even know where is taking the conversion value to pass from COP to USD.

Gioyik commented 4 years ago

I reverted back (with brew) to 1.17.1.1 and the output is the expected one, seems I'll stick with this version and avoid update for now.

simonmichael commented 4 years ago

Thanks for the report. Yes, I’m sorry about the change in output. Can you check the upgrade note in https://hledger.org/release-notes.html#journal-format and the doc https://hledger.org/hledger.html#market-prices and issue #1239 and let me know your thoughts.

Gioyik commented 4 years ago

oh, I checked the Changelog file in the repo but seems I didn't read the whole document, sorry for that. I'm trying to get the same report as before but has been more complicated than I thought. I'm using:

hledger bs date:today -V --value=now,USD,COP

This returns:

Balance Sheet 2020-06-08, current value

                          ||                      2020-06-08
==========================++=================================
 Assets                   ||
--------------------------++---------------------------------
 Assets                   ||                  500.200,00 COP, 1 AMD, 2 KO, 1 PEP
   CO                     ||                  500.200,00 COP
     Cash                 ||                  500.200,00 COP
   USD                    ||                  1 AMD, 2 KO, 1 PEP
     Stock                ||                  1 AMD, 2 KO, 1 PEP
==========================++=================================
 Liabilities              ||
--------------------------++---------------------------------
 Liabilities              ||        100.269,29 COP, 6.19 USD
   CO                     ||        100.269,29 COP, 6.19 USD
     Bank                 ||        100.269,29 COP, 6.19 USD
       XXX                ||        100.269,29 COP, 6.19 USD
         Credit Card      ||        100.269,29 COP, 6.19 USD
--------------------------++---------------------------------

It's almost back to normal, however, Stock is not being transformed into USD, this is one of my stock transactions:

2020-06-04 1 AMD @ 11.11 USD
  Assets:US:Cash  -11.11 USD @@ 1 AMD
  Assets:US:Stock

so close!

simonmichael commented 4 years ago

It’s a little hard to say what’s up without a full example.. what P directives do you have ?

The —value looks strange, there should be at most one commodity. The extra -V is redundant.

adept commented 4 years ago

Btw, this looks odd to me (note how description says exact opposite of what is recorded):

2020-06-04 1 AMD @ 11.11 USD
  Assets:US:Cash  -11.11 USD @@ 1 AMD
  Assets:US:Stock

Taken at face value, it is a debit (sell) of $11.11 at cost of 1 AMD.

On the surface, everything is fine:

$ hledger bal -f /tmp/amd.journal 
               1 AMD           
          -11.11 USD  Assets:US
          -11.11 USD    Cash
               1 AMD    Stock
--------------------
               1 AMD
          -11.11 USD

Valued at cost, this will look weird, as you have dollars valued in AMD stock:

$ hledger bal -f /tmp/amd.journal -V
                   0  Assets:US
               AMD-1    Cash
               1 AMD    Stock
--------------------
                   0

Compare this with:

2020-06-04 1 AMD @ 11.11 USD
  Assets:US:Cash  
  Assets:US:Stock  1 AMD @@ 11.11 USD

Same picture as before without -V:

$ hledger bal -f /tmp/amd2.journal 
               1 AMD           
          -11.11 USD  Assets:US
          -11.11 USD    Cash
               1 AMD    Stock
--------------------
               1 AMD
          -11.11 USD

But much better with -V:

$ hledger bal -f /tmp/amd2.journal -V
                   0  Assets:US
          -11.11 USD    Cash
               USD11    Stock
--------------------
                   0
Gioyik commented 4 years ago

I found the same when trying to figure out what's going on with my transactions in the journal files, seems I need to keep learning how to improve the way register transactions, this is my third month with hledger. Thanks for the example @adept.

For some reason, I was thinking the reason for the root issue described by me at the beginning was my stock transactions, but found was another type of transaction I was doing, let me paste an example here for reference:

2018-11-01
  income:salary
  assets:usbank  1000.00 USD

2018-11-01
  income:consulting
  assets:cobank  50000,00 COP

2018-11-02
  assets:cobank
  expenses:food  5000,00 COP

2018-11-02
  assets:stock  2 AMD @@ 40.00 USD
  assets:usbank

P 2018-11-03 AMD 21.00 USD

2018-11-04
  liabilities:credit card  -50.00 USD
  expenses:video games

2018-11-04
  liabilities:credit card  -50.000,00 COP
  expenses:food

P 2018-11-04 AMD 22.00 USD

2018-11-05
  liabilities:credit card  25.00 USD
  assets:usbank

2018-11-06
  income:salary
  assets:usbank  1000.00 USD

Until this moment everything looks fine:

$ hledger bal -f /test.journal -V

               ||                 2018-11-06
===============++============================
 Assets        ||
---------------++----------------------------
 assets        || 45.000,00 COP, 1979.00 USD
   cobank      ||              45.000,00 COP
   stock       ||                  44.00 USD
   usbank      ||                1935.00 USD
---------------++----------------------------
               || 45.000,00 COP, 1979.00 USD
===============++============================
 Liabilities   ||
---------------++----------------------------
 liabilities   ||   50.000,00 COP, 25.00 USD
   credit card ||   50.000,00 COP, 25.00 USD
---------------++----------------------------
               ||   50.000,00 COP, 25.00 USD
===============++============================
 Net:          || -5.000,00 COP, 1954.00 USD

But I move money from my usbank to my cobank, and this is how I was registering the transaction:

2018-06-06 from US bank to CO bank
  expenses:transfer fee  30.00 USD
  assets:receivable  3.000.000,00 COP  @@ 970.00 USD
  assets:usbank

I'm not sure why I thought it was correct, maybe because, in the end, my balances were accurate with reality so I didn't care too much about checking if they made sense, but now that I have read more how this type of conversions needs to be done, I see what's wrong. This is the output:

               ||  2018-11-06
===============++=============
 Assets        ||
---------------++-------------
 assets        || 1963.55 USD
   cobank      ||   14.55 USD
   receivable  ||  970.00 USD
   stock       ||   44.00 USD
   usbank      ||  935.00 USD
---------------++-------------
               || 1963.55 USD
===============++=============
 Liabilities   ||
---------------++-------------
 liabilities   ||   41.17 USD
   credit card ||   41.17 USD
---------------++-------------
               ||   41.17 USD
===============++=============
 Net:          || 1922.38 USD

As you can see, in 1.18 my COP balance gets converted to USD, something didn't happen in 1.17. However, I'm not blaming hledger at all, I have been reading issues #1177 and I think I have come with a better transaction form for my case:

2018-06-06 from US bank to CO bank
  expenses:transfer fee  30.00 USD
  assets:usbank  -30.00 USD
  assets:receivable  3.000.000,00 COP
  assets:usbank  -970.00 USD
  equity:conversion

This is the output:

               ||                   2018-11-06
===============++==============================
 Assets        ||
---------------++------------------------------
 assets        || 3.045.000,00 COP, 979.00 USD
   cobank      ||                45.000,00 COP
   receivable  ||             3.000.000,00 COP
   stock       ||                    44.00 USD
   usbank      ||                   935.00 USD
---------------++------------------------------
               || 3.045.000,00 COP, 979.00 USD
===============++==============================
 Liabilities   ||
---------------++------------------------------
 liabilities   ||     50.000,00 COP, 25.00 USD
   credit card ||     50.000,00 COP, 25.00 USD
---------------++------------------------------
               ||     50.000,00 COP, 25.00 USD
===============++==============================
 Net:          || 2.995.000,00 COP, 954.00 USD

it makes more sense to me now but not saying is actually the best/correct way to do it, I'm probably still misunderstanding something. Any feedback is highly appreciated.

simonmichael commented 4 years ago

It's hard for me to follow the discussions/inexact examples above. From looking at another example, I've noticed two ways in which 1.18 can cause problems:

  1. 1.18 uses a more recent market price inferred from a transaction, instead of the P-declared market price which 1.17 used. Symptom: numbers look different compared to 1.17, due to different conversion rates.

  2. 1.18 selects different default valuation commodities because of the market prices inferred from transactions. Symptom: -V shows different commodities compared to 1.17, though -X DESIREDCOMMODITY works as before.

simonmichael commented 4 years ago

PS, I'll continue the above thread over on #1239, and keep this issue for the original problem(s) reported here, which may or may not be the same.

Gioyik commented 4 years ago

no problem, thanks for taking the time to dig more in this.

simonmichael commented 4 years ago

@Gioyik, re your 2020-06-12 comment: I think hledger 1.18 is working as intended there: it successfully converts to USD, unlike hledger 1.17. With the #1177-compliant journal entry, it's not expected to work, because we can't detect the transaction price from those entries; P directives must be used in that case.

Gioyik commented 4 years ago

@simonmichael ok, I kinda understand in 1.18 should behave in this way, but in my case, I manage two currencies, I don't like my COP balances to be converted to USD or vice versa because they both are from different accounts and mean different things. So, does this end meaning that -V will convert my balances into one currency if I manage more than one? I'm ok if that's the expected behavior of the flag, I'll be using the compliant journal entry and P directives to not let hledger convert COP currency accounts into USD when I don't want them to be converted.

simonmichael commented 4 years ago

Hmm, -V “converts to value”. What would you want to happen ? Convert only certain commodities ?

Gioyik commented 4 years ago

in my previous comment, I describe that 1.17 only the stock I had set P values was converted to value with -V. This was the output:

               ||                   2018-11-06
===============++==============================
 Assets        ||
---------------++------------------------------
 assets        || 3.045.000,00 COP, 979.00 USD
   cobank      ||                45.000,00 COP
   receivable  ||             3.000.000,00 COP
   stock       ||                    44.00 USD
   usbank      ||                   935.00 USD
---------------++------------------------------
               || 3.045.000,00 COP, 979.00 USD
===============++==============================
 Liabilities   ||
---------------++------------------------------
 liabilities   ||     50.000,00 COP, 25.00 USD
   credit card ||     50.000,00 COP, 25.00 USD
---------------++------------------------------
               ||     50.000,00 COP, 25.00 USD
===============++==============================
 Net:          || 2.995.000,00 COP, 954.00 USD

As you see, I still have accounts with COP balance and USD and COP balances at the same time, accounts:stock was converted to value and that has been the normal behavior since I started to use hledger, but with 1.18, my stock is converted to value and my accounts with COP balances, generating this output:

               ||  2018-11-06
===============++=============
 Assets        ||
---------------++-------------
 assets        || 1963.55 USD
   cobank      ||   14.55 USD
   receivable  ||  970.00 USD
   stock       ||   44.00 USD
   usbank      ||  935.00 USD
---------------++-------------
               || 1963.55 USD
===============++=============
 Liabilities   ||
---------------++-------------
 liabilities   ||   41.17 USD
   credit card ||   41.17 USD
---------------++-------------
               ||   41.17 USD
===============++=============
 Net:          || 1922.38 USD

I don't want my COP accounts to be converted to USD, I was only expecting the stock to be converted to value based on P values set in my journal files.

This is the journal to replicate the behavior I'm describing between both versions:

2018-11-01
  income:salary
  assets:usbank  1000.00 USD

2018-11-01
  income:consulting
  assets:cobank  50000,00 COP

2018-11-02
  assets:cobank
  expenses:food  5000,00 COP

2018-11-02
  assets:stock  2 AMD @@ 40.00 USD
  assets:usbank

P 2018-11-03 AMD 21.00 USD

2018-11-04
  liabilities:credit card  -50.00 USD
  expenses:video games

2018-11-04
  liabilities:credit card  -50.000,00 COP
  expenses:food

P 2018-11-04 AMD 22.00 USD

2018-11-05
  liabilities:credit card  25.00 USD
  assets:usbank

2018-11-06
  income:salary
  assets:usbank  1000.00 USD

2018-06-06 from US bank to CO bank
  expenses:transfer fee  30.00 USD
  assets:receivable  3.000.000,00 COP  @@ 970.00 USD
  assets:usbank

The last transaction from US bank to CO bank is the one who makes hledeger 1.18 convert my COP balances into USD, because if you remove it, hledger 1.18 with -V output balances with COP and USD, not everything converted into USD.

I hope it makes sense, sorry if I'm not good at explaining my case.

simonmichael commented 4 years ago

@Gioyik ah, I didn't pick up on it, thanks for this timely feedback. So: you keep assets in two commodities and you don't want everything converted to one commodity (at least, not always). And for this, 1.17's simple reliance on P directives worked well, while 1.18's more aggressive conversion based also on transaction prices is undesirable, requiring you to take extra steps to defeat it (eg by using current master and adding more P directives, or by rewriting transactions in #1177's @-free style).

Based on this I start to think that inferring market prices from transactions should be an option, off by default (or, on by default only when there are no P directives at all) ?

For comparison, Ledger takes market prices from both P and transactions by default, and you can disable the former with --price/-I, but not the latter.

@Gioyik: what would you want -X to do in your example ? Eg -X USD or -X COP.

amitaibu commented 4 years ago

inferring market prices from transactions should be an option, off by default

I'd be in favor of that. I wouldn't mind what the default would be, but in my use case, inferring prices is not needed.

Gioyik commented 4 years ago

@simonmichael nice, your resume is accurate with my use case, in my opinion, 1.18 breaks -V output for different currency accounts if you transfer from one currency to another with @@. This type of transfers make you lose track of per currency account value in 1.18:

2018-06-06 from US bank to CO bank
  expenses:transfer fee  30.00 USD
  assets:receivable  3.000.000,00 COP  @@ 970.00 USD
  assets:usbank

In 1.17 that was not the case, but I understand why it changed.

Based on this I start to think that inferring market prices from transactions should be an option, off by default (or, on by default only when there are no P directives at all) ?

I like this approach because it doesn't break previous behavior and people don't need to rewrite @@ transactions with equity accounts, but most importantly, it doesn't break value output for different currency accounts.

what would you want -X to do in your example ? Eg -X USD or -X COP.

Because I mainly relay in P values, I would expect -X to convert to the commodity I want if I have P values for. For example, if I have stock and P values for in USD, convert those, but don't try to convert everything else into USD, if I have P values for stock in COP, then go for it and convert them into COP if I pass -X COP to hledger.

simonmichael commented 4 years ago

New docs mockup posted on #1239.

simonmichael commented 4 years ago

The --infer-values flag has landed in master:

  • by default everything works the same way as in 1.17
  • you can add --infer-values to infer market prices from transactions, and get value reports without the need for P directives
  • semantics and docs are clearer

@Gioyik & @amitaibu, this avoids the problems you were reporting I believe, but I would appreciate any extra testing.

simonmichael commented 4 years ago

Correction! --infer-value, with no s.

Gioyik commented 4 years ago

whoa, thanks so much @simonmichael, --infer-value makes more sense, keeps previous journal transactions in order and now it's up to me decide when I need it.

Thanks again for taking the time to hear the feedback and get back really fast with a fix for this specific case 👍

simonmichael commented 4 years ago

Fixed by 1.18.1, thanks all.

amitaibu commented 4 years ago

Works great, thanks! :+1: