register command csv output amount column missing commodity when amount is zero

[trevorld]

Sorry if this is a duplicate issue but I couldn't find a mention of it. I observe missing commodities in the amount column in the csv output from hledger register when the amount is exactly zero.

$ bean-example > example.beancount
$ bean-report example.beancount hledger > example.hledger
$ hledger register -f example.hledger -o example.csv
$ csvgrep -r "2016/08/04" -c date example.csv | csvgrep -r "Assets" -c account | csvlook
|---------+------------+------------------------------+------------------------------+----------------+---------------------------------------------------------|
|  txnidx | date       | description                  | account                      | amount         | total                                                   |
|---------+------------+------------------------------+------------------------------+----------------+---------------------------------------------------------|
|  237    | 2016/08/04 | BANK FEES | Monthly bank fee | Assets:US:BofA:Checking      | -4.00000 USD   | 265.021 RGAGX, -27003.97697 USD, 57.393 VBMPX           |
|  239    | 2016/08/04 | Hooli | Payroll              | Assets:US:BofA:Checking      | 2550.60000 USD | 265.021 RGAGX, -24449.37697 USD, 57.393 VBMPX           |
|  239    | 2016/08/04 | Hooli | Payroll              | Assets:US:Federal:PreTax401k | 0              | 265.021 RGAGX, -26999.97697 USD, 57.393 VBMPX           |
|  239    | 2016/08/04 | Hooli | Payroll              | Assets:US:Hooli:Vacation     | 5 VACHR        | 265.021 RGAGX, -26999.97697 USD, 5 VACHR, 57.393 VBMPX  |
|---------+------------+------------------------------+------------------------------+----------------+---------------------------------------------------------|

In the third row the amount should be a 0.00 IRAUSD but is instead 0. Here is the transaction in example.hledger:

2016-08-18 * "Hooli" "Payroll"
  Assets:US:BofA:Checking                         2550.60 USD   
  Income:US:Hooli:Salary                         -4615.38 USD   
  Income:US:Hooli:GroupTermLife                    -24.32 USD   
  Expenses:Health:Life:GroupTermLife                24.32 USD   
  Expenses:Health:Dental:Insurance                   2.90 USD   
  Expenses:Health:Medical:Insurance                 27.38 USD   
  Expenses:Health:Vision:Insurance                  42.30 USD   
  Expenses:Taxes:Y2016:US:Medicare                 106.62 USD   
  Expenses:Taxes:Y2016:US:Federal                 1062.92 USD   
  Expenses:Taxes:Y2016:US:State                    365.08 USD   
  Expenses:Taxes:Y2016:US:CityNYC                  174.92 USD   
  Expenses:Taxes:Y2016:US:SDI                        1.12 USD   
  Expenses:Taxes:Y2016:US:SocSec                   281.54 USD   
  Assets:US:Federal:PreTax401k                       0.00 IRAUSD
  Assets:US:Hooli:Vacation                              5 VACHR 
  Income:US:Hooli:Vacation                             -5 VACHR 

$ hledger --version
hledger 1.9

[simonmichael]

Thanks for the report. I don't know if this is a bug.. you will probably see the same in console output ? Generally we consider 0 to be commodityless (it could be any or all commodities, how would you know ?) Though, I do remember some change in this area to fix the charts in hledger-web.

[simonmichael]

Another way to say it: hledger doesn't know or enforce that certain accounts might contain only a particular commodity.

[trevorld]

it could be any or all commodities, how would you know ?

I thought you would know because a user specifically told it what commodity it was in the hledger transaction? i.e.

Assets:US:Federal:PreTax401k                       0.00 IRAUSD

I see that in such cases that ledger csv actually drops zero amount observations from its csv output

$ bean-report example.beancount ledger > example.ledger
$ ledger csv -f example.ledger > ledger_example.csv
$ csvgrep -r "2016/08/04" -c 1 ledger_example.csv | csvgrep -r "Assets" -c 4 | csvlook -H
|-------------+---------+--------------------------------------+--------------------------+---------+---------+---------+----------|
|  column1    | column2 | column3                              | column4                  | column5 | column6 | column7 | column8  |
|-------------+---------+--------------------------------------+--------------------------+---------+---------+---------+----------|
|  2016/08/04 |         | BANK FEES | Monthly bank fee         | Assets:US:BofA:Checking  | USD     | -4      | *       |          |
|  2016/08/04 |         | Hooli | Payroll                      | Assets:US:BofA:Checking  | USD     | 2550.6  | *       |          |
|  2016/08/04 |         | Hooli | Payroll                      | Assets:US:Hooli:Vacation | VACHR   | 5       | *       |          |
|-------------+---------+--------------------------------------+--------------------------+---------+---------+---------+----------|

The current behaviour lead to an annoying extra row when I computed the net worth of the example beancount file in my ledger R package (by default I currently import such files via bean-report hledger followed by hledger register -o csv) because I computed market value by aggregating by market value for each commodity found in the data and this introduces a new "missing" commodity to the data. Although I probably shouldn't strip zeroes upon importing the register in general (people may very well be interested in plotting/analysing zero events) I guess I could safely filter out such zeroes before computing a net worth total.

> library("ledger")
> example_beancount_file <- tempfile(fileext = ".beancount")
> system(paste("bean-example -o", example_beancount_file), ignore.stderr=TRUE)
> ledger::net_worth(example_beancount_file)
# A tibble: 4 x 5
        date commodity net_worth   assets liabilities
      <date>     <chr>     <dbl>    <dbl>       <dbl>
1 2018-06-11    IRAUSD    4100.0   4100.0        0.00
2 2018-06-11       USD  111440.5 113725.7    -2285.27
3 2018-06-11     VACHR      72.0     72.0        0.00
4 2018-06-11      <NA>       0.0      0.0        0.00

[simonmichael]

I thought you would know because a user specifically told it what commodity it was in the hledger transaction? i.e.

Assets:US:Federal:PreTax401k 0.00 IRAUSD

True, we could be expected to remember that.

I had the same problem in hledger-web's chart but apparently resolved it with just a local workaround, f2d9c6e9.

[simonmichael]

So, if anyone wants to try making zero amounts better at remembering a commodity symbol (just one, I suppose), I would be interested to test it. I expect it will have some impact, but I'm not sure what.

[robert-e-davidson3]

This is also true of import, which breaks strict mode.

CSV:

2020-01-01T01:01:01Z,Rewards Income,DAI,0.00111111,1.02,0.00,0.00,0.00,Received 0.00111111 DAI from Coinbase Rewards

(the %quantity value is "0.00")

Rule:

if %transaction_type Rewards Income
 amount1 +%quantity @ %usd_spot_price USD
 amount2 -%quantity @ %usd_spot_price USD
 account2 revenues:coinbase:reward

Produces:

2020-01-01 Received 0.0011111 DAI from Coinbase Rewards
    assets:coinbase             0 @ 1.02 USD
    revenues:coinbase:reward    0 @ 1.02 USD

Resulting in error:

% hledger accounts -s                  
hledger: undeclared commodity ""
in transaction at: "/home/robertdavidson/org/finance/2020.journal" (lines 60-62)

  2020-01-01 Received 0.00111111 DAI from Coinbase Rewards
      assets:coinbase             0 @ 1.02 USD
      revenues:coinbase:reward    0 @ 1.02 USD

[simonmichael]

@robert-e-davidson3, how does the currency get set for a non-zero transaction ?