search

beancount / beangrow

Returns calculations on portfolios in Beancount
GNU General Public License v2.0
56 stars 21 forks source link

Error message in Beancount 2.3.3 #7

Open prabusw opened 3 years ago

prabusw commented 3 years ago

Martin,

When i try to run the code, i'm getting the below error message in Beancount 2.3.3... I can send you the ledger file privately, if you mail me.

$:~/beancount/beangrow-master/beangrow$ python compute_returns.py ~prabu/beancount/prabu/prabu2021_02_20.beancount config.pbtxt return_reports /home/prabu/beancount/beangrow-master/beangrow/returns.py:63: RuntimeWarning: invalid value encountered in power return np.sum(cash_flows / (1. + irr) years) /home/prabu/beancount/beangrow-master/beangrow/returns.py:63: RuntimeWarning: invalid value encountered in power return np.sum(cash_flows / (1. + irr) years) /home/prabu/beancount/beangrow-master/beangrow/returns.py:63: RuntimeWarning: invalid value encountered in power return np.sum(cash_flows / (1. + irr) ** years) Traceback (most recent call last): File "compute_returns.py", line 105, in main() File "compute_returns.py", line 100, in main reports.write_price_directives(path.join(output_prices, "prices.beancount"), File "/home/prabu/beancount/beangrow-master/beangrow/reports.py", line 406, in write_price_directives days_late = (required_date - actual_date).days TypeError: unsupported operand type(s) for -: 'datetime.date' and 'NoneType'

Here is the verbose output prabu@homepc-lm:~/beancount/beangrow-master/beangrow$ python compute_returns.py ~prabu/beancount/prabu/prabu2021_02_20.beancount config.pbtxt return_reports -v INFO : Reading ledger: /home/prabu/beancount/prabu/prabu2021_02_20.beancount INFO : Processing account: Assets:TH:Investment:KGI:INTUCH-R INFO : Processing account: Assets:US:Investment:ETrade:BP INFO : Processing account: Assets:US:Investment:ETrade:CVX INFO : Processing account: Assets:US:Investment:ETrade:EQIX INFO : Processing account: Assets:US:Investment:ETrade:IXC INFO : Processing account: Assets:US:Investment:ETrade:TOT INFO : Processing account: Assets:US:Investment:ETrade:RDS-B INFO : Processing account: Assets:US:Investment:ETrade:XLE INFO : Processing account: Assets:US:Investment:ETrade:XOM INFO : Writing details file: return_reports/investments/Assets_TH_Investment_KGI_INTUCH-R.org INFO : Writing details file: return_reports/investments/Assets_US_Investment_ETrade_BP.org INFO : Writing details file: return_reports/investments/Assets_US_Investment_ETrade_CVX.org INFO : Writing details file: return_reports/investments/Assets_US_Investment_ETrade_EQIX.org INFO : Writing details file: return_reports/investments/Assets_US_Investment_ETrade_IXC.org INFO : Writing details file: return_reports/investments/Assets_US_Investment_ETrade_TOT.org INFO : Writing details file: return_reports/investments/Assets_US_Investment_ETrade_RDS-B.org INFO : Writing details file: return_reports/investments/Assets_US_Investment_ETrade_XLE.org INFO : Writing details file: return_reports/investments/Assets_US_Investment_ETrade_XOM.org INFO : Writing returns dir for strategy:GlobalEnergy: return_reports/groups/strategy:GlobalEnergy DEBUG : top of axes not in the figure, so title not moved DEBUG : top of axes not in the figure, so title not moved /home/prabu/beancount/beangrow-master/beangrow/returns.py:63: RuntimeWarning: invalid value encountered in power return np.sum(cash_flows / (1. + irr) years) INFO : Writing returns dir for strategy:GlobalEnergy: return_reports/groups/strategy:GlobalEnergy.org INFO : Writing returns dir for currency.EQIX: return_reports/groups/currency.EQIX DEBUG : top of axes not in the figure, so title not moved DEBUG : top of axes not in the figure, so title not moved /home/prabu/beancount/beangrow-master/beangrow/returns.py:63: RuntimeWarning: invalid value encountered in power return np.sum(cash_flows / (1. + irr) years) INFO : Writing returns dir for currency.EQIX: return_reports/groups/currency.EQIX.org INFO : Writing returns dir for Thailand.Intuch: return_reports/groups/Thailand.Intuch DEBUG : top of axes not in the figure, so title not moved DEBUG : top of axes not in the figure, so title not moved /home/prabu/beancount/beangrow-master/beangrow/returns.py:63: RuntimeWarning: invalid value encountered in power return np.sum(cash_flows / (1. + irr) ** years) INFO : Writing returns dir for Thailand.Intuch: return_reports/groups/Thailand.Intuch.org INFO : Producing price page for ('BP', 'USD') INFO : Producing price page for ('CVX', 'USD') INFO : Producing price page for ('EQIX', 'USD') INFO : Producing price page for ('INTUCH-R', 'THB') INFO : Producing price page for ('IXC', 'USD') INFO : Producing price page for ('RDS-B', 'USD') INFO : Producing price page for ('TOT', 'USD') INFO : Producing price page for ('XLE', 'USD') INFO : Producing price page for ('XOM', 'USD') Traceback (most recent call last): File "compute_returns.py", line 105, in main() File "compute_returns.py", line 100, in main reports.write_price_directives(path.join(output_prices, "prices.beancount"), File "/home/prabu/beancount/beangrow-master/beangrow/reports.py", line 406, in write_price_directives days_late = (required_date - actual_date).days TypeError: unsupported operand type(s) for -: 'datetime.date' and 'NoneType'

redstreet commented 3 years ago

I had exactly the same issue (among several others), and had it figured out, but unfortunately seem to have lost my notes on what was causing it. I'd be happy to debug this if you'd be okay sharing your ledger file with me. If so, please message me. Perhaps you could anonymize it, change numbers etc., or produce a separate minimal example.

prabusw commented 3 years ago

Dear Redstreet,

Thanks for writing back. I'm attaching the ledger file and the pbtxt that i used to receive the above error.

I've not yet fully gone through the document created by Martin yet for beangrow. I'm still trying to understand it. The changes related to Cash flow are still missing in my ledger.

Please let me know if you need any clarification in the ledger.

Cheers, Prabu Anand

On Sun, 28 Feb 2021 at 16:18, Red S notifications@github.com wrote:

I had exactly the same issue (among several others), and had it figured out, but unfortunately seem to have lost my notes on what was causing it. I'd be happy to debug this if you'd be okay sharing your ledger file with me. If so, please message me. Perhaps you could anonymize it, change numbers etc., or produce a separate minimal example.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/beancount/beangrow/issues/7#issuecomment-787432300, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQRA6MK3UTX4X6JLMQHI253TBINQNANCNFSM4YDREHIQ .

redstreet commented 3 years ago

Hello @prabusw, the attachment doesn't seem to have come through. In case you meant to send it to my email, my address is ---removed---.

redstreet commented 3 years ago

Thanks for sharing, @prabusw! I took a look. There are several issues to fix in beangrow, and a few tips that might help you:

  1. Your source doesn't include explicit price entries needed to compute returns. That's totally fine and not surprising - beangrow should error out gracefully, but instead crashes with the TypeError (I'll send a PR for this later, @blais). This results in meaningless returns values for the second (out of your three) groups in your config, and also is the source of the final TypeError. To fix this, include this line in your beancount source so the prices can come from your buy/sell transactions to start with:

    plugin "beancount.plugins.implicit_prices"

    This should make the TypeError go away.

  2. After including the above, you will want to run:

    beangrow/beangrow/download_prices_from_file.py <your_returns_output_dir>/prices/prices.beancount

    and include the resulting price database in your source. Without that, beangrow uses the price values on the closest dates it can find, which in your case could be weeks or months off, which means your returns numbers will be off. Do take a look at your returns numbers after doing this and see if they make sense. I haven't examined them. Ignore the two appearances of "20.0%", which is the solver crashing (the RuntimeWarning below).

  3. The RuntimeWarning, is raised by the solver for the net present value. It's for your account that was zeroed out in 2020. All your other returns numbers should be correct apart from this one. I think I know what the problem is here. More on this later, probably next weekend when I can look into it again.

Hope that helped! Do take a look at your returns numbers after (1) and (2) above, and feel free to post questions here or in the forum.

prabusw commented 3 years ago

Dear Redstreet.

Thanks for the pointer to the plugin and the script that generates the prices. These are the steps i've followed:

  1. Included the below line in ledger_p.beancount as advised..

plugin "beancount.plugins.implicit_prices"

I had to change the below line in the script download_prices_from_file.py, as i'm still on Beancount 2.3.3 from beancount.prices.sources import yahoo

from beanprice.sources import yahoo

  1. Ran the compute returns script as below to generate the folder return_reports2 $./compute_returns.py ledger_p.beancount config_a.pbtxt return_reports2

  2. I'm still confused about the purpose of the below script in step3. How does prices.beancount get created? How does the script pick my ledger file, if I don't mention it as an argument? $./download_prices_from_file.py return_report/prices/prices.beancount

Note: I still have not tried many things in beancount like plugins, intelligent importers like those that you've published, reports using export options etc. Only recently I have learnt emacs to use beancount major mode in it. Beancount on emacs is extremely easy to use, once one learns emacs. All my beancount data entry last year was done using fava. Thank you for your help along with Martin and the fava team.

On Mon, 1 Mar 2021 at 16:23, Red S notifications@github.com wrote:

Thanks for sharing, @prabusw https://github.com/prabusw. I took a look. There are several issues for beangrow to fix, and a few tips that might help you:

  1. Your source doesn't include explicit price entries needed to compute returns. beangrow should error out gracefully, but instead crashes with the TypeError. This results in meaningless returns values for the second (out of your three) groups in your config, and also is the source of the final TypeError. To fix this, first include this line in your beancount source:

plugin "beancount.plugins.implicit_prices"

  1. After fixing the above, you will want to run:

beangrow/beangrow/download_prices_from_file.py /prices/prices.beancount

and include the resulting price database in your source. Without that, beangrow uses the price values on the closest dates it can find, which in your case could be weeks or months off, which means your returns numbers will be off.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/beancount/beangrow/issues/7#issuecomment-787854255, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQRA6MMP227D37AYGFNIIZDTBNWYZANCNFSM4YDREHIQ .

redstreet commented 3 years ago

Hello @prabusw, When compute_returns.py it's run for the first time on your ledger file, it figures out the price entries it needs on the dates it needs them in order to make an accurate returns computation.

As an example, to calculate the returns for each calendar year, it needs a price entry for each commodity on Jan 1 of each calendar year. Your ledger file may not have these entries. So ./compute_returns.py takes the closest price entries it can find to those dates and creates the prices.beancount with a list of those missing entries.

When you run the download_prices script, it therefore does not need access to your ledger. It only needs to know the list of dates and commodities that are in prices.beancount, for which it goes and fetches the actual prices (remember, the entries in prices.beancount thus far are the closest ones the first pass above could find, and not exact).

Martin's "Filling in Missing Price Points" in his document in the readme file of beangrow explains this very well.

So once the prices are fetched, you append these to your ledger file and then run compute_returns again, and this time it will produce more accurate results.

Feel free to look at your returns numbers right at the beginning when you run compute_returns for the first time. Open up the index.html files in your returns directory if you already haven't, in a browser. Then, download prices, rerun, and look at the returns again.

Hope that helped, and glad to hear you're finding beancount and its ecosystem useful :).