Loading Asset>Balance takes too long

[sdhawade]

Loading the Asset Balance page takes greater than 5 seconds. So if one drills down into a stock and then comes back it takes 5 seconds again. Another issue is Warnings that XIRR is not converging, wonder if it related to why it takes long to load the page as it may be taking too many iterations before giving up..

Can the computed data be cached and updated only if the journal/prices are updated ?

See log below : 2024-01-06 22:47:21 INFO GET 200 5790ms /api/assets/balance clientIP=192.0.2.1file=D:/a/paisa/paisa/internal/server/logger.go:43func=github.com/ananthakumaran/paisa/internal/server.Logger.func1referer=http://wails.localhost/assets/balance 2024-01-06 22:47:21 WARNING XIRR didn't converge file=D:/a/paisa/paisa/internal/xirr/xirr.go:71func=github.com/ananthakumaran/paisa/internal/xirr.calculateXIRR 2024-01-06 22:47:21 WARNING XIRR didn't converge file=D:/a/paisa/paisa/internal/xirr/xirr.go:71func=github.com/ananthakumaran/paisa/internal/xirr.calculateXIRR 2024-01-06 22:47:08 INFO GET 200 16ms /api/gain/Assets:Equity:ZeroDha:TATATECH clientIP=192.0.2.1file=D:/a/paisa/paisa/internal/server/logger.go:43func=github.com/ananthakumaran/paisa/internal/server.Logger.func1referer=http://wails.localhost/assets/gain/Assets:Equity:ZeroDha:TATATECH 2024-01-06 22:46:17 INFO GET 200 5775ms /api/assets/balance clientIP=192.0.2.1file=D:/a/paisa/paisa/internal/server/logger.go:43func=github.com/ananthakumaran/paisa/internal/server.Logger.func1referer=http://wails.localhost/assets/balance 2024-01-06 22:46:17 WARNING XIRR didn't converge file=D:/a/paisa/paisa/internal/xirr/xirr.go:71func=github.com/ananthakumaran/paisa/internal/xirr.calculateXIRR 2024-01-06 22:46:12 WARNING XIRR didn't converge file=D:/a/paisa/paisa/internal/xirr/xirr.go:71func=github.com/ananthakumaran/paisa/internal/xirr.calculateXIRR 2024-01-06 22:45:53 INFO GET 200 9242ms /api/price clientIP=192.0.2.1file=D:/a/paisa/paisa/internal/server/logger.go:43func=github.com/ananthakumaran/paisa/internal/server.Logger.func1referer=http://wails.localhost/ledger/price 2024-01-06 22:45:44 INFO POST 200 17716ms /api/sync clientIP=192.0.2.1file=D:/a/paisa/paisa/internal/server/logger.go:43func=github.com/ananthakumaran/paisa/internal/server.Logger.func1referer=http://wails.localhost/ledger/price

[ananthakumaran]

XIRR might be one reason, though, it doesn't explain why /api/price is so slow. For me, none of the APIs go above 500ms (at which point I start optimizing things). Paisa comes with a debug log level which will show much more info, you can enable it by setting the PAISA_DEBUG env variable. Run the following command PAISA_DEBUG=true paisa serve and see if anything stands out.

[sdhawade]

Hi I tried setting the env variable to true but this does not seem to generate any additional logs.. Can you tell me what I am doing wrong ?

Shell setting

logs 2024-01-07 23:33:23 INFO GET 200 5733ms /api/assets/balance clientIP=192.0.2.1file=D:/a/paisa/paisa/internal/server/logger.go:43func=github.com/ananthakumaran/paisa/internal/server.Logger.func1referer=http://wails.localhost/assets/balance 2024-01-07 23:33:23 WARNING XIRR didn't converge file=D:/a/paisa/paisa/internal/xirr/xirr.go:71func=github.com/ananthakumaran/paisa/internal/xirr.calculateXIRR 2024-01-07 23:33:18 WARNING XIRR didn't converge file=D:/a/paisa/paisa/internal/xirr/xirr.go:71func=github.com/ananthakumaran/paisa/internal/xirr.calculateXIRR

[ananthakumaran]

I tested it on my Windows 10 VM and it works as expected (you can see the DEBUG logs with SQL queries and their timings), this is from Powershell. I don't know if the Powershell makes any difference.

[sdhawade]

Got it, you were refering to the cli version, I was using the windows thick client..

---- And the trace to load the Asset Balance-- DEBUG [logger.go:63] Trace SELECT * FROM postings WHERE (account like "Assets:%" or account like "Income:CapitalGains:%") AND forecast = false ORDER BY date ASC, amount desc [138.5221ms] WARNING [xirr.go:71] calculateXIRR XIRR didn't converge WARNING [xirr.go:71] calculateXIRR XIRR didn't converge INFO [logger.go:43] func1 GET 200 5757ms /api/assets/balance clientIP="::1" referer="http://localhost:7500/assets/balance"

Seems the SQL query runs prety quickly and looks more n more like a XIRR calc issue..

[ananthakumaran]

can you check for the price page? why is it so slow? Also what's the rough transactions count? Please post more debug logs.

[sdhawade]

Price page loads under a second.. DEBUG [logger.go:63] Trace SELECT DISTINCT commodity FROM postings WHERE commodity != "INR" [4.9986ms] INFO [logger.go:43] func1 GET 200 771ms /api/price clientIP="::1" referer="http://localhost:7500/ledger/price"

Transaction count from ledger>transactions is 7289 transaction(s)

[ananthakumaran]

@sdhawade looks like you have more than 100 commodities? Is Balance the only page that is very slow? or there are other pages as well. I am mostly testing with 3k transactions, I will try to do some profiling with 10k transactions / 100+ commodities

The XIRR part is a little tricky, unlike other computations which are formula-based, for XIRR you make a guess and then try to check if the guess is correct and gradually improve your guesses. In some cases, this approach will not work and Paisa will try with a fallback approach which is quite slow, combined with the number of accounts you have, this might be very slow. Let me check if the fallback can be avoided or improved.

[sdhawade]

Yes these are the commodities bought and sold over the years. The current count (active) is 89 and the total number of commodities held over time is 245. Am aware of the complexity of XIRR hence was suggesting calculating that in the database when the transactions/prices are updated so that every page refresh does not need it.

[ananthakumaran]

Agreed, I will figure out a way to cache xirr calculation

[ananthakumaran]

@sdhawade can you try 0.6.5 and see if it improves the situation?

[sdhawade]

Big improvement 👍 INFO [logger.go:43] func1 GET 200 822ms /api/assets/balance clientIP="::1" referer="http://localhost:7500/assets/balance"