search

bisq-network / bisq

A decentralized bitcoin exchange network
https://bisq.network
GNU Affero General Public License v3.0
4.69k stars 1.27k forks source link

Speed up bond repository update and remove some raw types #7082

Closed stejbac closed 4 months ago

stejbac commented 5 months ago

Alleviate a cubic time bug during the update of the bonded roles repository, reducing it to quadratic running time. On Mainnet, this gives a roughly ten-fold speedup and should allow better scaling in the event that many new bonded roles are added.

Replace calls to BondedRolesRepository.findBondedAssetByHash with a lookup into a lazily initialised map of bonded assets (Roles) by hash (reset at the start of each call to BondRepository.update() to prevent stale caching). This avoids rescanning the roles list for every pair of roles and lockup tx outputs, thus reducing the number of steps (to highest order) from:

roles #roles #lockup-tx-outputs

to:

roles * #lockup-tx-outputs

(The logs show 2 or 3 calls to BondedRepository.update() every time a new block arrives, and while this was only taking around a second or so on Mainnet, it could, I think, potentially grow to something problematic with cubic scaling in the number of bonded roles.)

--

In addition to the above optimisation, this PR also generifies the raw types which involve the DAO classes (mainly Bond occurrences) and fixes a broken test encountered during that cleanup.

The code changes touch the DAO packages, though not in a way that should cause any functional change.

--

Before the optimisation (last commit in the PR), the following update times are seen in the logs on my laptop:

steven@debian:~/Documents/Java Projects/bisq$ grep "Apr-19.*update took" ~/.local/share/Bisq/bisq_1.log | head -n 40
Apr-19 16:41:39.982 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 5 ms 
Apr-19 16:45:11.840 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 896 ms 
Apr-19 16:45:13.109 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 917 ms 
Apr-19 17:00:47.838 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 612 ms 
Apr-19 17:00:50.993 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 823 ms 
Apr-19 17:00:55.297 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 722 ms 
Apr-19 17:13:02.580 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 608 ms 
Apr-19 17:13:03.728 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 617 ms 
Apr-19 17:13:07.579 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 793 ms 
Apr-19 17:24:23.431 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 614 ms 
Apr-19 17:24:25.180 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 706 ms 
Apr-19 17:24:28.818 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 617 ms 
Apr-19 17:41:35.146 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 610 ms 
Apr-19 17:41:38.878 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 737 ms 
Apr-19 17:43:21.343 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 678 ms 
Apr-19 17:43:23.798 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 624 ms 
Apr-19 17:47:53.033 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 605 ms 
Apr-19 17:47:55.819 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 720 ms 
Apr-19 17:59:54.365 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 735 ms 
Apr-19 17:59:56.519 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 622 ms 
Apr-19 18:03:49.322 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 603 ms 
Apr-19 18:03:51.959 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 623 ms 
Apr-19 18:24:27.585 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 597 ms 
Apr-19 18:24:30.208 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 608 ms 
Apr-19 18:32:32.426 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 825 ms 
Apr-19 18:43:15.572 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 688 ms 
Apr-19 18:43:17.389 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 665 ms 
Apr-19 18:43:20.846 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 832 ms 
Apr-19 18:54:44.159 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 671 ms 
Apr-19 18:54:50.715 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 638 ms 
Apr-19 18:54:53.389 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 700 ms 
Apr-19 18:57:36.168 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 645 ms 
Apr-19 18:57:39.637 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 838 ms 
Apr-19 18:57:42.323 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 981 ms 
Apr-19 19:03:17.933 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 631 ms 
Apr-19 19:03:23.409 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 671 ms 
Apr-19 19:03:27.281 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 650 ms 
Apr-19 19:10:10.297 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 622 ms 
Apr-19 19:10:14.412 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 663 ms 
Apr-19 19:10:18.469 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 644 ms

(The first very short update time of 5ms occurs when the Bisq instance is starting up and the DAO state has not yet been loaded.)

After the optimisation, the following update times are seen in the logs:

steven@debian:~/Documents/Java Projects/bisq$ grep "Apr-19.*update took" ~/.local/share/Bisq/bisq.log | head -n 40
Apr-19 22:20:59.456 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 91 ms 
Apr-19 22:21:06.250 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 87 ms 
Apr-19 22:21:08.282 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 75 ms 
Apr-19 22:21:27.895 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 58 ms 
Apr-19 22:21:30.207 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 61 ms 
Apr-19 22:21:54.428 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 67 ms 
Apr-19 22:21:56.546 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 72 ms 
Apr-19 22:21:57.590 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 60 ms 
Apr-19 22:36:55.658 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 58 ms 
Apr-19 22:36:58.761 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 62 ms 
Apr-19 22:37:02.351 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 57 ms 
Apr-19 22:40:36.976 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 59 ms 
Apr-19 22:40:39.478 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 84 ms 
Apr-19 22:40:42.849 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 62 ms 
Apr-19 22:47:37.600 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 60 ms 
Apr-19 22:47:38.518 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 99 ms 
Apr-19 22:47:40.648 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 74 ms 
Apr-19 22:53:42.378 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 61 ms 
Apr-19 22:53:46.480 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 64 ms 
Apr-19 22:53:49.793 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 62 ms 
Apr-19 22:57:50.609 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 64 ms 
Apr-19 22:57:56.863 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 66 ms 
Apr-19 22:57:59.834 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 148 ms 
Apr-19 23:01:20.784 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 72 ms 
Apr-19 23:01:24.188 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 87 ms 
Apr-19 23:15:13.056 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 57 ms 
Apr-19 23:15:13.982 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 52 ms 
Apr-19 23:15:18.548 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 62 ms 
Apr-19 23:24:37.167 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 60 ms 
Apr-19 23:24:38.181 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 62 ms 
Apr-19 23:24:41.172 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 62 ms 
Apr-19 23:33:01.655 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 66 ms 
Apr-19 23:33:02.848 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 67 ms 
Apr-19 23:33:06.434 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 72 ms 
Apr-19 23:33:44.921 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 61 ms 
Apr-19 23:33:48.624 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 60 ms 
Apr-19 23:33:50.790 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 73 ms 
Apr-19 23:39:43.736 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 58 ms 
Apr-19 23:39:45.941 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 85 ms 
Apr-19 23:39:49.207 [JavaFX Application Thread] DEBUG b.c.d.g.b.BondRepository: update took 59 ms

Thus, there is something around a ten-fold speedup. As can be seen from the logs, update() usually gets called 3 times per incoming block (though this may depend on whether accounting is enabled, which it was on my instance).