indexer: backups are now a deterministic set of SQL statements

vocdoni / vocdoni-node

A set of libraries and tools for the Vocdoni decentralized backend infrastructure, the main ground of our universally verifiable, privacy-centric and scalable digital voting protocol

GNU Affero General Public License v3.0

86 stars 16 forks source link

indexer: backups are now a deterministic set of SQL statements #1338

Open altergui opened 3 months ago

altergui commented 3 months ago

this is much less performant that simply exporting/importing the raw binary database, but makes the backup deterministic, allowing StateSync to pull the snapshot from many nodes at the same time

coveralls commented 3 months ago

Pull Request Test Coverage Report for Build 9498381141

Details

46 of 69 (66.67%) changed or added relevant lines in 2 files are covered.
12 unchanged lines in 5 files lost coverage.
Overall coverage decreased (-0.03%) to 60.918%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
vochain/indexer/indexer.go	44	67	65.67%
<!--	Total:	46	69	66.67%	-->

Files with Coverage Reduction	New Missed Lines	%
cmd/end2endtest/account.go	1	66.43%
api/accounts.go	2	54.42%
apiclient/account.go	2	38.89%
vochain/appsetup.go	2	55.45%
vochain/indexer/indexer.go	5	67.55%
<!--	Total:	12	-->

Totals
Change from base Build 9495839404:	-0.03%
Covered Lines:	16080
Relevant Lines:	26396

💛 - Coveralls

coveralls commented 3 months ago

Pull Request Test Coverage Report for Build 9498587081

Details

40 of 62 (64.52%) changed or added relevant lines in 2 files are covered.
114 unchanged lines in 5 files lost coverage.
Overall coverage decreased (-0.05%) to 60.903%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
vochain/indexer/indexer.go	38	60	63.33%
<!--	Total:	40	62	64.52%	-->

Files with Coverage Reduction	New Missed Lines	%
cmd/end2endtest/account.go	1	66.43%
api/accounts.go	2	54.42%
apiclient/account.go	2	38.89%
vochain/appsetup.go	2	55.45%
vochain/indexer/indexer.go	107	67.55%
<!--	Total:	114	-->

Totals
Change from base Build 9495839404:	-0.05%
Covered Lines:	16076
Relevant Lines:	26396

💛 - Coveralls

coveralls commented 3 months ago

Pull Request Test Coverage Report for Build 9498735272

Details

49 of 77 (63.64%) changed or added relevant lines in 2 files are covered.
8 unchanged lines in 5 files lost coverage.
Overall coverage decreased (-0.03%) to 60.924%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
vochain/indexer/indexer.go	47	75	62.67%
<!--	Total:	49	77	63.64%	-->

Files with Coverage Reduction	New Missed Lines	%
cmd/end2endtest/account.go	1	66.43%
vochain/indexer/indexer.go	1	68.09%
api/accounts.go	2	54.42%
apiclient/account.go	2	38.89%
vochain/appsetup.go	2	55.45%
<!--	Total:	8	-->

Totals
Change from base Build 9495839404:	-0.03%
Covered Lines:	16098
Relevant Lines:	26423

💛 - Coveralls

p4u commented 3 months ago

Could you explain why this is now deterministic? thanks

coveralls commented 3 months ago

Pull Request Test Coverage Report for Build 9546598969

Details

60 of 88 (68.18%) changed or added relevant lines in 3 files are covered.
5 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.02%) to 61.32%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
vochain/indexer/indexer.go	51	79	64.56%
<!--	Total:	60	88	68.18%	-->

Files with Coverage Reduction	New Missed Lines	%
vochain/indexer/indexer.go	5	67.7%
<!--	Total:	5		-->

Totals
Change from base Build 9546446432:	0.02%
Covered Lines:	15939
Relevant Lines:	25993

💛 - Coveralls

altergui commented 3 months ago

it's deterministic because the SQL statements produced by each node are the same (bit-by-bit) . this was not the case with the raw db format.