Migrate to sqlx - Githubissues

lalithsuresh commented 1 year ago

Initial lift to migrate the code from rusqlite to sqlx to interface with a database.

Some key changes:

DB methods are all async now
Schema changes to accommodate sqlx quirks
Cannot test a connection before daemonization because tokio does not survive forks

Remaining todos:

Test with postgres. Will require ManagerConfig to be populated with connection strings directly.
More tests for db.rs.
Use proptests?

github-actions[bot] commented 1 year ago

Benchmark results

Nexmark

5 out of 21 queries have regressed :exclamation:
Compared results from 0705657 (main) with d276a1e (PR) No benchmark results found for current main revision, compared against 0705657fd7a925826d1d5b2681157087e13001b8

name	main~6 [kOp/s]	PR [kOp/s]	Tput change [%]	Assessment	Peak RSS diff
q0	5627.91	5463.2	-3	:heavy_check_mark:	7.2 MB
q1	4863.2	5147.95	6	:evergreen_tree:	6.9 MB
q2	5642.4	4541.87	-20	:small_red_triangle_down:	26.1 MB
q3	5505.84	5371.01	-2	:heavy_check_mark:	2.7 MB
q4	3427.48	3264.81	-5	:heavy_check_mark:	-17.2 MB
q5	5462.96	4220.8	-23	:small_red_triangle_down:	-17.2 MB
q6	3336.49	3616.1	8	:evergreen_tree:	-119.8 MB
q7	2973.22	3175.19	7	:evergreen_tree:	4.3 GB
q8	3971.36	5259.06	32	:evergreen_tree:	4.3 GB
q9	601.186	616.08	2	:heavy_check_mark:	134.3 MB
q12	4520.29	4742.33	5	:heavy_check_mark:	134.3 MB
q13	3512.47	3272.98	-7	:small_red_triangle_down:	134.3 MB
q14	5671.55	4461.76	-21	:small_red_triangle_down:	134.3 MB
q15	4922.14	4951.24	1	:heavy_check_mark:	134.3 MB
q16	1041.26	1033.34	-1	:heavy_check_mark:	134.3 MB
q17	2929.37	3113.77	6	:evergreen_tree:	134.3 MB
q18	858.772	879.646	2	:heavy_check_mark:	42.8 MB
q19	1030.12	996.163	-3	:heavy_check_mark:	42.8 MB
q20	1070.56	1068.71	0	:heavy_check_mark:	42.8 MB
q21	5056.51	4608.02	-9	:small_red_triangle_down:	42.8 MB
q22	4532.23	4383.47	-3	:heavy_check_mark:	42.8 MB

Galen

Compared results from 0705657 (main) with d276a1e (PR) No benchmark results found for current main revision, compared against 0705657fd7a925826d1d5b2681157087e13001b8

name	main~6 [s]	PR [s]	Runtime change [%]	Assessment
galen	27.3996	28.6334	5	:heavy_check_mark:

LDBC

2 out of 4 queries have regressed :exclamation:
Compared results from 0705657 (main) with d276a1e (PR) No benchmark results found for current main revision, compared against 0705657fd7a925826d1d5b2681157087e13001b8

algorithm	dataset	threads	main~6 [kEVPS]	PR [kEVPS]	Tput change [%]	Assessment	Peak RSS diff
bfs	graph500-22	1	1748.77	1758.72	1	:heavy_check_mark:	139.3 kB
bfs	datagen-8_4-fb	6	7583.63	6719.91	-11	:small_red_triangle_down:	-169.4 MB
pagerank	graph500-22	1	692.573	685.634	-1	:heavy_check_mark:	12.3 kB
pagerank	datagen-8_4-fb	6	2043.84	1729.64	-15	:small_red_triangle_down:	-306.5 MB

Nexmark (with Persistence)

Compared results from 0705657 (main) with d276a1e (PR) No benchmark results found for current main revision, compared against 0705657fd7a925826d1d5b2681157087e13001b8

name	main~6 [kOp/s]	PR [kOp/s]	Tput change [%]	PR DRAM [kOp/s]	DRAM diff [%]	Assessment
q0	2466.27	2411.09	-2	2407.51	0	:heavy_check_mark:
q1	1733.5	1705.52	-2	1710.43	0	:heavy_check_mark:
q2	2425.78	2347.35	-3	2372.76	-1	:heavy_check_mark:
q3	2063.22	1984.47	-4	2265.62	-12	:heavy_check_mark:
q4	361.595	363.199	0	1402.01	-74	:heavy_check_mark:
q5	2046.01	2030.82	-1	2256.95	-10	:heavy_check_mark:
q6	338.322	338.908	0	1362.08	-75	:heavy_check_mark:
q7	256.483	254.164	-1	1271.04	-80	:heavy_check_mark:
q8	2159.79	2092.23	-3	2204.41	-5	:heavy_check_mark:
q9	33.0534	33.1384	0	386.457	-91	:heavy_check_mark:
q12	845.838	844.968	0	1784.92	-53	:heavy_check_mark:
q13	439.336	439.348	0	994.879	-56	:heavy_check_mark:
q14	1725.8	1659.61	-4	1679.43	-1	:heavy_check_mark:
q15	197.558	196.216	-1	1171.86	-83	:heavy_check_mark:
q16	26.7153	27.0341	1	287.833	-91	:heavy_check_mark:
q17	81.3879	81.2307	0	801.52	-90	:heavy_check_mark:
q18	61.1309	60.4636	-1	801.412	-92	:heavy_check_mark:
q19	67.9718	67.7252	0	655.82	-90	:heavy_check_mark:
q20	169.329	167.895	-1	928.834	-82	:heavy_check_mark:
q21	1524.07	1490.97	-2	1479.27	1	:heavy_check_mark:
q22	2108.78	2066.42	-2	1992.49	4	:heavy_check_mark:

codecov[bot] commented 1 year ago

Codecov Report

Merging #342 (1522793) into main (0705657) will increase coverage by 1.57%. The diff coverage is 79.29%.

:exclamation: Current head 1522793 differs from pull request most recent head d276a1e. Consider uploading reports for the commit d276a1e to get more accurate results

Additional details and impacted files

[![Impacted file tree graph](https://codecov.io/gh/vmware/database-stream-processor/pull/342/graphs/tree.svg?width=650&height=150&src=pr&token=0wZcmD11gt&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware)](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware) ```diff @@ Coverage Diff @@ ## main #342 +/- ## ========================================== + Coverage 77.79% 79.37% +1.57% ========================================== Files 166 167 +1 Lines 32747 33513 +766 ========================================== + Hits 25475 26600 +1125 + Misses 7272 6913 -359 ``` | [Impacted Files](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware) | Coverage Δ | | |---|---|---| | [crates/dbsp/src/operator/time\_series/window.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL29wZXJhdG9yL3RpbWVfc2VyaWVzL3dpbmRvdy5ycw==) | `98.62% <ø> (ø)` | | | [crates/dbsp/src/operator/trace.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL29wZXJhdG9yL3RyYWNlLnJz) | `84.55% <ø> (ø)` | | | [crates/dbsp/src/trace/layers/unordered.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL3RyYWNlL2xheWVycy91bm9yZGVyZWQucnM=) | `0.00% <0.00%> (ø)` | | | [crates/pipeline\_manager/src/compiler.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL3BpcGVsaW5lX21hbmFnZXIvc3JjL2NvbXBpbGVyLnJz) | `0.42% <0.00%> (+0.42%)` | :arrow_up: | | [crates/pipeline\_manager/src/config.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL3BpcGVsaW5lX21hbmFnZXIvc3JjL2NvbmZpZy5ycw==) | `0.00% <0.00%> (ø)` | | | [crates/pipeline\_manager/src/main.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL3BpcGVsaW5lX21hbmFnZXIvc3JjL21haW4ucnM=) | `0.11% <0.00%> (-0.01%)` | :arrow_down: | | [crates/pipeline\_manager/src/runner.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL3BpcGVsaW5lX21hbmFnZXIvc3JjL3J1bm5lci5ycw==) | `0.00% <0.00%> (ø)` | | | [crates/dbsp/src/trace/layers/mod.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL3RyYWNlL2xheWVycy9tb2QucnM=) | `35.71% <20.00%> (+13.21%)` | :arrow_up: | | [crates/pipeline\_manager/src/db.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL3BpcGVsaW5lX21hbmFnZXIvc3JjL2RiLnJz) | `53.31% <63.25%> (+53.31%)` | :arrow_up: | | [crates/dbsp/src/trace/layers/ordered\_leaf.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2Ric3Avc3JjL3RyYWNlL2xheWVycy9vcmRlcmVkX2xlYWYucnM=) | `57.89% <76.19%> (+57.89%)` | :arrow_up: | | ... and [5 more](https://codecov.io/gh/vmware/database-stream-processor/pull/342?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware) | | ... and [2 files with indirect coverage changes](https://codecov.io/gh/vmware/database-stream-processor/pull/342/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware)

vmware-archive / database-stream-processor