ryzhyk commented 1 year ago

Automatically download train, test, and inference datasets from the notebook.
Use relative paths to dataset files.
Run the notebook in CI using much smaller datasets.
Add some documentation to the notebook. Will write more as we extend the notebook.

codecov[bot] commented 1 year ago

Codecov Report

Merging #323 (9494a2d) into main (d8b98f4) will increase coverage by 0.00%. The diff coverage is n/a.

Additional details and impacted files

[![Impacted file tree graph](https://codecov.io/gh/vmware/database-stream-processor/pull/323/graphs/tree.svg?width=650&height=150&src=pr&token=0wZcmD11gt&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware)](https://codecov.io/gh/vmware/database-stream-processor/pull/323?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware) ```diff @@ Coverage Diff @@ ## main #323 +/- ## ======================================= Coverage 77.81% 77.81% ======================================= Files 166 166 Lines 32600 32600 ======================================= + Hits 25367 25368 +1 + Misses 7233 7232 -1 ``` | [Impacted Files](https://codecov.io/gh/vmware/database-stream-processor/pull/323?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware) | Coverage Δ | | |---|---|---| | [crates/adapters/src/controller/stats.rs](https://codecov.io/gh/vmware/database-stream-processor/pull/323?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vmware#diff-Y3JhdGVzL2FkYXB0ZXJzL3NyYy9jb250cm9sbGVyL3N0YXRzLnJz) | `86.68% <0.00%> (+0.27%)` | :arrow_up: |

github-actions[bot] commented 1 year ago

Benchmark results

Nexmark

Compared results from d8b98f4 (main) with 9494a2d (PR)

name	main [kOp/s]	PR [kOp/s]	Tput change [%]	Assessment	Peak RSS diff
q0	3917.87	3936.52	0	:heavy_check_mark:	76.6 MB
q1	3961.08	3905.21	-1	:heavy_check_mark:	27.4 MB
q2	3947.23	3938.58	0	:heavy_check_mark:	16.0 MB
q3	3914.4	3939.56	1	:heavy_check_mark:	-15.2 MB
q4	2942.08	2995.02	2	:heavy_check_mark:	-6.8 MB
q5	3892.7	3927.02	1	:heavy_check_mark:	-6.8 MB
q6	3181.85	3141.7	-1	:heavy_check_mark:	-174.9 MB
q7	2692.51	2887.35	7	:evergreen_tree:	-1.1 GB
q8	3779.36	3864.05	2	:heavy_check_mark:	-1.1 GB
q9	591.907	613.806	4	:heavy_check_mark:	37.1 MB
q12	3688.17	3760.9	2	:heavy_check_mark:	37.1 MB
q13	3197.91	3217.16	1	:heavy_check_mark:	37.1 MB
q14	3952.13	3974.69	1	:heavy_check_mark:	37.1 MB
q15	3979.16	3985.68	0	:heavy_check_mark:	37.1 MB
q16	1039.56	1036.41	0	:heavy_check_mark:	37.1 MB
q17	3124.86	3102.7	-1	:heavy_check_mark:	37.1 MB
q18	882.48	901.486	2	:heavy_check_mark:	73.8 MB
q19	982.635	1019.91	4	:heavy_check_mark:	73.8 MB
q20	1099.08	1054.82	-4	:heavy_check_mark:	73.8 MB
q21	3895.77	3900.96	0	:heavy_check_mark:	73.8 MB
q22	3940.57	3911.57	-1	:heavy_check_mark:	73.8 MB

Galen

Compared results from d8b98f4 (main) with 9494a2d (PR)

name	main [s]	PR [s]	Runtime change [%]	Assessment
galen	28.3098	28.0169	-1	:heavy_check_mark:

LDBC

Compared results from d8b98f4 (main) with 9494a2d (PR)

algorithm	dataset	threads	main [kEVPS]	PR [kEVPS]	Tput change [%]	Assessment	Peak RSS diff
bfs	graph500-22	1	1910.53	1905.91	0	:heavy_check_mark:	53.2 kB
bfs	datagen-8_4-fb	6	7924.87	8041.02	1	:heavy_check_mark:	-105.7 MB
pagerank	graph500-22	1	677.401	680.121	0	:heavy_check_mark:	-114.7 kB
pagerank	datagen-8_4-fb	6	1934.06	2018.96	4	:heavy_check_mark:	73.3 MB

Nexmark (with Persistence)

Compared results from d8b98f4 (main) with 9494a2d (PR)

name	main [kOp/s]	PR [kOp/s]	Tput change [%]	PR DRAM [kOp/s]	DRAM diff [%]	Assessment
q0	2440.88	2406.98	-1	2392.84	1	:heavy_check_mark:
q1	1710.69	1734.08	1	1666.52	4	:heavy_check_mark:
q2	2439.29	2436.64	0	2402.69	1	:heavy_check_mark:
q3	2033.27	2016.98	-1	2270.69	-11	:heavy_check_mark:
q4	363.604	362.7	0	1405.29	-74	:heavy_check_mark:
q5	2018.73	1940.67	-4	2271.71	-15	:heavy_check_mark:
q6	337.041	335.058	-1	1386.95	-76	:heavy_check_mark:
q7	258.968	258.296	0	1279.72	-80	:heavy_check_mark:
q8	2139.46	2069.78	-3	2244.29	-8	:heavy_check_mark:
q9	33.3495	33.1093	-1	383.533	-91	:heavy_check_mark:
q12	885.122	878.686	-1	1785.42	-51	:heavy_check_mark:
q13	439.402	444.205	1	924.724	-52	:heavy_check_mark:
q14	1676.11	1713.85	2	1716.2	0	:heavy_check_mark:
q15	199.161	200.805	1	1181.12	-83	:heavy_check_mark:
q16	27.0979	26.7785	-1	287.779	-91	:heavy_check_mark:
q17	83.1951	83.1492	0	760.981	-89	:heavy_check_mark:
q18	61.0489	61.2544	0	791.292	-92	:heavy_check_mark:
q19	68.6324	68.7639	0	655.601	-90	:heavy_check_mark:
q20	169.373	170.216	0	944.115	-82	:heavy_check_mark:
q21	1519.32	1517.99	0	1504.19	1	:heavy_check_mark:
q22	2107.89	2116.52	0	1999.67	6	:heavy_check_mark:

vmware-archive / database-stream-processor