search

questdb / net-questdb-client

Apache License 2.0
34 stars 7 forks source link

feat: support for ILP/HTTP #18

Closed nwoolmer closed 7 months ago

nwoolmer commented 8 months ago

Closes #16 Closes #9

Overview

This PR upgrades the client to support ILP over HTTP transport for data sent to QuestDB. This protocol provides a better dev-ex for data ingestion, including improved error messages, initialisation via configuration string, authentication, transactions, auto-flushing, and more.

Benchmarks

Varying connection limit against real server In this case, multiple WALs are generated, so its best to enforce 1 connection per sender to avoid this. ``` BenchmarkDotNet v0.13.12, Windows 10 (10.0.19045.4170/22H2/2022Update) AMD Ryzen 7 3700X, 1 CPU, 16 logical and 8 physical cores .NET SDK 8.0.201 [Host] : .NET 6.0.9 (6.0.922.41905), X64 RyuJIT AVX2 Job=MediumRun Toolchain=InProcessNoEmitToolchain IterationCount=15 LaunchCount=1 WarmupCount=10 ``` | Method | rows_per_iteration | batch_size | number_of_tables | connection_limit | Mean | Error | StdDev | |------------------------ |------------------- |----------- |----------------- |----------------- |------------:|----------:|---------:| | **HttpRandomTableEveryRow** | **100000** | **5000** | **10** | **1** | **2,152.15 ms** | **5.757 ms** | **5.103 ms** | | TcpRandomTableEveryRow | 100000 | 5000 | 10 | 1 | 42.29 ms | 0.688 ms | 0.537 ms | | **HttpRandomTableEveryRow** | **100000** | **5000** | **10** | **4** | **2,157.63 ms** | **8.333 ms** | **7.795 ms** | | TcpRandomTableEveryRow | 100000 | 5000 | 10 | 4 | 43.09 ms | 2.051 ms | 1.713 ms | | **HttpRandomTableEveryRow** | **100000** | **5000** | **10** | **16** | **2,156.78 ms** | **10.562 ms** | **9.880 ms** | | TcpRandomTableEveryRow | 100000 | 5000 | 10 | 16 | 43.94 ms | 2.831 ms | 2.210 ms |
Varying connection limit with fixed batch size, rows, and tables, against real server Using a single sender with sequential writes and awaiting results. ``` BenchmarkDotNet v0.13.12, macOS Sonoma 14.0 (23A344) [Darwin 23.0.0] Apple M2, 1 CPU, 8 logical and 8 physical cores .NET SDK 6.0.125 [Host] : .NET 6.0.25 (6.0.2523.56901), Arm64 RyuJIT AdvSIMD DEBUG Job=MediumRun Toolchain=InProcessNoEmitToolchain IterationCount=15 LaunchCount=1 WarmupCount=10 ``` | Method | rows_per_iteration | batch_size | number_of_tables | connection_limit | Mean | Error | StdDev | |------------------------ |------------------- |----------- |----------------- |----------------- |--------:|---------:|---------:| | **HttpRandomTableEveryRow** | **10000000** | **75000** | **3** | **1** | **7.964 s** | **0.0444 s** | **0.0371 s** | | TcpRandomTableEveryRow | 10000000 | 75000 | 3 | 1 | 6.174 s | 0.0118 s | 0.0098 s | | **HttpRandomTableEveryRow** | **10000000** | **75000** | **3** | **2** | **8.046 s** | **0.1056 s** | **0.0988 s** | | TcpRandomTableEveryRow | 10000000 | 75000 | 3 | 2 | 6.211 s | 0.0598 s | 0.0530 s | | **HttpRandomTableEveryRow** | **10000000** | **75000** | **3** | **4** | **8.052 s** | **0.1389 s** | **0.1231 s** | | TcpRandomTableEveryRow | 10000000 | 75000 | 3 | 4 | 6.204 s | 0.0468 s | 0.0391 s | | **HttpRandomTableEveryRow** | **10000000** | **75000** | **3** | **8** | **8.031 s** | **0.1359 s** | **0.1272 s** | | TcpRandomTableEveryRow | 10000000 | 75000 | 3 | 8 | 6.245 s | 0.1674 s | 0.1307 s | | **HttpRandomTableEveryRow** | **10000000** | **75000** | **3** | **16** | **8.148 s** | **0.1037 s** | **0.0919 s** | | TcpRandomTableEveryRow | 10000000 | 75000 | 3 | 16 | 6.237 s | 0.0654 s | 0.0579 s |
Varying connections and tables against dummy server ``` BenchmarkDotNet v0.13.12, macOS Sonoma 14.0 (23A344) [Darwin 23.0.0] Apple M2, 1 CPU, 8 logical and 8 physical cores .NET SDK 6.0.125 [Host] : .NET 6.0.25 (6.0.2523.56901), Arm64 RyuJIT AdvSIMD DEBUG Job=MediumRun Toolchain=InProcessNoEmitToolchain IterationCount=15 LaunchCount=1 WarmupCount=10 ``` | Method | rows_per_iteration | batch_size | number_of_tables | connection_limit | Mean | Error | StdDev | |-------------------- |------------------- |----------- |----------------- |----------------- |-----------:|------------:|------------:| | **RandomTableEveryRow** | **10000** | **1000** | **1** | **1** | **8.634 ms** | **0.3652 ms** | **0.3237 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **1** | **2** | **8.254 ms** | **0.0248 ms** | **0.0219 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **1** | **4** | **8.194 ms** | **0.0319 ms** | **0.0283 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **1** | **8** | **8.194 ms** | **0.0243 ms** | **0.0228 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **1** | **16** | **8.195 ms** | **0.0229 ms** | **0.0214 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **2** | **1** | **9.337 ms** | **1.1715 ms** | **1.0958 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **2** | **2** | **8.423 ms** | **0.1780 ms** | **0.1665 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **2** | **4** | **8.237 ms** | **0.0452 ms** | **0.0423 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **2** | **8** | **8.400 ms** | **0.1446 ms** | **0.1352 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **2** | **16** | **8.237 ms** | **0.0789 ms** | **0.0738 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **4** | **1** | **8.419 ms** | **0.1201 ms** | **0.1065 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **4** | **2** | **8.278 ms** | **0.0559 ms** | **0.0496 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **4** | **4** | **8.331 ms** | **0.1742 ms** | **0.1455 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **4** | **8** | **8.690 ms** | **0.1810 ms** | **0.1693 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **4** | **16** | **8.558 ms** | **0.2130 ms** | **0.1993 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **8** | **1** | **8.331 ms** | **0.1173 ms** | **0.1098 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **8** | **2** | **8.599 ms** | **0.2219 ms** | **0.2075 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **8** | **4** | **8.303 ms** | **0.0786 ms** | **0.0735 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **8** | **8** | **8.670 ms** | **0.3009 ms** | **0.2814 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **8** | **16** | **8.496 ms** | **0.2513 ms** | **0.2350 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **16** | **1** | **8.737 ms** | **0.2033 ms** | **0.1902 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **16** | **2** | **8.726 ms** | **0.2537 ms** | **0.2373 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **16** | **4** | **8.682 ms** | **0.1651 ms** | **0.1544 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **16** | **8** | **8.363 ms** | **0.0414 ms** | **0.0324 ms** | | **RandomTableEveryRow** | **10000** | **1000** | **16** | **16** | **8.756 ms** | **0.2144 ms** | **0.2006 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **1** | **1** | **7.667 ms** | **0.1552 ms** | **0.1376 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **1** | **2** | **7.651 ms** | **0.0869 ms** | **0.0770 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **1** | **4** | **7.743 ms** | **0.1195 ms** | **0.1118 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **1** | **8** | **7.628 ms** | **0.0978 ms** | **0.0915 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **1** | **16** | **7.701 ms** | **0.1121 ms** | **0.1049 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **2** | **1** | **7.607 ms** | **0.1226 ms** | **0.1147 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **2** | **2** | **7.677 ms** | **0.1236 ms** | **0.1095 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **2** | **4** | **7.673 ms** | **0.1446 ms** | **0.1353 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **2** | **8** | **7.678 ms** | **0.1019 ms** | **0.0953 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **2** | **16** | **7.703 ms** | **0.0833 ms** | **0.0650 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **4** | **1** | **7.677 ms** | **0.0935 ms** | **0.0875 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **4** | **2** | **7.632 ms** | **0.1327 ms** | **0.1241 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **4** | **4** | **7.733 ms** | **0.1504 ms** | **0.1333 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **4** | **8** | **7.704 ms** | **0.1325 ms** | **0.1240 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **4** | **16** | **7.713 ms** | **0.1030 ms** | **0.0964 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **8** | **1** | **7.720 ms** | **0.1497 ms** | **0.1400 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **8** | **2** | **7.758 ms** | **0.1446 ms** | **0.1207 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **8** | **4** | **7.597 ms** | **0.0627 ms** | **0.0587 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **8** | **8** | **7.676 ms** | **0.1246 ms** | **0.1104 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **8** | **16** | **7.611 ms** | **0.0874 ms** | **0.0775 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **16** | **1** | **7.715 ms** | **0.0355 ms** | **0.0315 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **16** | **2** | **7.624 ms** | **0.2089 ms** | **0.1745 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **16** | **4** | **7.643 ms** | **0.0363 ms** | **0.0322 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **16** | **8** | **7.683 ms** | **0.0470 ms** | **0.0417 ms** | | **RandomTableEveryRow** | **10000** | **10000** | **16** | **16** | **7.661 ms** | **0.0370 ms** | **0.0328 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **1** | **1** | **7.493 ms** | **0.0391 ms** | **0.0346 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **1** | **2** | **7.459 ms** | **0.0371 ms** | **0.0310 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **1** | **4** | **7.572 ms** | **0.0655 ms** | **0.0581 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **1** | **8** | **7.470 ms** | **0.0557 ms** | **0.0465 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **1** | **16** | **7.637 ms** | **0.1999 ms** | **0.1772 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **2** | **1** | **7.431 ms** | **0.0493 ms** | **0.0461 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **2** | **2** | **7.469 ms** | **0.0227 ms** | **0.0212 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **2** | **4** | **7.660 ms** | **0.0955 ms** | **0.0797 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **2** | **8** | **7.551 ms** | **0.0590 ms** | **0.0523 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **2** | **16** | **7.637 ms** | **0.1327 ms** | **0.1108 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **4** | **1** | **7.518 ms** | **0.0296 ms** | **0.0277 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **4** | **2** | **7.714 ms** | **0.1792 ms** | **0.1676 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **4** | **4** | **8.095 ms** | **0.6229 ms** | **0.5827 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **4** | **8** | **7.572 ms** | **0.0599 ms** | **0.0500 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **4** | **16** | **7.683 ms** | **0.0733 ms** | **0.0685 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **8** | **1** | **7.567 ms** | **0.0172 ms** | **0.0143 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **8** | **2** | **7.677 ms** | **0.1433 ms** | **0.1119 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **8** | **4** | **7.641 ms** | **0.1042 ms** | **0.0975 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **8** | **8** | **7.690 ms** | **0.1194 ms** | **0.1117 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **8** | **16** | **7.627 ms** | **0.1080 ms** | **0.1010 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **16** | **1** | **7.649 ms** | **0.0355 ms** | **0.0332 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **16** | **2** | **7.700 ms** | **0.0527 ms** | **0.0493 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **16** | **4** | **7.963 ms** | **0.0875 ms** | **0.0819 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **16** | **8** | **7.798 ms** | **0.1916 ms** | **0.1792 ms** | | **RandomTableEveryRow** | **10000** | **100000** | **16** | **16** | **7.862 ms** | **0.1673 ms** | **0.1565 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **1** | **1** | **80.558 ms** | **1.5264 ms** | **1.4278 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **1** | **2** | **81.165 ms** | **0.7973 ms** | **0.7458 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **1** | **4** | **79.784 ms** | **1.7330 ms** | **1.6210 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **1** | **8** | **79.856 ms** | **2.0942 ms** | **1.9589 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **1** | **16** | **81.402 ms** | **1.6310 ms** | **1.5256 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **2** | **1** | **78.174 ms** | **1.5686 ms** | **1.3905 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **2** | **2** | **79.054 ms** | **1.3144 ms** | **1.2295 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **2** | **4** | **79.350 ms** | **1.7068 ms** | **1.5965 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **2** | **8** | **78.024 ms** | **1.0377 ms** | **0.8665 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **2** | **16** | **81.530 ms** | **1.5431 ms** | **1.4434 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **4** | **1** | **80.155 ms** | **1.4791 ms** | **1.3835 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **4** | **2** | **79.707 ms** | **2.4151 ms** | **2.2590 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **4** | **4** | **80.178 ms** | **1.7416 ms** | **1.6291 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **4** | **8** | **81.325 ms** | **1.8048 ms** | **1.6882 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **4** | **16** | **79.007 ms** | **1.0846 ms** | **1.0145 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **8** | **1** | **79.001 ms** | **0.9274 ms** | **0.8221 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **8** | **2** | **82.067 ms** | **0.9731 ms** | **0.9102 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **8** | **4** | **79.130 ms** | **0.8382 ms** | **0.6544 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **8** | **8** | **80.084 ms** | **1.4176 ms** | **1.3260 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **8** | **16** | **80.749 ms** | **1.9880 ms** | **1.8596 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **16** | **1** | **80.523 ms** | **2.5823 ms** | **2.4155 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **16** | **2** | **80.677 ms** | **2.2088 ms** | **2.0661 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **16** | **4** | **81.563 ms** | **2.3916 ms** | **2.2371 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **16** | **8** | **81.732 ms** | **2.4997 ms** | **2.3382 ms** | | **RandomTableEveryRow** | **100000** | **1000** | **16** | **16** | **83.265 ms** | **1.4023 ms** | **1.2431 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **1** | **1** | **71.547 ms** | **1.0776 ms** | **0.9553 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **1** | **2** | **70.822 ms** | **0.8753 ms** | **0.7759 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **1** | **4** | **71.532 ms** | **1.2315 ms** | **1.1520 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **1** | **8** | **70.926 ms** | **0.5101 ms** | **0.4260 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **1** | **16** | **70.125 ms** | **0.5067 ms** | **0.4491 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **2** | **1** | **71.919 ms** | **0.9320 ms** | **0.8718 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **2** | **2** | **71.685 ms** | **1.1961 ms** | **1.1188 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **2** | **4** | **71.295 ms** | **1.2305 ms** | **1.0908 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **2** | **8** | **70.484 ms** | **0.1624 ms** | **0.1268 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **2** | **16** | **71.058 ms** | **1.3729 ms** | **1.0718 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **4** | **1** | **73.586 ms** | **2.9665 ms** | **2.7748 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **4** | **2** | **70.981 ms** | **0.5283 ms** | **0.4411 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **4** | **4** | **72.447 ms** | **0.9895 ms** | **0.9256 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **4** | **8** | **71.838 ms** | **0.4742 ms** | **0.3702 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **4** | **16** | **72.348 ms** | **1.0001 ms** | **0.9355 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **8** | **1** | **71.885 ms** | **0.8948 ms** | **0.8370 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **8** | **2** | **70.638 ms** | **0.2892 ms** | **0.2564 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **8** | **4** | **72.694 ms** | **1.1963 ms** | **0.9990 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **8** | **8** | **71.730 ms** | **0.9180 ms** | **0.8138 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **8** | **16** | **72.005 ms** | **0.5961 ms** | **0.5576 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **16** | **1** | **70.315 ms** | **1.3413 ms** | **1.1201 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **16** | **2** | **70.235 ms** | **0.8971 ms** | **0.8392 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **16** | **4** | **68.976 ms** | **0.1972 ms** | **0.1647 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **16** | **8** | **70.534 ms** | **0.7061 ms** | **0.6604 ms** | | **RandomTableEveryRow** | **100000** | **10000** | **16** | **16** | **70.229 ms** | **0.9102 ms** | **0.8514 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **1** | **1** | **65.841 ms** | **0.4531 ms** | **0.4238 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **1** | **2** | **65.946 ms** | **0.8886 ms** | **0.8312 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **1** | **4** | **66.046 ms** | **0.5703 ms** | **0.5055 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **1** | **8** | **65.168 ms** | **0.4889 ms** | **0.4573 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **1** | **16** | **65.164 ms** | **0.5965 ms** | **0.5580 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **2** | **1** | **65.244 ms** | **0.4711 ms** | **0.4177 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **2** | **2** | **65.827 ms** | **0.6749 ms** | **0.6313 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **2** | **4** | **65.683 ms** | **0.3051 ms** | **0.2705 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **2** | **8** | **66.027 ms** | **0.4698 ms** | **0.4395 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **2** | **16** | **66.183 ms** | **0.5623 ms** | **0.4984 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **4** | **1** | **65.212 ms** | **0.4348 ms** | **0.4067 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **4** | **2** | **65.802 ms** | **0.6815 ms** | **0.6375 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **4** | **4** | **65.252 ms** | **0.5579 ms** | **0.4658 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **4** | **8** | **66.687 ms** | **0.3938 ms** | **0.3684 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **4** | **16** | **65.934 ms** | **0.5378 ms** | **0.5031 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **8** | **1** | **66.092 ms** | **0.6074 ms** | **0.5681 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **8** | **2** | **65.702 ms** | **0.5488 ms** | **0.5134 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **8** | **4** | **66.059 ms** | **0.5166 ms** | **0.4832 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **8** | **8** | **65.791 ms** | **0.5608 ms** | **0.4971 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **8** | **16** | **65.748 ms** | **0.5311 ms** | **0.4708 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **16** | **1** | **66.505 ms** | **0.5098 ms** | **0.4768 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **16** | **2** | **66.760 ms** | **0.8419 ms** | **0.7875 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **16** | **4** | **66.468 ms** | **0.5752 ms** | **0.5099 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **16** | **8** | **66.801 ms** | **0.5124 ms** | **0.4793 ms** | | **RandomTableEveryRow** | **100000** | **100000** | **16** | **16** | **66.156 ms** | **0.5230 ms** | **0.4892 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **1** | **1** | **840.358 ms** | **65.7567 ms** | **61.5088 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **1** | **2** | **783.492 ms** | **14.5843 ms** | **12.1786 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **1** | **4** | **803.321 ms** | **17.0362 ms** | **15.9356 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **1** | **8** | **795.490 ms** | **13.2974 ms** | **12.4384 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **1** | **16** | **813.059 ms** | **17.9520 ms** | **16.7923 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **2** | **1** | **800.169 ms** | **15.0138 ms** | **12.5372 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **2** | **2** | **800.964 ms** | **19.2593 ms** | **17.0729 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **2** | **4** | **799.552 ms** | **16.0019 ms** | **14.1853 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **2** | **8** | **821.563 ms** | **25.2905 ms** | **22.4194 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **2** | **16** | **829.631 ms** | **53.1803 ms** | **49.7449 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **4** | **1** | **789.513 ms** | **7.4499 ms** | **6.2210 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **4** | **2** | **794.921 ms** | **31.3110 ms** | **26.1461 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **4** | **4** | **797.826 ms** | **19.0048 ms** | **14.8377 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **4** | **8** | **793.093 ms** | **25.0755 ms** | **19.5773 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **4** | **16** | **903.927 ms** | **139.3332 ms** | **130.3324 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **8** | **1** | **856.273 ms** | **104.7013 ms** | **92.8150 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **8** | **2** | **803.342 ms** | **11.2366 ms** | **10.5107 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **8** | **4** | **830.300 ms** | **53.6882 ms** | **44.8321 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **8** | **8** | **800.295 ms** | **30.3684 ms** | **23.7097 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **8** | **16** | **812.827 ms** | **33.1155 ms** | **27.6529 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **16** | **1** | **800.446 ms** | **16.0423 ms** | **14.2210 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **16** | **2** | **808.606 ms** | **28.2774 ms** | **22.0771 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **16** | **4** | **787.712 ms** | **2.7555 ms** | **2.3010 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **16** | **8** | **788.060 ms** | **5.3892 ms** | **4.5002 ms** | | **RandomTableEveryRow** | **1000000** | **1000** | **16** | **16** | **791.254 ms** | **7.2115 ms** | **6.7456 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **1** | **1** | **671.156 ms** | **5.8329 ms** | **4.5539 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **1** | **2** | **664.939 ms** | **1.9321 ms** | **1.7128 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **1** | **4** | **663.865 ms** | **1.3863 ms** | **1.1577 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **1** | **8** | **663.892 ms** | **0.9830 ms** | **0.9195 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **1** | **16** | **664.851 ms** | **1.0979 ms** | **0.8571 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **2** | **1** | **665.160 ms** | **1.1686 ms** | **1.0359 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **2** | **2** | **669.873 ms** | **13.9593 ms** | **10.8985 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **2** | **4** | **669.546 ms** | **5.2731 ms** | **4.6745 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **2** | **8** | **666.316 ms** | **1.2539 ms** | **1.1116 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **2** | **16** | **666.483 ms** | **1.1345 ms** | **1.0612 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **4** | **1** | **670.174 ms** | **1.8348 ms** | **1.5321 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **4** | **2** | **670.785 ms** | **2.1819 ms** | **1.9342 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **4** | **4** | **671.048 ms** | **1.1605 ms** | **1.0288 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **4** | **8** | **669.585 ms** | **0.7595 ms** | **0.7104 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **4** | **16** | **668.804 ms** | **0.5490 ms** | **0.4584 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **8** | **1** | **677.688 ms** | **7.9715 ms** | **7.0666 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **8** | **2** | **684.086 ms** | **4.4642 ms** | **3.9574 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **8** | **4** | **680.647 ms** | **7.2306 ms** | **6.0379 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **8** | **8** | **675.138 ms** | **3.2897 ms** | **2.5684 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **8** | **16** | **673.942 ms** | **6.6120 ms** | **6.1849 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **16** | **1** | **700.365 ms** | **9.8537 ms** | **9.2171 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **16** | **2** | **687.272 ms** | **4.9944 ms** | **4.6717 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **16** | **4** | **697.751 ms** | **11.4408 ms** | **10.1420 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **16** | **8** | **688.149 ms** | **5.1927 ms** | **4.8573 ms** | | **RandomTableEveryRow** | **1000000** | **10000** | **16** | **16** | **691.488 ms** | **3.4607 ms** | **3.0678 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **1** | **1** | **654.724 ms** | **6.5186 ms** | **5.4433 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **1** | **2** | **661.519 ms** | **7.9292 ms** | **7.4169 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **1** | **4** | **646.211 ms** | **2.7854 ms** | **2.6055 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **1** | **8** | **650.706 ms** | **2.9705 ms** | **2.7786 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **1** | **16** | **654.595 ms** | **10.2656 ms** | **9.1002 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **2** | **1** | **649.656 ms** | **3.0622 ms** | **2.7145 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **2** | **2** | **646.317 ms** | **1.3116 ms** | **1.2268 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **2** | **4** | **645.951 ms** | **0.9173 ms** | **0.7660 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **2** | **8** | **646.418 ms** | **1.7576 ms** | **1.6441 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **2** | **16** | **645.517 ms** | **1.3447 ms** | **1.2578 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **4** | **1** | **649.716 ms** | **1.5769 ms** | **1.4750 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **4** | **2** | **649.128 ms** | **1.4118 ms** | **1.3206 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **4** | **4** | **648.522 ms** | **1.1153 ms** | **0.9314 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **4** | **8** | **648.860 ms** | **1.2574 ms** | **1.1762 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **4** | **16** | **648.440 ms** | **1.3486 ms** | **1.2615 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **8** | **1** | **646.729 ms** | **1.2878 ms** | **1.0754 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **8** | **2** | **647.494 ms** | **1.5458 ms** | **1.4459 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **8** | **4** | **647.720 ms** | **1.4084 ms** | **1.3174 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **8** | **8** | **648.670 ms** | **1.3269 ms** | **1.1763 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **8** | **16** | **648.707 ms** | **1.5742 ms** | **1.4725 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **16** | **1** | **660.171 ms** | **1.2378 ms** | **1.0973 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **16** | **2** | **658.851 ms** | **0.8582 ms** | **0.7607 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **16** | **4** | **658.427 ms** | **0.8289 ms** | **0.6921 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **16** | **8** | **658.476 ms** | **1.7586 ms** | **1.6450 ms** | | **RandomTableEveryRow** | **1000000** | **100000** | **16** | **16** | **657.832 ms** | **0.9608 ms** | **0.8987 ms** |
Basic HTTP inserts against dummy server ``` BenchmarkDotNet v0.13.12, macOS Sonoma 14.0 (23A344) [Darwin 23.0.0] Apple M2, 1 CPU, 8 logical and 8 physical cores .NET SDK 6.0.125 [Host] : .NET 6.0.25 (6.0.2523.56901), Arm64 RyuJIT AdvSIMD Job=MediumRun Toolchain=InProcessNoEmitToolchain IterationCount=15 LaunchCount=1 WarmupCount=10 ``` | Method | rows_per_iteration | batch_size | Mean | Error | StdDev | |----------------- |------------------- |----------- |-------------:|------------:|------------:| | **BasicInsertsHttp** | **10000** | **1000** | **3.726 ms** | **0.0254 ms** | **0.0237 ms** | | **BasicInsertsHttp** | **10000** | **10000** | **2.987 ms** | **0.0208 ms** | **0.0195 ms** | | **BasicInsertsHttp** | **10000** | **25000** | **3.182 ms** | **0.2134 ms** | **0.1996 ms** | | **BasicInsertsHttp** | **10000** | **75000** | **3.036 ms** | **0.0761 ms** | **0.0675 ms** | | **BasicInsertsHttp** | **10000** | **100000** | **2.929 ms** | **0.0210 ms** | **0.0186 ms** | | **BasicInsertsHttp** | **100000** | **1000** | **31.423 ms** | **0.1254 ms** | **0.1112 ms** | | **BasicInsertsHttp** | **100000** | **10000** | **24.084 ms** | **0.3213 ms** | **0.2848 ms** | | **BasicInsertsHttp** | **100000** | **25000** | **19.977 ms** | **0.4273 ms** | **0.3788 ms** | | **BasicInsertsHttp** | **100000** | **75000** | **19.388 ms** | **0.4284 ms** | **0.3578 ms** | | **BasicInsertsHttp** | **100000** | **100000** | **19.351 ms** | **0.2408 ms** | **0.2135 ms** | | **BasicInsertsHttp** | **1000000** | **1000** | **310.226 ms** | **1.3325 ms** | **1.1813 ms** | | **BasicInsertsHttp** | **1000000** | **10000** | **205.236 ms** | **1.8343 ms** | **1.7158 ms** | | **BasicInsertsHttp** | **1000000** | **25000** | **196.102 ms** | **2.0113 ms** | **1.7830 ms** | | **BasicInsertsHttp** | **1000000** | **75000** | **188.413 ms** | **1.9675 ms** | **1.5361 ms** | | **BasicInsertsHttp** | **1000000** | **100000** | **181.678 ms** | **1.1773 ms** | **1.1012 ms** | | **BasicInsertsHttp** | **10000000** | **1000** | **3,285.702 ms** | **300.2922 ms** | **280.8935 ms** | | **BasicInsertsHttp** | **10000000** | **10000** | **2,034.505 ms** | **10.0566 ms** | **8.9149 ms** | | **BasicInsertsHttp** | **10000000** | **25000** | **1,848.660 ms** | **8.3648 ms** | **6.9850 ms** | | **BasicInsertsHttp** | **10000000** | **75000** | **1,810.437 ms** | **5.1005 ms** | **4.5215 ms** | | **BasicInsertsHttp** | **10000000** | **100000** | **1,822.474 ms** | **6.4584 ms** | **5.7252 ms** |

Features

Feature list, compacted - [x] Initialisation via [conf string](https://github.com/questdb/questdb-confstr-rs). - [x] Initialisation via options pattern. - [x] Submission of data to QuestDB `/write` endpoint. - [x] Authentication support - [x] Support auth timeout limit (`auth_timeout`) - [x] Basic - [x] `username`, `password` - [x] Token - [x] `token` - [x] Tcp Jwk - [x] TLS - [x] `tls_verify` - [ ] ~`tls_ca`~ - [x] `tls_roots` - [x] `tls_roots_password` - [x] Unsupported: `token_x`, `token_y` - [x] Auto-flush support (`auto_flush`) - [x] By number of rows (`auto_flush_rows`) - [x] By number of bytes (`auto_flush_bytes`) - [x] By elapsed time (`auto_flush_interval`) - [x] Transaction support - [x] Single table only - [x] Table name is implicit after transaction opened - [x] Auto flush won't run inside transactions - [ ] ~Bind interface support (`bind_interface`)~ - [x] Buffer sizes: - [x] Specify starting buffer size (`init_buf_size`) - [x] Enforce maximum buffer size (`max_buf_size`) - [x] HTTP Request settings - [x] Support dynamic timeout - [x] Extra request time based on buffer length (`request_min_throughput`) - [x] Base request timeout (`request_timeout`) - [x] Retrying - [x] Support standard set of returnable errors - [x] Support specified period for retries (`retry_timeout`) - [x] Additional symbol/column functions that take nullable parameters and noop if null. - [x] Revised examples - [x] Config binding

Testing

Feedback

- [x] Refactor into interface - [x] Drop interface implementations from Buffer. - [x] Reimplement validation to drop dependency. - [ ] Refactor SendBuffer usage (partial) - [x] Make sure Dispose is complete. - [x] Bind/link cancellation tokens in Http send. - [x] General code cleanup - [x] Coverage check - 91% - [x] Profile - Sawtooth memory usage in streaming case, most collections stay gen 0 - Most time spent encoding UTF-8 into the buffer.

Docs

Release

Migration Notes

By default, users should use the new HTTP transport to send data.

LineTcpSender has been deprecated. Ideally, users should adopt the new Sender class instead.

Users requiring the very highest performance should continue to use ILP/TCP with custom database tuning. Against mocked tests, potentially there is a ~20% performance overhead with ILP over HTTP, for small batch sizes, but YMMV.

nwoolmer commented 7 months ago

Looks good, thanks @nwoolmer !

Thank you for all your feedback!