search

DataDog / libdatadog

Datadog shared rust-based library. For now only used in other products (e.g. Ruby or PHP libraries).
Apache License 2.0
48 stars 9 forks source link

[PROF-9476] Add experimental profiling managed string storage #725

Open ivoanjo opened 2 weeks ago

ivoanjo commented 2 weeks ago

What does this PR do?

This PR builds on the work started by @alexjf on #607 to introduce "managed string storage" for profiling.

The idea is to introduce another level of string storage for profiling that is decoupled in lifetime from individual profiles, and that is managed by the libdatadog client.

At its core, managed string storage provides a hashtable that stores strings and returns ids. These ids can then be provided to libdatadog instead of CharSlices when recording profiling samples.

For FFI users, this PR adds the following APIs to manage strings:

A key detail of the current implementation is that each intern call with the same string will increase an internal usage counter, and unintern call with reduce the counter.

Then at advance_gen time, if the counter is zero, we get rid of the string.

Then to interact with profiles, there's a new ddog_prof_Profile_with_string_storage API to create a profile with a given ManagedStringStorage, and all structures that make up a Sample (Mapping, Function, Label) etc have been extended so that they either take a CharSlice or a ManagedStringId.

Thus, after interning all strings for a sample, it's possible to add a sample to a profile entirely by referencing strings by ids, rather than CharSlices.

Motivation

The initial use-case is to support heap profiling -- "samples" related to heap profiling usually live across multiple profiles (as long as a given object is alive) and so this data must be kept somewhere. Previously for Ruby we were keeping this on the Ruby profiler side, but having libdatadog manage this instead presents a few optimization opportunities.

We also hope to replace a few other "string tables" that other profilers had to build outside of libdatadog for similar use-cases.

This topic was also discussed in the following two documents (Datadog-only, sorry!):

Additional Notes

In keeping with the experimental nature of this feature, I've tried really hard to not disturb existing profiling API users with the new changes.

That is -- I was going for, if you're not using managed string storage, you should NOT be affected AT ALL by it -- be it API changes or overhead.

(This is why on the pure-Rust profiling crate side, I ended up duplicating a bunch of structures and functions. I couldn't think of a great way to not disturb existing API users other than introducing alternative methods, but to be honest the duplication is all in very simple methods so I don't think this substantially increases complexity/maintenance vs trying to be smarter to bend Rust to our will.)

There's probably a lot of improvements we can make, but with this PR I'm hoping to have something in a close to "good enough" state, that we can merge this in and then start iterating on master, rather than have this continue living in a branch for a lot longer.

This doesn't mean we shouldn't fix or improve things before merging, but I'll be trying to identify what needs to go in now and what can go in as separate, follow-up PRs.

As an addendum, there's still a bunch of expects sprinkled that should be turned into proper errors. I plan to do a pass on all of those. (But again, none of the panics affect existing code, so they're harmless and inert unless you're experimenting with the new APIs)

How to test the change?

The branch in https://github.com/DataDog/dd-trace-rb/tree/ivoanjo/prof-9476-managed-string-storage-try2 is where I'm testing the changes on the Ruby profiler side.

It may not be entirely up-to-date with the latest ffi changes on the libdatadog side (I've been prettying up the API), but it shows how to use this concept, while passing all the profiling unit/integration tests, and has shown improvements in memory and latency in the reliability environment.

pr-commenter[bot] commented 2 weeks ago

Benchmarks

Comparison

Benchmark execution time: 2024-11-28 15:28:02

Comparing candidate commit 3cb2cb08 in PR branch ivoanjo/prof-9476-managed-string-storage-try3-clean with baseline commit 17f36e91 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 2 unstable metrics.

Candidate

Candidate benchmark details ### Group 1 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |-------------------------------------------------------------------------------------------------------|--------------|----------------|---------------------------------|---------------------------------|----------------|----------------|----------------|----------------|--------------------|--------|--------|-----|------------|----|-----------| |normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...|execution_time|508.410µs |509.713µs ± 0.805µs |509.590µs ± 0.251µs |509.870µs |510.690µs |511.925µs |518.353µs |1.72% |6.609 |65.402 |0.16%|0.057µs |1 |200 | |normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...|throughput |1929187.253op/s |1961893.471op/s ± 3065.001op/s |1962363.258op/s ± 967.903op/s |1963255.083op/s |1964339.417op/s |1965682.648op/s |1966917.831op/s |0.23% |-6.498 |63.742 |0.16%|216.728op/s |1 |200 | |normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて |execution_time|465.938µs |466.936µs ± 0.524µs |466.890µs ± 0.259µs |467.175µs |467.626µs |468.402µs |470.504µs |0.77% |2.606 |14.375 |0.11%|0.037µs |1 |200 | |normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて |throughput |2125380.443op/s |2141625.155op/s ± 2395.432op/s |2141830.312op/s ± 1190.256op/s |2142950.232op/s |2144626.306op/s |2145673.035op/s |2146209.991op/s |0.20% |-2.574 |14.119 |0.11%|169.383op/s |1 |200 | |normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters |execution_time|175.904µs |176.619µs ± 0.346µs |176.586µs ± 0.243µs |176.862µs |177.216µs |177.484µs |177.704µs |0.63% |0.389 |-0.024 |0.20%|0.024µs |1 |200 | |normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters |throughput |5627346.157op/s |5661928.723op/s ± 11092.402op/s |5662978.733op/s ± 7780.175op/s |5669983.781op/s |5678625.372op/s |5683636.867op/s |5684905.647op/s |0.39% |-0.378 |-0.038 |0.20%|784.351op/s |1 |200 | |normalization/normalize_service/normalize_service/[empty string] |execution_time|44.934µs |45.391µs ± 0.222µs |45.437µs ± 0.130µs |45.534µs |45.665µs |45.696µs |46.935µs |3.30% |1.322 |10.232 |0.49%|0.016µs |1 |200 | |normalization/normalize_service/normalize_service/[empty string] |throughput |21306196.794op/s|22031208.704op/s ± 106869.611op/s|22008361.305op/s ± 63250.433op/s |22106994.095op/s|22206847.724op/s|22239498.703op/s|22254785.410op/s|1.12% |-1.176 |9.078 |0.48%|7556.823op/s|1 |200 | |normalization/normalize_service/normalize_service/test_ASCII |execution_time|41.799µs |42.091µs ± 0.217µs |42.061µs ± 0.177µs |42.272µs |42.414µs |42.516µs |43.236µs |2.79% |0.923 |2.268 |0.51%|0.015µs |1 |200 | |normalization/normalize_service/normalize_service/test_ASCII |throughput |23128991.657op/s|23758875.664op/s ± 121883.572op/s|23774851.455op/s ± 100642.335op/s|23865312.568op/s|23912143.006op/s|23918916.998op/s|23924287.779op/s|0.63% |-0.872 |1.959 |0.51%|8618.470op/s|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |-------------------------------------------------------------------------------------------------------|--------------|----------------------------------------------------------|-------------------|------------------------|---------------| |normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...|execution_time|[509.601µs; 509.824µs] or [-0.022%; +0.022%] |None |None |None | |normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...|throughput |[1961468.692op/s; 1962318.251op/s] or [-0.022%; +0.022%] |None |None |None | |normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて |execution_time|[466.863µs; 467.008µs] or [-0.016%; +0.016%] |None |None |None | |normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて |throughput |[2141293.172op/s; 2141957.139op/s] or [-0.016%; +0.016%] |None |None |None | |normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters |execution_time|[176.571µs; 176.667µs] or [-0.027%; +0.027%] |None |None |None | |normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters |throughput |[5660391.423op/s; 5663466.023op/s] or [-0.027%; +0.027%] |None |None |None | |normalization/normalize_service/normalize_service/[empty string] |execution_time|[45.361µs; 45.422µs] or [-0.068%; +0.068%] |None |None |None | |normalization/normalize_service/normalize_service/[empty string] |throughput |[22016397.604op/s; 22046019.805op/s] or [-0.067%; +0.067%]|None |None |None | |normalization/normalize_service/normalize_service/test_ASCII |execution_time|[42.061µs; 42.121µs] or [-0.071%; +0.071%] |None |None |None | |normalization/normalize_service/normalize_service/test_ASCII |throughput |[23741983.774op/s; 23775767.555op/s] or [-0.071%; +0.071%]|None |None |None | ### Group 2 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |--------------------------------------|--------------|-------|-----------------|-----------------|-------|-------|-------|-------|--------------------|--------|--------|-----|-------|----|-----------| |concentrator/add_spans_to_concentrator|execution_time|9.123ms|9.155ms ± 0.015ms|9.153ms ± 0.010ms|9.163ms|9.178ms|9.192ms|9.227ms|0.80% |1.070 |3.048 |0.16%|0.001ms|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |--------------------------------------|--------------|----------------------------------------|-------------------|------------------------|---------------| |concentrator/add_spans_to_concentrator|execution_time|[9.153ms; 9.157ms] or [-0.022%; +0.022%]|None |None |None | ### Group 3 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |------------------------|--------------|--------|------------------|------------------|--------|--------|--------|--------|--------------------|--------|--------|-----|-------|----|-----------| |sql/obfuscate_sql_string|execution_time|75.289µs|75.542µs ± 0.233µs|75.516µs ± 0.038µs|75.562µs|75.665µs|75.794µs|78.652µs|4.15% |11.978 |157.153 |0.31%|0.016µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |------------------------|--------------|------------------------------------------|-------------------|------------------------|---------------| |sql/obfuscate_sql_string|execution_time|[75.509µs; 75.574µs] or [-0.043%; +0.043%]|None |None |None | ### Group 4 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |--------------------|--------------|-------|-----------------|-----------------|-------|-------|--------|--------|--------------------|--------|--------|------|-------|----|-----------| |write only interface|execution_time|1.433µs|3.325µs ± 1.490µs|3.160µs ± 0.026µs|3.185µs|3.270µs|14.635µs|15.645µs|395.16% |7.597 |57.740 |44.71%|0.105µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |--------------------|--------------|----------------------------------------|-------------------|------------------------|---------------| |write only interface|execution_time|[3.118µs; 3.531µs] or [-6.211%; +6.211%]|None |None |None | ### Group 5 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |----------------------------------------|--------------|---------|--------------------|-------------------|---------|---------|---------|---------|--------------------|--------|--------|-----|-------|----|-----------| |normalization/normalize_trace/test_trace|execution_time|298.259ns|309.142ns ± 12.687ns|304.726ns ± 4.391ns|310.816ns|335.947ns|348.652ns|349.979ns|14.85% |1.598 |1.553 |4.09%|0.897ns|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |----------------------------------------|--------------|--------------------------------------------|-------------------|------------------------|---------------| |normalization/normalize_trace/test_trace|execution_time|[307.384ns; 310.900ns] or [-0.569%; +0.569%]|None |None |None | ### Group 6 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |-------------------------------------------------------------------------------------------------------|--------------|----------------|---------------------------------|--------------------------------|----------------|----------------|----------------|----------------|--------------------|--------|--------|-----|-------------|----|-----------| |normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...|execution_time|280.148µs |280.699µs ± 0.254µs |280.706µs ± 0.152µs |280.818µs |281.121µs |281.323µs |281.962µs |0.45% |1.007 |2.771 |0.09%|0.018µs |1 |200 | |normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...|throughput |3546575.463op/s |3562543.615op/s ± 3222.185op/s |3562444.368op/s ± 1929.028op/s |3564797.261op/s |3566717.452op/s |3568457.498op/s |3569540.935op/s |0.20% |-0.997 |2.725 |0.09%|227.843op/s |1 |200 | |normalization/normalize_name/normalize_name/bad-name |execution_time|27.221µs |27.327µs ± 0.111µs |27.318µs ± 0.042µs |27.357µs |27.435µs |27.473µs |28.658µs |4.90% |8.654 |100.755 |0.41%|0.008µs |1 |200 | |normalization/normalize_name/normalize_name/bad-name |throughput |34894490.692op/s|36594811.073op/s ± 144000.424op/s|36605747.750op/s ± 56225.027op/s|36663150.496op/s|36712144.918op/s|36725205.591op/s|36735752.039op/s|0.36% |-8.319 |95.393 |0.39%|10182.368op/s|1 |200 | |normalization/normalize_name/normalize_name/good |execution_time|16.924µs |17.001µs ± 0.025µs |16.999µs ± 0.015µs |17.016µs |17.043µs |17.057µs |17.089µs |0.53% |0.222 |0.541 |0.15%|0.002µs |1 |200 | |normalization/normalize_name/normalize_name/good |throughput |58516852.854op/s|58821889.947op/s ± 86873.653op/s |58826132.077op/s ± 51820.219op/s|58867443.224op/s|58959138.398op/s|59020309.179op/s|59087566.764op/s|0.44% |-0.211 |0.534 |0.15%|6142.895op/s |1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |-------------------------------------------------------------------------------------------------------|--------------|----------------------------------------------------------|-------------------|------------------------|---------------| |normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...|execution_time|[280.663µs; 280.734µs] or [-0.013%; +0.013%] |None |None |None | |normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...|throughput |[3562097.051op/s; 3562990.179op/s] or [-0.013%; +0.013%] |None |None |None | |normalization/normalize_name/normalize_name/bad-name |execution_time|[27.311µs; 27.342µs] or [-0.056%; +0.056%] |None |None |None | |normalization/normalize_name/normalize_name/bad-name |throughput |[36574853.999op/s; 36614768.147op/s] or [-0.055%; +0.055%]|None |None |None | |normalization/normalize_name/normalize_name/good |execution_time|[16.997µs; 17.004µs] or [-0.020%; +0.020%] |None |None |None | |normalization/normalize_name/normalize_name/good |throughput |[58809850.094op/s; 58833929.800op/s] or [-0.020%; +0.020%]|None |None |None | ### Group 7 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |-----------------------|--------------|-------|-----------------|-----------------|-------|-------|-------|-------|--------------------|--------|--------|-----|-------|----|-----------| |tags/replace_trace_tags|execution_time|2.595µs|2.654µs ± 0.016µs|2.653µs ± 0.005µs|2.661µs|2.686µs|2.693µs|2.696µs|1.63% |-0.456 |2.305 |0.61%|0.001µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |-----------------------|--------------|----------------------------------------|-------------------|------------------------|---------------| |tags/replace_trace_tags|execution_time|[2.652µs; 2.656µs] or [-0.085%; +0.085%]|None |None |None | ### Group 8 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |-----------------|--------------|--------|-------------------|------------------|--------|--------|--------|---------|--------------------|--------|--------|------|-------|----|-----------| |two way interface|execution_time|18.881µs|24.964µs ± 13.519µs|19.246µs ± 0.294µs|25.550µs|45.344µs|59.420µs|151.656µs|687.99% |5.095 |39.549 |54.02%|0.956µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |-----------------|--------------|------------------------------------------|-------------------|------------------------|---------------| |two way interface|execution_time|[23.090µs; 26.837µs] or [-7.505%; +7.505%]|None |None |None | ### Group 9 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |----------------------------------------------------------|--------------|-----------------|----------------------------------|----------------------------------|-----------------|-----------------|-----------------|-----------------|--------------------|--------|--------|-----|-------------|----|-----------| |credit_card/is_card_number/ |execution_time|4.622µs |4.631µs ± 0.006µs |4.631µs ± 0.002µs |4.634µs |4.636µs |4.640µs |4.694µs |1.36% |7.077 |77.925 |0.12%|0.000µs |1 |200 | |credit_card/is_card_number/ |throughput |213036470.956op/s|215914947.352op/s ± 257409.519op/s|215942883.129op/s ± 109800.064op/s|216031837.693op/s|216186537.676op/s|216273623.812op/s|216363707.392op/s|0.19% |-6.970 |76.366 |0.12%|18201.602op/s|1 |200 | |credit_card/is_card_number/ 3782-8224-6310-005 |execution_time|91.159µs |92.292µs ± 0.479µs |92.308µs ± 0.137µs |92.416µs |92.647µs |93.012µs |97.771µs |5.92% |7.429 |84.354 |0.52%|0.034µs |1 |200 | |credit_card/is_card_number/ 3782-8224-6310-005 |throughput |10228027.780op/s |10835416.170op/s ± 54187.795op/s |10833285.380op/s ± 16020.688op/s |10853136.159op/s |10894495.719op/s |10933357.827op/s |10969885.852op/s |1.26% |-6.954 |77.591 |0.50%|3831.656op/s |1 |200 | |credit_card/is_card_number/ 378282246310005 |execution_time|86.639µs |87.575µs ± 0.500µs |87.579µs ± 0.149µs |87.714µs |87.886µs |88.413µs |93.042µs |6.24% |7.010 |72.702 |0.57%|0.035µs |1 |200 | |credit_card/is_card_number/ 378282246310005 |throughput |10747806.186op/s |11419192.616op/s ± 62708.231op/s |11418236.867op/s ± 19388.863op/s |11440427.693op/s |11494998.262op/s |11538864.663op/s |11542192.485op/s |1.09% |-6.583 |66.764 |0.55%|4434.142op/s |1 |200 | |credit_card/is_card_number/37828224631 |execution_time|4.616µs |4.628µs ± 0.006µs |4.628µs ± 0.002µs |4.630µs |4.634µs |4.660µs |4.677µs |1.06% |4.281 |28.316 |0.13%|0.000µs |1 |200 | |credit_card/is_card_number/37828224631 |throughput |213808655.402op/s|216055628.475op/s ± 290193.686op/s|216085511.473op/s ± 99587.027op/s |216176299.426op/s|216364989.787op/s|216524608.656op/s|216647164.078op/s|0.26% |-4.233 |27.882 |0.13%|20519.792op/s|1 |200 | |credit_card/is_card_number/378282246310005 |execution_time|84.227µs |84.753µs ± 0.192µs |84.758µs ± 0.113µs |84.859µs |85.037µs |85.323µs |85.396µs |0.75% |0.218 |1.014 |0.23%|0.014µs |1 |200 | |credit_card/is_card_number/378282246310005 |throughput |11710149.196op/s |11799114.684op/s ± 26733.577op/s |11798238.523op/s ± 15713.769op/s |11816294.671op/s |11841308.406op/s |11863094.679op/s |11872690.548op/s |0.63% |-0.198 |0.991 |0.23%|1890.349op/s |1 |200 | |credit_card/is_card_number/37828224631000521389798 |execution_time|58.592µs |58.671µs ± 0.054µs |58.666µs ± 0.021µs |58.687µs |58.730µs |58.752µs |59.275µs |1.04% |7.035 |76.011 |0.09%|0.004µs |1 |200 | |credit_card/is_card_number/37828224631000521389798 |throughput |16870533.748op/s |17044245.484op/s ± 15611.801op/s |17045761.240op/s ± 5959.452op/s |17051225.397op/s |17061492.808op/s |17065147.850op/s |17067081.014op/s |0.13% |-6.957 |74.853 |0.09%|1103.921op/s |1 |200 | |credit_card/is_card_number/x371413321323331 |execution_time|6.551µs |6.564µs ± 0.004µs |6.564µs ± 0.002µs |6.567µs |6.572µs |6.574µs |6.580µs |0.25% |0.128 |0.824 |0.07%|0.000µs |1 |200 | |credit_card/is_card_number/x371413321323331 |throughput |151965576.008op/s|152338702.240op/s ± 102125.546op/s|152340145.557op/s ± 54463.198op/s |152392292.774op/s|152510938.745op/s|152560891.414op/s|152654429.137op/s|0.21% |-0.123 |0.819 |0.07%|7221.367op/s |1 |200 | |credit_card/is_card_number_no_luhn/ |execution_time|4.619µs |4.631µs ± 0.003µs |4.630µs ± 0.002µs |4.633µs |4.636µs |4.640µs |4.641µs |0.22% |-0.054 |0.689 |0.07%|0.000µs |1 |200 | |credit_card/is_card_number_no_luhn/ |throughput |215490214.799op/s|215958248.269op/s ± 159315.090op/s|215962983.947op/s ± 86091.602op/s |216044732.065op/s|216204106.470op/s|216331975.349op/s|216498683.067op/s|0.25% |0.059 |0.692 |0.07%|11265.278op/s|1 |200 | |credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 |execution_time|68.894µs |70.196µs ± 0.430µs |70.228µs ± 0.278µs |70.490µs |70.877µs |71.095µs |71.180µs |1.36% |-0.282 |-0.250 |0.61%|0.030µs |1 |200 | |credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 |throughput |14048855.632op/s |14246444.825op/s ± 87425.737op/s |14239331.194op/s ± 56332.458op/s |14305148.427op/s |14405380.201op/s |14429018.473op/s |14514948.976op/s |1.94% |0.312 |-0.229 |0.61%|6181.933op/s |1 |200 | |credit_card/is_card_number_no_luhn/ 378282246310005 |execution_time|63.942µs |64.445µs ± 0.172µs |64.436µs ± 0.110µs |64.561µs |64.730µs |64.856µs |65.099µs |1.03% |0.418 |0.382 |0.27%|0.012µs |1 |200 | |credit_card/is_card_number_no_luhn/ 378282246310005 |throughput |15361298.230op/s |15517258.355op/s ± 41458.264op/s |15519278.840op/s ± 26456.715op/s |15545537.933op/s |15578070.516op/s |15587789.764op/s |15639120.928op/s |0.77% |-0.401 |0.354 |0.27%|2931.542op/s |1 |200 | |credit_card/is_card_number_no_luhn/37828224631 |execution_time|4.619µs |4.630µs ± 0.004µs |4.630µs ± 0.002µs |4.632µs |4.635µs |4.638µs |4.640µs |0.22% |0.019 |0.225 |0.08%|0.000µs |1 |200 | |credit_card/is_card_number_no_luhn/37828224631 |throughput |215521153.287op/s|215997788.830op/s ± 168219.950op/s|216005378.844op/s ± 102115.648op/s|216094549.018op/s|216293288.343op/s|216380184.645op/s|216493216.629op/s|0.23% |-0.014 |0.225 |0.08%|11894.947op/s|1 |200 | |credit_card/is_card_number_no_luhn/378282246310005 |execution_time|62.229µs |62.487µs ± 0.113µs |62.492µs ± 0.069µs |62.554µs |62.669µs |62.763µs |62.924µs |0.69% |0.416 |0.791 |0.18%|0.008µs |1 |200 | |credit_card/is_card_number_no_luhn/378282246310005 |throughput |15892254.855op/s |16003279.683op/s ± 28812.109op/s |16002090.410op/s ± 17708.868op/s |16022355.267op/s |16049560.965op/s |16060092.264op/s |16069702.184op/s |0.42% |-0.402 |0.761 |0.18%|2037.324op/s |1 |200 | |credit_card/is_card_number_no_luhn/37828224631000521389798|execution_time|58.587µs |58.672µs ± 0.040µs |58.666µs ± 0.018µs |58.689µs |58.724µs |58.779µs |58.927µs |0.44% |1.626 |8.376 |0.07%|0.003µs |1 |200 | |credit_card/is_card_number_no_luhn/37828224631000521389798|throughput |16970168.032op/s |17043847.072op/s ± 11483.793op/s |17045557.342op/s ± 5205.255op/s |17049769.250op/s |17061547.892op/s |17068102.209op/s |17068597.195op/s |0.14% |-1.610 |8.273 |0.07%|812.027op/s |1 |200 | |credit_card/is_card_number_no_luhn/x371413321323331 |execution_time|6.556µs |6.566µs ± 0.005µs |6.566µs ± 0.002µs |6.568µs |6.573µs |6.576µs |6.611µs |0.68% |3.428 |29.836 |0.08%|0.000µs |1 |200 | |credit_card/is_card_number_no_luhn/x371413321323331 |throughput |151257022.328op/s|152290417.504op/s ± 116913.683op/s|152291479.292op/s ± 51313.799op/s |152345619.896op/s|152465578.412op/s|152527837.237op/s|152538478.658op/s|0.16% |-3.382 |29.340 |0.08%|8267.046op/s |1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |----------------------------------------------------------|--------------|------------------------------------------------------------|-------------------|------------------------|---------------| |credit_card/is_card_number/ |execution_time|[4.631µs; 4.632µs] or [-0.017%; +0.017%] |None |None |None | |credit_card/is_card_number/ |throughput |[215879272.869op/s; 215950621.836op/s] or [-0.017%; +0.017%]|None |None |None | |credit_card/is_card_number/ 3782-8224-6310-005 |execution_time|[92.226µs; 92.359µs] or [-0.072%; +0.072%] |None |None |None | |credit_card/is_card_number/ 3782-8224-6310-005 |throughput |[10827906.263op/s; 10842926.077op/s] or [-0.069%; +0.069%] |None |None |None | |credit_card/is_card_number/ 378282246310005 |execution_time|[87.505µs; 87.644µs] or [-0.079%; +0.079%] |None |None |None | |credit_card/is_card_number/ 378282246310005 |throughput |[11410501.858op/s; 11427883.373op/s] or [-0.076%; +0.076%] |None |None |None | |credit_card/is_card_number/37828224631 |execution_time|[4.628µs; 4.629µs] or [-0.019%; +0.019%] |None |None |None | |credit_card/is_card_number/37828224631 |throughput |[216015410.421op/s; 216095846.529op/s] or [-0.019%; +0.019%]|None |None |None | |credit_card/is_card_number/378282246310005 |execution_time|[84.726µs; 84.779µs] or [-0.031%; +0.031%] |None |None |None | |credit_card/is_card_number/378282246310005 |throughput |[11795409.668op/s; 11802819.701op/s] or [-0.031%; +0.031%] |None |None |None | |credit_card/is_card_number/37828224631000521389798 |execution_time|[58.663µs; 58.678µs] or [-0.013%; +0.013%] |None |None |None | |credit_card/is_card_number/37828224631000521389798 |throughput |[17042081.839op/s; 17046409.130op/s] or [-0.013%; +0.013%] |None |None |None | |credit_card/is_card_number/x371413321323331 |execution_time|[6.564µs; 6.565µs] or [-0.009%; +0.009%] |None |None |None | |credit_card/is_card_number/x371413321323331 |throughput |[152324548.622op/s; 152352855.859op/s] or [-0.009%; +0.009%]|None |None |None | |credit_card/is_card_number_no_luhn/ |execution_time|[4.630µs; 4.631µs] or [-0.010%; +0.010%] |None |None |None | |credit_card/is_card_number_no_luhn/ |throughput |[215936168.730op/s; 215980327.809op/s] or [-0.010%; +0.010%]|None |None |None | |credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 |execution_time|[70.136µs; 70.255µs] or [-0.085%; +0.085%] |None |None |None | |credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 |throughput |[14234328.459op/s; 14258561.192op/s] or [-0.085%; +0.085%] |None |None |None | |credit_card/is_card_number_no_luhn/ 378282246310005 |execution_time|[64.421µs; 64.469µs] or [-0.037%; +0.037%] |None |None |None | |credit_card/is_card_number_no_luhn/ 378282246310005 |throughput |[15511512.639op/s; 15523004.072op/s] or [-0.037%; +0.037%] |None |None |None | |credit_card/is_card_number_no_luhn/37828224631 |execution_time|[4.629µs; 4.630µs] or [-0.011%; +0.011%] |None |None |None | |credit_card/is_card_number_no_luhn/37828224631 |throughput |[215974475.163op/s; 216021102.498op/s] or [-0.011%; +0.011%]|None |None |None | |credit_card/is_card_number_no_luhn/378282246310005 |execution_time|[62.472µs; 62.503µs] or [-0.025%; +0.025%] |None |None |None | |credit_card/is_card_number_no_luhn/378282246310005 |throughput |[15999286.602op/s; 16007272.764op/s] or [-0.025%; +0.025%] |None |None |None | |credit_card/is_card_number_no_luhn/37828224631000521389798|execution_time|[58.667µs; 58.678µs] or [-0.009%; +0.009%] |None |None |None | |credit_card/is_card_number_no_luhn/37828224631000521389798|throughput |[17042255.529op/s; 17045438.615op/s] or [-0.009%; +0.009%] |None |None |None | |credit_card/is_card_number_no_luhn/x371413321323331 |execution_time|[6.566µs; 6.567µs] or [-0.011%; +0.011%] |None |None |None | |credit_card/is_card_number_no_luhn/x371413321323331 |throughput |[152274214.392op/s; 152306620.616op/s] or [-0.011%; +0.011%]|None |None |None | ### Group 10 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |----------------------------------------------------------------------------|--------------|--------|------------------|------------------|--------|--------|--------|--------|--------------------|--------|--------|-----|-------|----|-----------| |benching deserializing traces from msgpack to their internal representation |execution_time|57.983ms|58.301ms ± 0.183ms|58.279ms ± 0.080ms|58.347ms|58.669ms|58.884ms|59.422ms|1.96% |1.988 |7.235 |0.31%|0.013ms|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |----------------------------------------------------------------------------|--------------|------------------------------------------|-------------------|------------------------|---------------| |benching deserializing traces from msgpack to their internal representation |execution_time|[58.275ms; 58.326ms] or [-0.043%; +0.043%]|None |None |None | ### Group 11 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |----------------------------------------------|--------------|---------|-------------------|-------------------|---------|---------|---------|---------|--------------------|--------|--------|-----|-------|----|-----------| |benching string interning on wordpress profile|execution_time|149.590µs|150.656µs ± 0.406µs|150.598µs ± 0.137µs|150.761µs|151.056µs|151.821µs|154.844µs|2.82% |5.858 |56.013 |0.27%|0.029µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |----------------------------------------------|--------------|--------------------------------------------|-------------------|------------------------|---------------| |benching string interning on wordpress profile|execution_time|[150.600µs; 150.712µs] or [-0.037%; +0.037%]|None |None |None | ### Group 12 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|---------------------------------------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cb2cb08 |1732807090 |ivoanjo/prof-9476-managed-string-storage-try3-clean| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |----------------------------|--------------|--------|------------------|------------------|--------|--------|--------|--------|--------------------|--------|--------|-----|-------|----|-----------| |redis/obfuscate_redis_string|execution_time|37.560µs|38.264µs ± 1.258µs|37.695µs ± 0.045µs|37.761µs|40.993µs|41.010µs|41.022µs|8.83% |1.701 |0.910 |3.28%|0.089µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |----------------------------|--------------|------------------------------------------|-------------------|------------------------|---------------| |redis/obfuscate_redis_string|execution_time|[38.089µs; 38.438µs] or [-0.456%; +0.456%]|None |None |None |

Baseline

Omitted due to size.

nsrip-dd commented 5 days ago

Forgot to comment sooner, but I tried out using this in the Python profiler. Commit here, very WIP because I was trying out other approaches as well at the time. The biggest issue I ran into was around forking. If the program forks while the RwLock in the table is held, the child process will deadlock trying to access the table. I think we'd need to just make a new table on fork. Not the end of the world, but I guess just something to look out for.

ivoanjo commented 4 days ago

If the program forks while the RwLock in the table is held, the child process will deadlock trying to access the table. I think we'd need to just make a new table on fork. Not the end of the world, but I guess just something to look out for.

Yeap, this is a good point. I think even without the lock, it's probably not a great idea to do anything other than throw away the table after the fork.

(It would, on the other hand, be cool to have a lock-free implementation for this, but I fear the cost would probably not be great).

Would having a way to clear the table ignoring the lock work as a temporary solution for Python? How do y'all handle this issue for the regular profile data structures? Reset on fork as well?

codecov-commenter commented 4 days ago

Codecov Report

Attention: Patch coverage is 8.28283% with 454 lines in your changes missing coverage. Please review.

Project coverage is 69.86%. Comparing base (17f36e9) to head (3cb2cb0).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #725 +/- ## ========================================== - Coverage 70.54% 69.86% -0.68% ========================================== Files 297 299 +2 Lines 43515 43999 +484 ========================================== + Hits 30698 30741 +43 - Misses 12817 13258 +441 ``` | [Components](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | Coverage Δ | | |---|---|---| | [crashtracker](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `44.86% <ø> (ø)` | | | [crashtracker-ffi](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `8.41% <ø> (ø)` | | | [datadog-alloc](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `98.73% <ø> (ø)` | | | [data-pipeline](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `89.09% <ø> (ø)` | | | [data-pipeline-ffi](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `0.00% <ø> (ø)` | | | [ddcommon](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `83.46% <ø> (ø)` | | | [ddcommon-ffi](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `69.12% <ø> (ø)` | | | [ddtelemetry](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `59.05% <ø> (ø)` | | | [ddtelemetry-ffi](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `22.13% <ø> (ø)` | | | [dogstatsd](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `89.45% <ø> (ø)` | | | [dogstatsd-client](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `79.77% <ø> (ø)` | | | [ipc](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `82.86% <ø> (+0.10%)` | :arrow_up: | | [profiling](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `78.10% <8.28%> (-6.20%)` | :arrow_down: | | [profiling-ffi](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `66.20% <8.00%> (-11.26%)` | :arrow_down: | | [serverless](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `0.00% <ø> (ø)` | | | [sidecar](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `38.01% <ø> (ø)` | | | [sidecar-ffi](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `0.00% <ø> (ø)` | | | [spawn-worker](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `50.36% <ø> (ø)` | | | [tinybytes](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `93.47% <ø> (ø)` | | | [trace-mini-agent](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `72.36% <ø> (ø)` | | | [trace-normalization](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `98.23% <ø> (ø)` | | | [trace-obfuscation](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `95.77% <ø> (ø)` | | | [trace-protobuf](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `77.67% <ø> (ø)` | | | [trace-utils](https://app.codecov.io/gh/DataDog/libdatadog/pull/725/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `93.29% <ø> (ø)` | |