search

RobThree / HumanoID

Friendly ID generator. Converts integers to words and back.
MIT License
7 stars 4 forks source link

Refactor benchmark tests #12

Closed mallardduck closed 2 years ago

mallardduck commented 2 years ago

TL;DR

Attempting to ensure test runs are consistent between generators & easier to grok in reports

Summary

Alright, this is another slightly large change-set, but I think it's worth it for more granular test results. Also as a preface, the AlwaysNew subset of tests are likely just temporary as they feel like good baselines for the discussion. I think it's good to see what effects those ones show PHP's native file/json functions have on the results compared to the libraries logic. So that said, I think we could remove those benchmarks once we are in a more stable period.

Per my comment about how the random numbers were used, I think that was likely making things "too random" between revisions/iterations. As such I've also included a new ID provider that provides a consistent set of IDs every time. This ensures that we can actually compare apples to apples against generators. Since now we have a benchmark for every generator we can say "when int 9 is passed here's the performance".

Benchmark layout

Each benchmark class instance consists of a setUp/tearDown and then the same basic tests. This makes it easy to setup new tests for different style generators just by editing the setUp. TBH we should be able to automate/generate these if we want.

Here's a description of what each benchmark method targets:

Initial results

Note: Manually sorted for better comparisons...

Mac mini M1 2020 / PHP version 7.4.29, xdebug ✔, opcache ✔

+-----------------------------+----------------------+-----+-------+-----+----------+------------------+---------+
| benchmark                   | subject              | set | revs  | its | mem_peak | mode             | rstdev  |
+-----------------------------+----------------------+-----+-------+-----+----------+------------------+---------+
| ZooBench                    | benchCreate          | 0   | 10000 | 5   | 3.319mb  | 98,199.060ops/s  | ±15.79% |
| ZooBench                    | benchCreate          | 1   | 10000 | 5   | 3.319mb  | 95,471.640ops/s  | ±1.13%  |
| ZooBench                    | benchCreate          | 2   | 10000 | 5   | 3.319mb  | 98,386.184ops/s  | ±4.25%  |
| ZooBench                    | benchCreate          | 3   | 10000 | 5   | 3.319mb  | 51,206.199ops/s  | ±2.81%  |
| ZooBench                    | benchCreate          | 4   | 10000 | 5   | 3.319mb  | 49,654.130ops/s  | ±3.70%  |
| ZooBench                    | benchCreate          | 5   | 10000 | 5   | 3.319mb  | 34,660.647ops/s  | ±3.90%  |
| ZooBench                    | benchCreate          | 6   | 10000 | 5   | 3.319mb  | 34,730.142ops/s  | ±0.89%  |
| ZooBench                    | benchCreate          | 7   | 10000 | 5   | 3.319mb  | 26,438.032ops/s  | ±2.53%  |
| SpaceBench                  | benchCreate          | 0   | 10000 | 5   | 1.316mb  | 49,319.316ops/s  | ±5.04%  |
| SpaceBench                  | benchCreate          | 1   | 10000 | 5   | 1.316mb  | 50,312.329ops/s  | ±11.41% |
| SpaceBench                  | benchCreate          | 2   | 10000 | 5   | 1.316mb  | 51,431.480ops/s  | ±6.84%  |
| SpaceBench                  | benchCreate          | 3   | 10000 | 5   | 1.316mb  | 34,811.466ops/s  | ±0.67%  |
| SpaceBench                  | benchCreate          | 4   | 10000 | 5   | 1.316mb  | 34,625.889ops/s  | ±4.57%  |
| SpaceBench                  | benchCreate          | 5   | 10000 | 5   | 1.316mb  | 16,905.746ops/s  | ±5.45%  |
| SpaceBench                  | benchCreate          | 6   | 10000 | 5   | 1.316mb  | 17,427.462ops/s  | ±1.69%  |
| SpaceBench                  | benchCreate          | 7   | 10000 | 5   | 1.316mb  | 12,911.392ops/s  | ±4.23%  |
| CustomSmallerGeneratorBench | benchCreate          | 0   | 10000 | 5   | 1.316mb  | 92,074.612ops/s  | ±4.48%  |
| CustomSmallerGeneratorBench | benchCreate          | 1   | 10000 | 5   | 1.316mb  | 49,762.659ops/s  | ±2.85%  |
| CustomSmallerGeneratorBench | benchCreate          | 2   | 10000 | 5   | 1.316mb  | 50,609.547ops/s  | ±5.25%  |
| CustomSmallerGeneratorBench | benchCreate          | 3   | 10000 | 5   | 1.316mb  | 34,739.902ops/s  | ±2.18%  |
| CustomSmallerGeneratorBench | benchCreate          | 4   | 10000 | 5   | 1.316mb  | 33,896.421ops/s  | ±1.84%  |
| CustomSmallerGeneratorBench | benchCreate          | 5   | 10000 | 5   | 1.316mb  | 21,210.088ops/s  | ±0.49%  |
| CustomSmallerGeneratorBench | benchCreate          | 6   | 10000 | 5   | 1.316mb  | 17,595.151ops/s  | ±5.09%  |
| CustomSmallerGeneratorBench | benchCreate          | 7   | 10000 | 5   | 1.316mb  | 11,868.362ops/s  | ±1.84%  |
| ZooBench                    | benchCreateRand      | 0   | 10000 | 5   | 3.319mb  | 93,105.900ops/s  | ±2.58%  |
| ZooBench                    | benchCreateRand      | 1   | 10000 | 5   | 3.319mb  | 93,245.846ops/s  | ±10.36% |
| ZooBench                    | benchCreateRand      | 2   | 10000 | 5   | 3.319mb  | 93,996.132ops/s  | ±1.81%  |
| ZooBench                    | benchCreateRand      | 3   | 10000 | 5   | 3.319mb  | 33,583.176ops/s  | ±2.49%  |
| ZooBench                    | benchCreateRand      | 4   | 10000 | 5   | 3.319mb  | 24,757.313ops/s  | ±5.92%  |
| SpaceBench                  | benchCreateRand      | 0   | 10000 | 5   | 1.316mb  | 80,736.130ops/s  | ±10.11% |
| SpaceBench                  | benchCreateRand      | 1   | 10000 | 5   | 1.316mb  | 37,978.693ops/s  | ±7.56%  |
| SpaceBench                  | benchCreateRand      | 2   | 10000 | 5   | 1.316mb  | 34,387.283ops/s  | ±0.76%  |
| SpaceBench                  | benchCreateRand      | 3   | 10000 | 5   | 1.316mb  | 16,084.073ops/s  | ±11.34% |
| SpaceBench                  | benchCreateRand      | 4   | 10000 | 5   | 1.316mb  | 12,734.647ops/s  | ±7.27%  |
| CustomSmallerGeneratorBench | benchCreateRand      | 0   | 10000 | 5   | 1.316mb  | 100,265.055ops/s | ±0.60%  |
| CustomSmallerGeneratorBench | benchCreateRand      | 1   | 10000 | 5   | 1.316mb  | 52,006.359ops/s  | ±0.27%  |
| CustomSmallerGeneratorBench | benchCreateRand      | 2   | 10000 | 5   | 1.316mb  | 34,076.722ops/s  | ±1.78%  |
| CustomSmallerGeneratorBench | benchCreateRand      | 3   | 10000 | 5   | 1.316mb  | 17,426.271ops/s  | ±1.76%  |
| CustomSmallerGeneratorBench | benchCreateRand      | 4   | 10000 | 5   | 1.316mb  | 11,866.423ops/s  | ±0.89%  |
| ZooBench                    | benchCreateAndDecode | 0   | 10000 | 5   | 3.319mb  | 70,002.621ops/s  | ±23.69% |
| ZooBench                    | benchCreateAndDecode | 1   | 10000 | 5   | 3.319mb  | 71,029.154ops/s  | ±2.20%  |
| ZooBench                    | benchCreateAndDecode | 2   | 10000 | 5   | 3.319mb  | 64,487.665ops/s  | ±5.93%  |
| ZooBench                    | benchCreateAndDecode | 3   | 10000 | 5   | 3.319mb  | 38,217.361ops/s  | ±8.93%  |
| ZooBench                    | benchCreateAndDecode | 4   | 10000 | 5   | 3.319mb  | 39,929.678ops/s  | ±1.79%  |
| ZooBench                    | benchCreateAndDecode | 5   | 10000 | 5   | 3.319mb  | 25,828.893ops/s  | ±6.26%  |
| ZooBench                    | benchCreateAndDecode | 6   | 10000 | 5   | 3.319mb  | 26,940.460ops/s  | ±1.23%  |
| ZooBench                    | benchCreateAndDecode | 7   | 10000 | 5   | 3.319mb  | 19,010.425ops/s  | ±11.58% |
| SpaceBench                  | benchCreateAndDecode | 0   | 10000 | 5   | 1.316mb  | 38,105.820ops/s  | ±1.05%  |
| SpaceBench                  | benchCreateAndDecode | 1   | 10000 | 5   | 1.316mb  | 38,173.585ops/s  | ±1.65%  |
| SpaceBench                  | benchCreateAndDecode | 2   | 10000 | 5   | 1.316mb  | 38,535.957ops/s  | ±2.29%  |
| SpaceBench                  | benchCreateAndDecode | 3   | 10000 | 5   | 1.316mb  | 24,285.195ops/s  | ±2.14%  |
| SpaceBench                  | benchCreateAndDecode | 4   | 10000 | 5   | 1.316mb  | 24,546.586ops/s  | ±1.12%  |
| SpaceBench                  | benchCreateAndDecode | 5   | 10000 | 5   | 1.316mb  | 13,312.817ops/s  | ±5.60%  |
| SpaceBench                  | benchCreateAndDecode | 6   | 10000 | 5   | 1.316mb  | 12,399.636ops/s  | ±2.22%  |
| SpaceBench                  | benchCreateAndDecode | 7   | 10000 | 5   | 1.316mb  | 9,215.610ops/s   | ±1.95%  |
| CustomSmallerGeneratorBench | benchCreateAndDecode | 0   | 10000 | 5   | 1.316mb  | 76,243.616ops/s  | ±1.21%  |
| CustomSmallerGeneratorBench | benchCreateAndDecode | 1   | 10000 | 5   | 1.316mb  | 41,117.014ops/s  | ±0.40%  |
| CustomSmallerGeneratorBench | benchCreateAndDecode | 2   | 10000 | 5   | 1.316mb  | 41,110.749ops/s  | ±0.84%  |
| CustomSmallerGeneratorBench | benchCreateAndDecode | 3   | 10000 | 5   | 1.316mb  | 27,320.004ops/s  | ±0.31%  |
| CustomSmallerGeneratorBench | benchCreateAndDecode | 4   | 10000 | 5   | 1.316mb  | 27,112.528ops/s  | ±0.61%  |
| CustomSmallerGeneratorBench | benchCreateAndDecode | 5   | 10000 | 5   | 1.316mb  | 16,424.558ops/s  | ±0.50%  |
| CustomSmallerGeneratorBench | benchCreateAndDecode | 6   | 10000 | 5   | 1.316mb  | 14,017.339ops/s  | ±1.52%  |
| CustomSmallerGeneratorBench | benchCreateAndDecode | 7   | 10000 | 5   | 1.316mb  | 9,357.598ops/s   | ±0.30%  |
+-----------------------------+----------------------+-----+-------+-----+----------+------------------+---------+
+-----------------------------+----------------------+-----+-------+-----+----------+------------------+---------+
| benchmark                   | subject              | set | revs  | its | mem_peak | mode             | rstdev  |
+-----------------------------+----------------------+-----+-------+-----+----------+------------------+---------+
| ReadsFileCreateBench        | benchCreate          | 0   | 1000  | 5   | 1.315mb  | 2,436.754ops/s   | ±2.42%  |
| ReadsFileCreateBench        | benchCreate          | 1   | 1000  | 5   | 1.315mb  | 2,495.340ops/s   | ±2.11%  |
| ReadsFileCreateBench        | benchCreate          | 2   | 1000  | 5   | 1.315mb  | 2,494.703ops/s   | ±3.50%  |
| ReadsFileCreateBench        | benchCreate          | 3   | 1000  | 5   | 1.315mb  | 2,373.190ops/s   | ±2.14%  |
| ReadsFileCreateBench        | benchCreate          | 4   | 1000  | 5   | 1.315mb  | 2,417.885ops/s   | ±0.33%  |
| ReadsFileCreateBench        | benchCreate          | 5   | 1000  | 5   | 1.315mb  | 2,269.290ops/s   | ±1.16%  |
| ReadsFileCreateBench        | benchCreate          | 6   | 1000  | 5   | 1.315mb  | 2,347.354ops/s   | ±4.75%  |
| ReadsFileCreateBench        | benchCreate          | 7   | 1000  | 5   | 1.315mb  | 2,030.786ops/s   | ±14.51% |
| CachesFileCreateBench       | benchCreate          | 0   | 1000  | 5   | 1.315mb  | 3,329.945ops/s   | ±3.37%  |
| CachesFileCreateBench       | benchCreate          | 1   | 1000  | 5   | 1.315mb  | 3,431.956ops/s   | ±0.54%  |
| CachesFileCreateBench       | benchCreate          | 2   | 1000  | 5   | 1.315mb  | 3,324.178ops/s   | ±1.60%  |
| CachesFileCreateBench       | benchCreate          | 3   | 1000  | 5   | 1.315mb  | 3,156.075ops/s   | ±2.96%  |
| CachesFileCreateBench       | benchCreate          | 4   | 1000  | 5   | 1.315mb  | 3,284.473ops/s   | ±2.38%  |
| CachesFileCreateBench       | benchCreate          | 5   | 1000  | 5   | 1.315mb  | 3,048.189ops/s   | ±1.29%  |
| CachesFileCreateBench       | benchCreate          | 6   | 1000  | 5   | 1.315mb  | 2,945.867ops/s   | ±2.30%  |
| CachesFileCreateBench       | benchCreate          | 7   | 1000  | 5   | 1.315mb  | 2,880.577ops/s   | ±1.79%  |
| CachesJsonCreateBench       | benchCreate          | 0   | 1000  | 5   | 1.315mb  | 3,509.448ops/s   | ±1.29%  |
| CachesJsonCreateBench       | benchCreate          | 1   | 1000  | 5   | 1.315mb  | 3,497.998ops/s   | ±0.67%  |
| CachesJsonCreateBench       | benchCreate          | 2   | 1000  | 5   | 1.315mb  | 3,530.635ops/s   | ±1.36%  |
| CachesJsonCreateBench       | benchCreate          | 3   | 1000  | 5   | 1.315mb  | 3,405.949ops/s   | ±0.76%  |
| CachesJsonCreateBench       | benchCreate          | 4   | 1000  | 5   | 1.315mb  | 3,424.857ops/s   | ±0.54%  |
| CachesJsonCreateBench       | benchCreate          | 5   | 1000  | 5   | 1.315mb  | 3,098.171ops/s   | ±0.72%  |
| CachesJsonCreateBench       | benchCreate          | 6   | 1000  | 5   | 1.315mb  | 3,106.300ops/s   | ±1.98%  |
| CachesJsonCreateBench       | benchCreate          | 7   | 1000  | 5   | 1.315mb  | 2,914.060ops/s   | ±0.96%  |
+-----------------------------+----------------------+-----+-------+-----+----------+------------------+---------+
caendesilva commented 2 years ago

Cool! Here are the results as XML if anyone wants it :)

https://gist.github.com/caendesilva/7a9327ab9562566db5d552397c2ad260