search

openssl / project

Tracking of project related issues
1 stars 0 forks source link

Identify performance regressions from 1.1.1 #577

Open nhorman opened 1 month ago

nhorman commented 1 month ago

To properly attempt to improve openssl performance, we need a starting point. this Epic is meant to capture comprehensive performance data immediately so that we have a better sense of where we are in terms of our performance behavior on a variety of environments. Specifically the matrix we should capture is:

Note the above matrix, is representative of our current primary platform list

To be consistent, results for each test should be captured in the following format for each test thread count number of iterations openssl 1.1.1 per operation avg usec openssl 3.0.0 avg usec master usec std deviation

With incremental thread counts (preferably in powers of two, up two and one step beyond the maximum number of cpus on a given platform

for reference, a script such as the following may be used to capture the required data:

#!/bin/bash
set -x
CMD=$1
shift 1
BASEARGS=$1

if [ "$CMD" == "./handshake" -o "$CMD" == "./x509storeissuer" ]
then
    BASEARGS=~/git/openssl/test/certs
fi

for i in $PERF_MASTER_LIB $PERF_111_LIB $PERF_300_LIB 
do
    SUFFIX=$(basename $i)
    rm -f $CMD.$SUFFIX
    for j in 1 2 4 8 16 32 64
    do
        declare -a USECS_ARRAY=()
        for k in $(seq 1 1 50)
        do
            export EVP_FETCH_TYPE=MD:MD5
            echo "Running $CMD against $i with $j threads, iteration $k"
            ARGS="-t $BASEARGS $j"
            OUTPUT=$(/bin/sh -c "LD_LIBRARY_PATH=$i $CMD $ARGS")
            USECS=$(echo $OUTPUT | awk '{print $1}')
            USECS_ARRAY+=($USECS)
        done
        TUSECS=0
        for value in "${USECS_ARRAY[@]}"
        do
            TUSECS=$(dc -e"$value $TUSECS + p")
        done
        AVG_USECS=$(dc -e"6 k $TUSECS ${#USECS_ARRAY[@]} / p")
        declare -a USEC_DEV_ARRAY=()
        for value in "${USECS_ARRAY[@]}"
        do
            DEVIATION=$(dc -e"6 k $value $AVG_USECS - p" | sed -e"s/-/_/g")
            DEVIATION=$(dc -e"6 k $DEVIATION $DEVIATION * p")
            USEC_DEV_ARRAY+=($DEVIATION)
        done
        echo ${USEC_DEV_ARRAY[@]}
        SUM_DEVIATIONS=0
        for value in "${USEC_DEV_ARRAY[@]}"
        do
            SUM_DEVIATIONS=$(dc -e"6 k $value $SUM_DEVIATIONS + p")
        done
        STD_DEVIATION=$(dc -e"6 k $SUM_DEVIATIONS ${#USECS_ARRAY[@]} 1 - / v p")
        echo "$j $AVG_USECS $STD_DEVIATION" >> $CMD.$SUFFIX
    done
done

results should be reported in this epic.

Following that report, we need to analyze the data to identify performance regressions and root cause them.

### Tasks
- [ ] https://github.com/openssl/project/issues/579
- [ ] https://github.com/openssl/project/issues/586
- [ ] https://github.com/openssl/project/issues/670
- [ ] https://github.com/openssl/project/issues/571
- [ ] https://github.com/openssl/project/issues/641
- [ ] https://github.com/openssl/project/issues/642
- [ ] https://github.com/openssl/project/issues/638
- [ ] https://github.com/openssl/project/issues/639
- [ ] https://github.com/openssl/project/issues/578
- [ ] https://github.com/openssl/project/issues/580
- [ ] https://github.com/openssl/project/issues/581
- [ ] https://github.com/openssl/project/issues/582
- [ ] https://github.com/openssl/project/issues/583
- [ ] https://github.com/openssl/project/issues/584
- [ ] https://github.com/openssl/project/issues/585
- [ ] https://github.com/openssl/project/issues/669
- [ ] https://github.com/openssl/project/issues/587
- [ ] https://github.com/openssl/project/issues/644
- [ ] https://github.com/openssl/project/issues/650
- [ ] Add tasks to improve performance here after #589 is complete
Sashan commented 4 weeks ago

Below is slightly modified version of script provided by @nhorman. The summary of changes is as follows:

edit from Jun 10th: there was a script which was buggy. debuged version can be found at the next comment.

Sashan commented 4 weeks ago

the loop which iterates over threads is wrong. it currently uses only 8 threads. this is the relict of my debugging effort which I used to roughly estimate the test run. Updated code below currently runs on both MacOSes. It iterates over all performance tools we have.

edit from jun 10th 2024: this is the version of run-perf.sh which worked for me. the results are found ./results directory once script is done. The version from Niel did 50 iterations, this version comes with 25 iterations for each tool invocation. You need to adjust TOOLS_PATH, OPENSSL_SRC so it matches your environement.

#!/bin/bash
set -x
CMD=$1
shift 1
BASEARGS=$1

TOOLS_PATH=/Users/sashan/tools.sashan/perf
OPENSSL_SRC=/Users/sashan/openssl/
CERT_DIR=$OPENSSL_SRC/test/certs
mkdir -p results

for TOOL in evp_fetch randbytes handshake sslnew newrawkey rsasign \
        x509storeissuer providerdoall rwlocks pkeyread ; do
    CMD=$TOOLS_PATH/$TOOL
    if [ "$TOOL" == "handshake" -o "$TOOL" == "x509storeissuer" ]
    then
        BASEARGS=$CERT_DIR
    else
        BASEARGS=""
    fi

    for i in master 3.0 1.1.1
    do  
        pushd `pwd`
        cd $TOOLS_PATH
        openssl.env $i;
        make clean
        make
        popd
        SUFFIX=$i
        for j in 1 2 4 8 16 32 64
        do  
            # format loop and key loops are hack. the thing is
            # the script assumes we do have a single tool to
            # perform particular test. Unfortunately this not
            # the case for pkeyread which executes different
            # tests according to command line options. So
            # those two loops (format, key) iterate over
            # options to exercise tests and capture results for
            # pkeuread test.
            for format in pem der ; do
                for key in dh dhx dsa ec rsa x25519 ; do
                    declare -a USECS_ARRAY=()
                    for k in $(seq 1 1 25)
                    do  
                        export EVP_FETCH_TYPE=MD:MD5
                        ARGS="--terse $BASEARGS $j"
                        # also note pkeyread has slightly different output, which needs
                        # to be rectified in dedicated branch by awk/sed.
                        if [ "$TOOL" == "pkeyread" ] ; then
                            echo "Running $CMD against $i with $j threads key $key, format $format, iteration $k"
                            OUTPUT=$(/bin/sh -c "LD_LIBRARY_PATH=/Users/sashan/openssl.binaries/openssl-$i/lib $CMD -f $format -k $key -t $j")
                            USECS=$(echo $OUTPUT | awk '{print $3}' |sed -e 's/us$//g')
                        else
                            echo "Running $CMD against $i with $j threads, iteration $k"
                            OUTPUT=$(/bin/sh -c "LD_LIBRARY_PATH=/Users/sashan/openssl.binaries/openssl-$i/lib $CMD $ARGS")
                            USECS=$(echo $OUTPUT | awk '{print $1}')
                        fi
                        USECS_ARRAY+=($USECS)
                    done
                    # this stuff comes from Niel I'm leaving it untouched
                    # as my math/stats is weak.
                    TUSECS=0
                    for value in "${USECS_ARRAY[@]}"
                    do  
                        TUSECS=$(dc -e"$value $TUSECS + p")
                    done
                    AVG_USECS=$(dc -e"6 k $TUSECS ${#USECS_ARRAY[@]} / p")
                    declare -a USEC_DEV_ARRAY=()
                    for value in "${USECS_ARRAY[@]}"
                    do  
                        DEVIATION=$(dc -e"6 k $value $AVG_USECS - p" | sed -e"s/-/_/g")
                        DEVIATION=$(dc -e"6 k $DEVIATION $DEVIATION * p")
                        USEC_DEV_ARRAY+=($DEVIATION)
                    done
                    echo ${USEC_DEV_ARRAY[@]}
                    SUM_DEVIATIONS=0
                    for value in "${USEC_DEV_ARRAY[@]}"
                    do  
                        SUM_DEVIATIONS=$(dc -e"6 k $value $SUM_DEVIATIONS + p")
                    done
                    STD_DEVIATION=$(dc -e"6 k $SUM_DEVIATIONS ${#USECS_ARRAY[@]} 1 - / v p")
                    if [ "$TOOL" == "pkeyread" ] ; then
                        echo "$j $AVG_USECS $STD_DEVIATION" >> results/$TOOL.$key.$format.$SUFFIX
                    else
                        echo "$j $AVG_USECS $STD_DEVIATION" >> results/$TOOL.$SUFFIX
                    fi
                    # only pkeyread needs to keep going through all keys
                    if [ "$TOOL" != "pkeyread" ] ; then
                        break;      # key
                    fi
                done #key
                # only pkeyread needs to keep going through all formats
                if [ "$TOOL" != "pkeyread" ] ; then
                    break;  # format
                fi
            done #format
        done # thredas
    done # version
done # tool
t8m commented 1 week ago

Results from linux x86_64:

evp_fetch

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 N/A N/A .243486 0 .123404 0 .125677 .001000 .126979 0 .126502 .001732
2 20 N/A N/A .704476 .042743 .419351 .027037 .452826 .016340 .422580 .025553 .360000 .051176
4 20 N/A N/A 2.260913 .274131 .870317 .026172 .883020 .022000 .896154 .023151 .868581 .022891
8 20 N/A N/A 6.806402 .381472 2.138642 .218467 1.875936 .215575 1.974716 .227525 1.811663 .180988
16 20 N/A N/A 13.840573 .609219 4.253584 .452609 4.196466 .569612 3.987304 .465942 3.959855 .492218
32 20 N/A N/A 24.696751 .951850 7.998405 .887415 8.413182 .688326 8.559601 .619138 7.800762 .715181
64 20 N/A N/A 45.606433 2.660230 14.607361 1.362538 16.338117 1.719713 15.947229 1.579854 15.157291 1.586333
128 20 N/A N/A 91.930684 2.910952 27.502788 1.142453 29.040132 1.219169 29.210566 .907218 27.335172 1.287565

randbytes

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 .784748 .004898 .701072 .006244 .536025 .005830 .376813 .005744 .373586 .004123 .377904 .004795
2 20 .931830 .019723 1.127062 .026702 .791333 .032341 .436255 .008544 .440112 .009486 .446011 .010392
4 20 1.069898 .044136 1.807690 .024248 1.400260 .036359 .505732 .025139 .501737 .023685 .496096 .013490
8 20 3.291035 .341499 3.432947 .162249 3.463844 .344688 .939772 .081719 .939681 .086226 .917963 .073000
16 20 9.454166 .519102 7.148176 .234243 6.866830 .388048 1.923636 .137669 1.892326 .130980 1.899463 .132204
32 20 20.232543 .733135 15.176948 .672649 15.053895 .565734 4.181749 .356178 4.121765 .207108 4.138167 .256169
64 20 41.562752 1.596420 43.574004 1.214741 42.428801 2.009129 7.679429 .384352 7.776608 .386891 7.673164 .402134
128 20 1648.875358 202.149546 1788.267415 476.000146 1733.998777 492.926268 14.569299 1.467719 15.177463 1.289802 14.667983 1.321493

handshake

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 897.967605 2.225298 1166.698955 1.343342 1065.342370 1.474801 1041.238430 1.256190 1029.675105 1.223939 1039.280240 1.313018
2 20 907.298395 3.260319 1215.395395 16.536096 1109.004740 3.031915 1080.615545 3.851716 1071.357675 4.178266 1062.513715 3.164873
4 20 955.728975 4.542752 1610.876035 35.069862 1186.131345 2.726274 1150.725100 2.553094 1134.629680 5.805519 1135.386905 3.078813
8 20 994.608785 2.265746 2982.657390 181.681671 1266.741120 20.665656 1224.781205 15.830346 1240.021220 16.437957 1260.183685 11.176064
16 20 1099.061985 6.944712 7151.454595 491.566248 1725.972735 25.191207 1587.731740 54.996907 1566.368475 37.450252 1602.697430 47.361537
32 20 1473.496186 39.845350 12955.116239 819.633908 2571.882513 65.517615 2260.928619 56.013888 2153.508995 29.507217 2292.641548 39.426555
64 20 1914.986922 48.259338 23695.823273 641.671593 4517.925134 148.549389 3659.136211 122.768223 3692.361465 130.239969 3863.194625 132.865528
128 20 3960.796385 151.399448 57510.820441 1879.318029 10188.726547 349.694482 7275.108326 123.152944 7153.238108 135.704018 7538.782184 255.137023

sslnew

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 1.578622 .007000 1.444965 .008544 1.449492 .032140 2.167341 .015099 2.252070 .019519 2.223618 .011000
2 20 1.762163 .060613 2.115975 .168101 1.766006 .041880 2.613258 .045803 2.998732 .059799 2.947290 .071365
4 20 5.939933 .135048 5.098922 .181160 3.011874 .097483 4.263507 .131901 4.048158 .103363 4.037002 .114568
8 20 19.277297 .972452 19.802931 1.072344 6.013077 .148616 8.890266 .569149 9.016344 .230911 9.142283 .271775
16 20 35.177239 1.380442 44.644090 1.670526 14.297900 .655475 16.669751 .736685 18.383768 .537734 18.072226 .664584
32 20 88.858023 2.675930 92.063144 3.361828 29.151799 1.194109 36.081559 1.855390 39.487740 1.887723 40.135090 1.248522
64 20 185.244815 5.741851 201.726114 8.614712 52.655954 2.001395 65.318045 2.645729 68.685541 2.575200 70.931132 3.260461
128 20 3812.022276 251.311062 2672.399144 367.723091 95.629072 3.002067 116.782440 5.447722 126.393361 4.265855 126.291732 5.819844

newrawkey

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 .111917 .004582 2.770079 .010440 2.273060 .007211 2.156657 .011000 1.722392 .024062 1.721051 .016522
2 20 .119483 .007810 4.693164 .249685 3.474624 .258110 3.274195 .251288 2.574186 .109219 2.520522 .091640
4 20 .133212 .009327 6.642451 .460789 5.159560 .283853 4.369389 .363505 3.440170 .258791 3.367448 .250089
8 20 .153734 .019026 14.205711 2.016568 11.107342 .866393 6.808856 .977585 5.217891 .936950 4.711515 .536432
16 20 .191715 .021633 31.596720 7.715760 25.064733 1.663495 11.089907 2.154340 8.819457 1.260351 8.773756 1.501707
32 20 .243405 .046776 59.345766 5.395573 57.746206 5.757416 19.646013 4.844942 16.597077 2.903826 16.003254 2.887616
64 20 .332723 .043646 132.879657 25.844835 135.402203 7.512421 39.193370 5.876775 34.446957 6.312611 34.758772 5.747280
128 20 .198936 .058608 5853.978907 1834.220988 4706.163554 1170.776920 76.938808 8.255037 72.977545 5.636181 72.838476 5.129213

rsasign

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 35.755469 .371837 36.492999 .051039 35.689345 .069021 36.003165 .105104 36.058115 .111350 36.348527 .216282
2 20 36.625809 .287673 37.890363 .273043 37.093691 .274899 37.837589 .169793 37.969191 .115637 37.743014 .148104
4 20 40.664752 .765316 42.728531 2.174464 41.450107 1.039285 41.971614 .491591 41.938001 .314469 41.951530 .361224
8 20 49.219190 1.072657 51.627973 1.652196 50.250302 1.196760 49.700612 .686017 49.662666 1.143240 50.279787 1.187827
16 20 73.685610 3.834260 71.460576 2.756623 70.417453 2.129127 68.842100 1.302427 69.405031 1.797244 69.135970 1.095780
32 20 491.577672 204.868476 147.846465 4.080327 145.936095 2.652486 146.577009 4.279461 148.707091 8.406270 164.924943 67.836254
64 20 1636.137541 11.298241 1497.601766 50.506319 1481.698011 42.866253 1475.452696 38.181185 1593.917065 50.647882 1620.549756 32.209631
128 20 3283.436327 19.847019 3103.346998 135.409776 3056.153777 132.244461 2999.346676 97.922071 3271.346923 24.845703 3286.893255 24.318498

x509storeissuer

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 1.017828 .019157 2.787840 .019621 2.649109 .024859 2.649838 .023086 2.667246 .034741 1.227468 .006928
2 20 1.166389 .033421 4.010627 .056780 3.563404 .087028 3.770803 .053000 3.519093 .084787 1.693910 .061530
4 20 1.604414 .054101 5.723358 .607031 4.504791 .071091 6.607145 .769779 5.447681 .344100 2.876898 .081651
8 20 4.946803 .227778 9.926157 .683994 7.140090 .499841 11.906435 2.273988 10.625887 .950738 7.358870 .217393
16 20 13.505683 .483910 22.122016 1.246412 13.477602 1.005203 26.962361 1.976453 25.803594 2.231513 17.596014 2.038236
32 20 28.884046 .531684 46.018785 1.665254 31.141246 .862524 138.089082 7.437566 119.031874 20.755009 93.097021 5.010768
64 20 58.490383 1.065191 109.833724 2.688914 80.696906 2.600861 337.535540 7.981681 312.286783 7.286363 224.049457 8.074328
128 20 1026.032618 214.713349 500.291567 183.364277 440.479222 114.587383 739.875957 64.487284 715.929613 95.910274 540.320115 7.075483

providerdoall

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 N/A N/A .312843 .045000 .219697 .042178 .186198 .042320 .174785 .045088 .201104 .032893
2 20 N/A N/A 1.567398 .104288 1.234398 .107363 .744180 .068891 .804646 .115325 .778763 .101375
4 20 N/A N/A 3.619061 .296585 2.756703 .222528 1.474103 .195517 1.674269 .231896 1.578674 .180141
8 20 N/A N/A 7.700505 .631558 6.440538 .583454 2.915459 .301292 2.948623 .179164 3.019905 .354379
16 20 N/A N/A 17.225584 2.736702 13.417214 1.070594 5.937969 .700950 5.908887 .906259 5.997185 .785396
32 20 N/A N/A 46.264520 2.615603 34.735076 4.262569 12.051944 1.048344 11.878799 1.137359 12.721666 1.563902
64 20 N/A N/A 97.188706 4.836893 81.491849 4.286475 24.267441 2.217605 23.689274 2.808288 23.211365 2.546745
128 20 N/A N/A 1197.867974 1050.030392 293.324126 282.611426 31.995605 2.262560 29.706528 3.831646 29.699265 3.920187

rwlocks-rlock

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 0 0 0 0 0 0 0 0 0 0 0 0
2 20 2.084954 .030740 2.078892 .029325 .845364 .920720 2.084977 .035341 .318142 1.178264 2.118905 .028053
4 20 .538477 .196397 .482107 .221921 .461693 .225048 .483760 .251342 .476211 .235408 .566216 .212000
8 20 .327685 .028017 .345039 .037121 .341348 .042755 .340384 .041376 .328197 .031320 .344774 .049223
16 20 .313549 .028460 .316191 .019442 .321787 .020223 .319470 .014966 .319301 .017916 .320236 .016643
32 20 .279673 .015459 .289872 .017349 .281189 .020396 .282942 .019697 .287452 .016970 .286675 .018303
64 20 .302050 .011661 .298320 .010677 .297114 .007280 .302354 .011180 .302594 .010295 .303578 .007745
128 20 .302731 .006403 .297120 .005744 .300040 .009591 .306142 .009695 .306845 .009695 .307916 .008062

rwlocks-wlock

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 .012287 .001414 .012340 .001414 .012849 .001000 .013091 .001000 .012267 .001732 .012336 .001414
2 20 2.160425 .320639 2.042598 .144024 2.087254 .022181 2.100206 .029034 2.078665 .035707 1.032296 .954516
4 20 .430782 .261371 .421700 .202007 .488072 .232447 .430117 .226773 .389972 .227385 .378467 .164830
8 20 .193471 .048989 .193820 .038444 .170548 .029342 .195965 .043185 .184925 .044395 .218394 .093674
16 20 .125442 .010392 .122851 .008426 .129342 .015394 .129474 .010344 .126383 .015132 .124772 .009797
32 20 .104035 .011958 .101520 .005830 .106365 .010862 .105000 .012041 .106676 .014491 .104400 .006480
64 20 .085252 .003605 .084127 .003464 .084082 .003605 .084101 .004123 .084975 .003605 .084225 .004000
128 20 .079254 .003162 .078366 .003605 .080096 .003464 .079588 .003162 .079281 .002828 .078887 .003316

pkeyread-dh-der

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 1250.997165 2.775555 1322.015610 1.346655 1268.190325 1.000685 1257.192960 1.420321 1326.828135 3.682804 1333.256295 3.461984
2 20 1259.357305 2.065016 1418.870390 43.030661 1298.107540 18.323572 1297.098220 18.734945 1289.750940 27.903169 1298.928025 15.192588
4 20 1319.096430 3.128889 1534.695025 46.337621 1381.281480 17.706446 1352.963875 17.766167 1346.398655 13.790705 1351.028900 18.229899
8 20 1341.541220 3.712705 1677.735620 58.396999 1431.819530 9.171769 1366.040465 7.951822 1367.920020 8.390914 1372.595145 8.406344
16 20 1356.159335 5.468834 2485.933100 128.541537 1443.399575 7.855456 1385.592575 5.001003 1384.241890 4.065323 1389.338920 5.786529
32 20 1374.559539 15.661886 4411.438019 316.291542 1881.458321 150.270200 1804.668695 166.097306 1793.247349 165.993205 1819.093026 141.955784
64 20 2013.577940 37.832543 8620.002289 263.539325 2673.995531 227.734368 2446.189286 135.044564 2453.469108 120.016833 2513.602403 126.797137
128 20 3404.126251 98.475745 20582.963973 681.593296 3839.294847 127.641886 3524.849886 61.798442 3567.698773 100.386011 3574.340066 44.147415

pkeyread-dhx-der

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 255.771875 .315448 333.582105 .498428 275.811690 .356794 263.545750 .340091 262.904205 .401084 263.355870 .570565
2 20 261.377875 .701741 477.462650 23.870578 286.457210 .969247 266.471190 1.258353 266.731675 .921973 267.859215 1.130576
4 20 272.082750 1.018672 697.144505 21.200706 304.363660 2.160581 280.553130 1.212949 280.932975 1.088047 282.232685 1.675678
8 20 276.512930 1.444864 1239.633180 32.715138 325.250750 7.148883 291.159150 1.681342 292.548030 2.157716 293.762900 2.415092
16 20 283.232110 4.247018 2392.480090 113.418669 400.730610 22.296715 302.879530 2.891013 303.259115 2.896651 306.952650 8.964163
32 20 302.323367 9.248207 4353.436451 310.131504 836.863173 123.417753 436.197529 41.409085 427.455406 36.119622 444.001697 37.957361
64 20 456.865943 14.802992 9070.384250 578.722525 1174.198825 87.029640 599.924403 73.550483 608.911728 73.548293 610.645038 86.751163
128 20 657.628970 29.401494 22551.460611 776.779698 2355.229435 407.876032 858.151448 69.596186 907.539942 107.286518 872.444358 46.414155

pkeyread-dsa-der

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 255.903295 .453937 417.113370 .399634 290.137010 .561035 268.149085 .493063 267.627555 .488549 269.140175 .568447
2 20 262.589795 .660340 763.097510 25.826358 304.844895 4.620495 272.090465 .868326 271.986305 1.396468 272.937985 1.279302
4 20 274.099950 .871825 1263.172085 38.930096 351.486610 6.821855 286.378220 1.014090 287.052565 1.426366 288.086765 1.391754
8 20 277.758875 1.797585 2926.021710 141.877481 434.841790 16.029435 297.159835 3.047254 299.084320 3.047325 299.481230 2.830215
16 20 283.864605 2.323885 5884.734365 475.354890 751.770760 33.022272 311.612325 3.839655 312.750945 8.040317 312.608215 3.590606
32 20 401.890610 60.551144 9974.204986 804.204263 1464.271395 166.405492 444.364821 40.197824 442.445177 46.255398 459.198212 41.202351
64 20 484.885255 47.522368 17665.550119 1276.797676 2408.405354 217.789393 642.124970 75.208330 615.341112 44.313578 667.530642 83.664404
128 20 700.495648 64.567153 36051.274945 2967.379505 4629.906655 281.845273 906.197656 75.552947 948.875672 89.804277 924.537677 67.086328

pkeyread-ec-der

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 17.124565 .305370 113.779945 .330563 40.583485 .397163 26.722755 .404953 26.830195 .384480 26.941735 .311663
2 20 18.148460 .714490 311.694160 13.535358 56.006925 3.095907 29.343075 .765846 29.682640 .827670 29.743020 .621897
4 20 20.137870 1.074074 787.831425 26.174571 90.291740 3.533958 33.380095 1.162221 33.782980 1.211930 33.790085 1.270233
8 20 23.800765 1.913230 1664.132310 69.801162 138.633035 6.736376 39.413405 1.887818 38.959920 1.800974 39.920125 2.432896
16 20 24.536770 3.092610 3209.804750 146.526984 298.806650 40.246407 57.510965 5.893131 55.215885 4.473660 58.824275 4.670033
32 20 36.858835 7.843897 5831.395696 517.372466 613.283496 90.754910 138.124061 21.222110 143.744598 25.962936 130.729717 21.367718
64 20 63.301224 15.470364 11584.563052 459.529707 1034.138082 106.029261 204.334773 23.279419 202.855807 24.319304 200.037962 27.927920
128 20 100.197582 40.647095 28433.550420 968.079267 3455.328060 2168.589544 352.345831 51.696620 377.908989 48.346504 381.502991 33.564313

pkeyread-rsa-der

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 4.350620 .440062 120.363500 .414857 36.761930 .343297 15.127995 .170008 14.960895 .410783 14.877785 .434980
2 20 5.291520 .785902 407.886260 13.680888 61.608205 5.400734 18.010180 .741194 18.019110 .735783 18.200140 .591133
4 20 6.680735 .920360 1021.279570 38.867570 154.858755 4.732353 22.435195 1.466403 22.404685 1.289143 22.271910 1.440637
8 20 11.007375 1.610668 2164.244245 77.721558 314.797650 17.628217 28.749465 2.118085 29.100105 2.444585 30.192435 2.451763
16 20 24.770325 3.287416 4121.361630 229.531564 653.780965 43.493103 57.250635 5.531840 54.072220 9.380357 56.234410 4.962247
32 20 50.863802 10.160088 7700.251956 609.844048 1226.615525 155.653842 132.216638 22.135969 139.329677 25.320758 125.473791 13.792101
64 20 105.755239 14.814370 14832.361569 574.771848 2216.970138 246.375611 221.410867 32.106184 219.743242 23.646970 224.838734 32.129246
128 20 203.005631 73.729086 34556.311629 1233.832958 5171.030463 1237.895975 411.621993 56.935244 443.103945 54.259943 446.643220 45.105517

pkeyread-x25519-der

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 31.648210 .420673 96.002650 .283358 48.273670 .376680 36.854385 .358145 36.821325 .355778 36.873195 .340166
2 20 32.812225 .596557 224.847450 11.745576 58.947465 1.336827 39.156275 .686070 39.390435 .743470 39.111550 .645521
4 20 34.896965 .804898 538.193135 14.239644 86.419395 2.477505 43.048485 .641177 42.888915 .847329 43.086475 .855748
8 20 37.531830 2.446947 1137.216770 36.989833 135.057180 6.229580 47.579180 1.739377 47.970395 2.025820 48.204110 1.852101
16 20 41.881995 2.230556 2126.466115 81.200824 278.411735 38.394225 62.665750 3.285511 62.638735 4.236618 61.470580 3.014235
32 20 50.534604 6.528192 4064.677675 252.129434 593.719458 127.431805 145.817901 29.148506 140.923197 21.584888 144.551852 24.684637
64 20 65.253746 12.386147 7940.320640 292.255110 966.580384 104.967506 184.570023 42.583489 179.912594 36.684039 171.686688 30.934182
128 20 68.019031 16.689182 20365.615753 472.330606 2586.045801 1055.913867 273.795910 43.461115 274.415595 39.426198 277.928515 48.546494

pkeyread-dh-pem

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 1255.854240 1.122266 1703.441175 2.969789 1331.205490 1.116172 1347.367520 1.857611 1277.733655 .915484 1284.651380 1.425810
2 20 1265.413200 2.269376 2101.468855 176.714467 1397.327580 30.732287 1329.953760 27.952816 1321.994645 8.675479 1329.962170 18.830446
4 20 1323.615065 4.567231 3559.547660 164.630436 1460.383475 11.987384 1398.303605 24.290203 1392.210135 16.679290 1398.551380 17.084920
8 20 1348.814325 7.211639 7681.046590 399.584112 1531.580865 17.914342 1429.034685 15.646504 1448.034655 58.026405 1443.766460 17.045987
16 20 1364.471250 10.608800 16270.041650 897.346580 1699.600075 40.692193 1452.060905 13.238577 1464.873990 21.518480 1463.320295 6.359588
32 20 1397.339526 50.332508 30128.033646 2030.996919 2916.695876 131.790444 1883.804298 209.172785 1918.786341 118.782236 1904.901881 158.297404
64 20 2036.694267 32.147694 54905.642431 3691.015027 4942.990908 215.738989 2443.029916 138.414631 2558.030861 116.609278 2558.980393 139.562674
128 20 3322.593947 161.318764 105391.815328 4262.660577 8919.010996 208.537898 3688.578406 108.827019 3767.053995 61.509112 3810.050934 120.464221

pkeyread-dhx-pem

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 259.669330 .640586 658.404275 .707241 326.755340 .529912 275.083990 .456157 277.317755 .635814 277.357330 .464790
2 20 265.724030 .644067 1432.501240 174.360258 354.478550 11.595853 281.929575 1.228782 282.501535 1.609578 288.823015 1.457581
4 20 277.120080 1.036543 3316.969190 89.493922 457.655625 6.292027 298.130995 1.553544 297.967520 1.655932 303.365760 1.315914
8 20 280.483330 1.484990 7783.674395 533.067364 680.296365 17.156771 311.553145 5.134650 312.657175 5.769123 309.915000 1.809192
16 20 287.423430 2.444204 16223.717825 1138.220039 1279.317590 61.716577 333.311360 10.822956 335.047195 10.735088 334.181170 10.316392
32 20 305.620367 10.135998 31405.831354 1735.526073 2455.158441 206.046294 562.047948 59.416772 569.482113 67.027878 592.029128 64.163033
64 20 461.114206 8.253753 54529.117033 3090.700885 4260.049358 414.792949 863.597541 126.207914 831.007872 141.775598 901.623352 150.879865
128 20 674.770530 81.633137 104872.736565 6180.397519 8202.129559 449.049855 1073.242573 80.653898 1185.844155 112.646838 1180.334399 82.357012

pkeyread-dsa-pem

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 259.365100 .568098 666.938415 1.441103 336.617570 1.398145 282.626740 .477139 285.062240 .472570 283.995915 .452442
2 20 265.982655 .685172 1412.026370 139.109922 370.817835 9.392394 290.084955 1.892949 293.998340 1.188762 295.089695 1.726137
4 20 276.800120 1.057196 3294.808690 102.118463 462.415200 9.321664 305.713440 1.531247 309.213460 1.675302 310.323155 1.309818
8 20 281.187025 1.449562 7658.491695 477.856160 666.639240 19.353618 321.942150 7.287636 320.405015 7.157106 318.049955 3.147393
16 20 290.190030 9.171569 14919.265205 1552.558283 1279.917015 85.881256 339.551455 7.530973 335.970450 6.373791 347.038910 14.407874
32 20 306.321580 10.870188 29612.294898 1558.147982 2507.094189 160.488101 586.838263 59.835864 568.447978 58.723771 610.264202 47.165971
64 20 462.107772 8.188063 54811.446442 3736.907736 4182.972178 377.396598 852.157817 168.457549 864.758822 136.891586 918.565734 177.156421
128 20 688.816613 76.146199 103647.126626 4737.413543 8293.456249 548.713210 1091.658930 113.101883 1154.233613 73.262000 1186.668824 61.530835

pkeyread-ec-pem

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 18.378810 .419761 422.599015 .459836 89.229455 .446378 37.160595 .434097 37.076035 .438941 37.238680 .346992
2 20 19.292850 .742583 1365.083590 45.484080 142.334715 7.952676 44.079360 1.056312 44.190445 1.040665 43.694310 .977323
4 20 21.513205 .967748 3400.321890 178.916831 281.878130 6.572441 50.518630 1.318835 51.238840 1.204375 51.141420 1.559287
8 20 24.232925 1.830540 7654.075170 481.141390 578.569655 10.424218 59.155440 2.852784 59.640215 2.508847 60.535535 2.887734
16 20 56.014705 6.920612 16302.195145 961.423874 1199.426685 42.647142 102.236410 20.622024 97.178275 15.101468 100.403905 12.338903
32 20 123.661301 11.667093 30772.768645 2020.453888 2189.928694 148.281252 269.734759 70.978121 273.674765 52.384320 261.046545 68.913508
64 20 238.318959 24.688371 55734.505637 2847.045831 4035.709514 427.965723 371.444735 78.407178 371.488316 45.243687 362.204279 45.883539
128 20 840.738251 436.501354 106427.615209 4518.396984 7938.857179 440.066689 672.769106 56.878387 694.330681 65.530317 698.677566 60.388746

pkeyread-rsa-pem

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 11.420505 .349691 417.957210 .656314 88.341610 .434031 38.089920 .432678 36.924230 .320526 38.624100 .413784
2 20 12.435785 .626047 1388.909365 37.228017 145.130595 9.877729 45.132835 .639985 44.378900 .968796 46.336395 .776348
4 20 13.885985 1.109749 3411.329400 102.112805 291.238375 6.416421 52.697430 1.333581 51.813750 1.362056 53.535735 1.516997
8 20 18.946265 2.581246 6816.726270 436.657228 611.371010 15.902151 64.245760 3.093877 63.754240 2.961470 64.510050 3.091631
16 20 56.899355 6.506297 13754.532540 1367.454259 1240.407640 48.378941 125.061190 22.728571 116.970565 23.174925 122.225840 22.868263
32 20 122.749760 11.323920 28457.811416 1971.327769 2196.793016 195.877916 284.789451 56.794198 289.240759 70.793039 294.013104 53.216985
64 20 235.872407 28.171014 57041.357439 2399.644130 4220.392093 438.610906 404.722531 46.488328 404.056503 55.313334 393.993456 42.467884
128 20 1150.763256 668.373519 104851.476443 4874.815818 7988.415466 502.530338 734.137648 62.142816 786.517316 49.837486 793.802244 47.619155

pkeyread-x25519-pem

thread count number of iterations openssl 1.1.1 per operation avg usec 1.1.1 std dev openssl 3.0 per operation avg usec 3.0 std dev openssl 3.1 per operation avg usec 3.1 std dev openssl 3.2 per operation avg usec 3.2 std dev openssl 3.3 per operation avg usec 3.3 std dev openssl master per operation avg usec master std dev
1 20 32.742910 .397856 423.354015 .634486 95.752460 .397071 44.333370 .423935 44.578225 .233508 44.737335 .285795
2 20 34.368825 .660791 1392.173685 58.709708 149.941925 9.496925 50.456120 .784121 51.342260 1.172837 51.481120 1.358294
4 20 35.743610 1.102233 3352.216075 110.449343 280.735940 9.448594 56.724470 1.416481 57.112905 1.234195 57.227170 1.188975
8 20 37.952570 1.293101 7589.319670 565.869002 605.516680 20.521111 63.234535 2.761193 64.214500 2.261131 64.119825 3.361999
16 20 46.274455 3.627630 16214.705315 1022.338512 1226.331340 58.248968 93.779210 15.531945 96.017845 12.953756 103.441695 21.578135
32 20 82.448602 12.709397 30733.571455 1663.620927 2139.982782 188.723770 240.779368 61.984170 253.149455 59.983026 261.375329 62.312790
64 20 150.771297 17.848477 57282.534793 2487.405207 4061.856538 343.423967 343.120098 59.392758 340.909549 41.433956 334.686554 40.850721
128 20 331.401463 204.973975 102940.634444 5922.000502 7789.357970 428.471329 573.038968 52.618381 628.112000 66.437069 621.116574 65.378627
t8m commented 1 week ago

My summary of the results - we are still 2-4 times slower on decoding keys and 2 times slower on handshake with high thread counts. The newrawkey results are also much slower but I think it is inevitable as the operation was really lightweight on 1.1.1 and it is not really possible to keep it so lightweight with providers.

Remarkable is randbytes where we are much faster now than in 1.1.1.

Thanks @Sashan and @vdukhovni for the work on the run-perf script.

t8m commented 1 week ago

We were really horrible on 3.0 however the current performance as of 3.3 and master in general is fairly OK and IMO acceptable with all the new features that the providers API accomplishes.

We are still missing some additional perf tests like testing a throughput of TLS connections.

nhorman commented 1 week ago

agreed, thank you @t8m, these are great results