Feature request - only read specific sections of ini file

[amtssp]

Hi. First thank you for providing shini it is working fine.

I would like to request a new feature, because if we have a long ini file it takes a long time to read, and we often only need to read a specific section of the ini file. So could you add the possibility to only read the variables in a specified section?

Thanks

[wallyhall]

Absolutely - if this project is seeing this much attention, I'll happily spend the time trying to optimise it!

As soon as possible, I'll investigate general optimisations (to increase the speed overall) and, as you suggest, a specific enhancement to deal with only the section in question.

[amtssp]

Thank you very much.

We have included your "Shini" script in our small free audio player for Raspberry - the piCorePlayer: https://sites.google.com/site/picoreplayer/home

[wallyhall]

That's really cool :-) May I ask for an example copy of the INI file you're parsing, and an example of the operation(s) you're performing on it?

I've made some fairly aggressive changes for bash >= version 3 and zsh (I'm not clear on the version compatibility for zsh), which has pushed the performance to almost 7x faster (in my tests).

Regarding processing a specific section, am I understanding correctly that you'd like to be able to specify shini_parse ... "SOMEBLOCK" so it only parses the content of that specific section?

[wallyhall]

I've included PHP's shipped php.ini as a performance test, and included test.sh which accepts a single argument for the specific INI section to parse. (It simply skips everything else.) Not specifying the argument parses the entire file as normal.

The overall performance increase is very significant with the earlier changes:

Before

time bash test_perf.sh
real    0m16.372s

After

time bash test_perf.sh
real    0m0.789s    

(Note, zsh seems to have the edge on performance in the current implementation.)

Including specifying the section, further gains can be made:

time bash ./test_perf.sh > /dev/null
real    0m0.838s

time bash ./test_perf.sh PHP > /dev/null
real    0m0.396s

time bash ./test_perf.sh opcache > /dev/null
real    0m0.309s

time bash ./test_perf.sh curl > /dev/null
real    0m0.282s

[amtssp]

This is great :-) Here is our script that among many things also use shini to read a lot of audio DAC.conf files in order to make us able to use the variables in the DACs

#!/bin/sh

# Version: 3.20 2017-03-12
#   Added check for /tmp/dropdown.cfg in pcp_sound_card_dropdown. SBP.
#   Added shini conf parser. SBP.
#   Suppress already mounted message during do_rebootstuff newconfig process. PH.

# Version: 3.11 2017-01-20
#   Fixed the issue with shairport not working when using USB cards. SBP.

# Version: 3.10 2016-12-11
#   Original. SBP.

pcp_debug_log "$0 - Running pcp-soundcard-functions..."

DACLOCATION="/usr/local/etc/pcp/cards"     # <=== Should be in pcp-functions? GE.
#The below line is for development only.
#DACLOCATION="/home/tc/www/cgi-bin/cards"
############

. /usr/local/bin/shini.sh

#========================================================================================
# Check if we use the special Audio kernel or not. As we load special overlays if using Audio kernel
#----------------------------------------------------------------------------------------
pcp_AudioKernel_status() {
    sudo  uname -a | grep pcpAudioCore > /dev/null 2>&1 && KERNELVERSION=Audiokernel || KERNELVERSION=Officialkernel
}
pcp_AudioKernel_status

#========================================================================================
# Disable/re_enable analog audio
#----------------------------------------------------------------------------------------
pcp_disable_analog() {
    sudo sed -i 's/^dtparam=audio=on/#dtparam=audio=on/g' $CONFIGTXT
    sudo sed -i 's/^audio_pwm_mode=2/#audio_pwm_mode=2/g' $CONFIGTXT
}

pcp_re_enable_analog() {
    pcp_disable_analog
    sudo sed -i "/dtparam=audio=on/c\dtparam=audio=on" $CONFIGTXT
    sudo sed -i "/audio_pwm_mode=2/c\audio_pwm_mode=2" $CONFIGTXT
}

#========================================================================================
# Remove Audio dtoverlays
#----------------------------------------------------------------------------------------
pcp_disable_i2s() {
    pcp_re_enable_analog
        # unset DTOVERLAY both the old one that is removed on chooseoutput.cgi and the one from config.cfg to clear all traces of audio dtoverlays
        shini_parse ${DACLOCATION}/${ORIG_AUDIO}.conf
        [ x"" != x"${DTOVERLAY}" ] && sed -i '/dtoverlay='"$DTOVERLAY"'/d' $CONFIGTXT

        shini_parse ${DACLOCATION}/${AUDIO}.conf
        [ x"" != x"${DTOVERLAY}" ] && sed -i '/dtoverlay='"$DTOVERLAY"'/d' $CONFIGTXT
}

#========================================================================================
# Add Audio dtoverlays
#----------------------------------------------------------------------------------------
pcp_enable_i2s() {
    pcp_re_enable_analog
    pcp_selected_soundcontrol
    [ x"$SPARAMS1" != x"" ] && PAR1=','"$SPARAMS1"'' || PAR1="$SPARAMS1"
    [ x"$SPARAMS2" != x"" ] && PAR2=','"$SPARAMS2"'' || PAR2="$SPARAMS2"
    [ x"$SPARAMS3" != x"" ] && PAR3=','"$SPARAMS3"'' || PAR3="$SPARAMS3"
    [ x"$SPARAMS4" != x"" ] && PAR4=','"$SPARAMS4"'' || PAR4="$SPARAMS4"
    [ x"$SPARAMS5" != x"" ] && PAR5=','"$SPARAMS5"'' || PAR5="$SPARAMS5"

    [ x"$DTOVERLAY" != x"" ] && sudo echo dtoverlay=$DTOVERLAY$PAR1$PAR2$PAR3$PAR4$PAR5  >> $CONFIGTXT
}

#========================================================================================
# Enable/disable HDMI settings in config.txt
#----------------------------------------------------------------------------------------
pcp_disable_HDMI() {
    sed -i '/hdmi_drive=2/d' $CONFIGTXT
    sed -i '/hdmi_force_hotplug=1/d' $CONFIGTXT
    sed -i '/hdmi_force_edid_audio=1/d' $CONFIGTXT
}

pcp_enable_HDMI() {
    pcp_disable_i2s
    sudo echo hdmi_drive=2 >> $CONFIGTXT
    sudo echo hdmi_force_hotplug=1 >> $CONFIGTXT
    sudo echo hdmi_force_edid_audio=1 >> $CONFIGTXT
    sudo amixer cset numid=3 2 >/dev/null 2>&1
}

#========================================================================================
# Section that controls loading of DAC overlays
#----------------------------------------------------------------------------------------
pcp_read_chosen_audio() {
    # if noumount, disk is probably already mounted....supress output
    [ "$1" == "noumount" ] && pcp_mount_mmcblk0p1 > /dev/null 2>&1 || pcp_mount_mmcblk0p1
    pcp_disable_HDMI
    pcp_disable_i2s
    pcp_enable_i2s
    # This is to allow no unmount of mmcblk0p1 during booting with newconfig.cfg
    [ "$1" != "noumount" ] && pcp_umount_mmcblk0p1
}

#========================================================================================
# Control of sound card routines:
#  - Needs to be populated for each type of soundcard.
#  - Some are without filter options.
#  - USB cards will probably never be supported this way as they are so different.
#----------------------------------------------------------------------------------------
# HELP - To find the controls:
#  - amixer -c 0 scontrols
#  - aplay -l
#----------------------------------------------------------------------------------------
pcp_generic_card_control() {
    case "$GENERIC_CARD" in
        TI51XX)
            amixer -c $CARD scontrols | grep -q 'DSP Program' && DSPfound="yes" || DSPfound="no" 
            if [ "$DSPfound" = "yes" ]; then
                DSP="DSP Program,0"
                FILTER1="Low latency IIR with de-emphasis"
                FILTER2="FIR interpolation with de-emphasis"
                FILTER3="High attenuation with de-emphasis"
                FILTER4="Fixed process flow"
                FILTER5="Ringing-less low latency FIR"
            fi
            ACTUAL_VOL=$(amixer -c $CARD sget $SSET | grep "Right: Playback" | awk '{ print $5 }' | tr -d "[]%")
            ACTUAL_DB=$(amixer -c $CARD sget $SSET | grep "Right: Playback" | awk '{ print $6 }' | tr -d "[]")
            ACTUAL_FILTER=$(amixer -c $CARD sget 'DSP Program,0' | grep "Item0:" | awk '{ print $2 }' | tr -d "'")

            case "$DSPFILTER" in
                FILTER1) FILTER="$FILTER1" ;;
                FILTER2) FILTER="$FILTER2" ;;
                FILTER3) FILTER="$FILTER3" ;;
                FILTER4) FILTER="$FILTER4" ;;
                FILTER5) FILTER="$FILTER5" ;;
            esac

            # Logic to make checked radiobuttons - needs to clear it otherwise we have two FILTERS_CHECK checked.
            FILTER1_CHECK=""
            FILTER2_CHECK=""
            FILTER3_CHECK=""
            FILTER4_CHECK=""
            FILTER5_CHECK=""

            case "$ACTUAL_FILTER" in
                Low) FILTER1_CHECK="checked" ;;
                FIR) FILTER2_CHECK="checked" ;;
                High) FILTER3_CHECK="checked" ;;
                Fixed) FILTER4_CHECK="checked" ;;
                Ringing-less) FILTER5_CHECK="checked" ;;
            esac
        ;;

        HIFIBERRY_AMP)
            ACTUAL_VOL=$(amixer -c $CARD sget $SSET | grep "Mono:" | awk '{ print $3 }' | tr -d "[]%")
        ;;

        ONBOARD)
            ACTUAL_VOL=$(amixer -c $CARD sget $SSET | grep "Mono: Playback" | awk '{ print $4 }' | tr -d "[]%")
            ACTUAL_DB=$(amixer -c $CARD sget $SSET | grep "Mono: Playback" | awk '{ print $5 }' | tr -d "[]")
        ;;

#       ES9023)
#           CARD="ALSA"
#           SSET="PCM"
#           ACTUAL_VOL=$(amixer -c $CARD sget $SSET | grep "Mono: Playback" | awk '{ print $4 }' | tr -d "[]%")
#           ACTUAL_DB=$(amixer -c $CARD sget $SSET | grep "Mono: Playback" | awk '{ print $5 }' | tr -d "[]")
#           TEXT="Choosing 384k will enable 352k and 384k sample rates. Selecting bclk_ratio_int_div will use bclk_ratio=50 for 16/24bps and bclk_ratio=100 for 32bps media when sample rate is a multiple of 8kHz and less than 192kHz. Which causes the selection of the 19M2 OSC as the parent for the PCM clock with an integer divider, rather than PLLD with fractional divider and MASH noise shaping."
#           PARAMS1="384k"
#           PARAMS2="bclk_ratio_int_div"
#       ;;
    esac
}

#========================================================================================
# This will get us all available AUDIO DACs
#----------------------------------------------------------------------------------------
AUDIO_OPTIONS_EXTRA=$(ls "$DACLOCATION" | grep ".conf" | awk -F'.conf' '{ print $1 }')

#========================================================================================
# Section that populate sound card drop-down list on Squeezelite.cgi
#----------------------------------------------------------------------------------------
pcp_sound_card_dropdown() {
    pcp_blank_audio_conf
    . $CONFIGCFG
    if [ -e /tmp/dropdown.cfg ]; then
        sudo sed -i 's/:selected:/:notselected:/' /tmp/dropdown.cfg
        sudo sed -i "/${AUDIO}:/s/:notselected:/:selected:/" /tmp/dropdown.cfg

    else
        CONF_AUDIO="$AUDIO"
        pcp_blank_audio_conf
        for i in $AUDIO_OPTIONS_EXTRA; do
            AUDIO=$i
            pcp_soundcontrol
            CHECKED="notselected"
            [ "$AUDIO" = "$CONF_AUDIO" ] && CHECKED="selected"
            if [ x"$LISTNAME" != x ]; then
            sudo echo "${AUDIO}:${CHECKED}:${LISTNAME}:${RPI_MODEL}" >> /tmp/dropdown.cfg
            fi
        done
    fi
}

#========================================================================================
# This will get us available options from the DAC
#----------------------------------------------------------------------------------------
old_pcp_soundcontrol() {
    pcp_blank_audio_conf
    echo "$AUDIO_OPTIONS_EXTRA" | grep -q "$AUDIO" && . "$DACLOCATION"/"$AUDIO".conf
}

# Declare a handler for parsed variables.  This is required.
__shini_parsed() {
#   printf "  %s_%s='%s'\n" "$1" "$2" "$3"
    eval \ $1_$2='"$3"'

    if [ "$KERNELVERSION" = "Audiokernel" ]; then
        if [ "$1" = "PCPAUDIOCORE" ]; then
#           printf "  %s_%s='%s'\n" "$1" "$2" "$3"
            eval \ $2='"$3"'
        fi
    fi

    if [ "$KERNELVERSION" = "Officialkernel" ]; then
        if [ "$1" = "PCPCORE" ]; then
#           printf "  %s_%s='%s'\n" "$1" "$2" "$3"
            eval \ $2='"$3"'
        fi
    fi

    if [ "$1" = "COMMON" ]; then
#       printf "  %s_%s='%s'\n" "$1" "$2" "$3"
        eval \ $2='"$3"'
    fi
}

pcp_soundcontrol() {
    # Parse
    shini_parse ${DACLOCATION}/${AUDIO}.conf
}

#========================================================================================
# This will get us available options from the selected DAC
#----------------------------------------------------------------------------------------
pcp_selected_soundcontrol() {
    pcp_blank_audio_conf
    . "$CONFIGCFG"
    shini_parse ${DACLOCATION}/${AUDIO}.conf
}

pcp_blank_audio_conf() {
    # Remove left-over from previous audio DAC
    CARD=""
    OUTPUT=""
    ALSA_PARAMS=""
    GENERIC_CARD=""
    CONTROL_PAGE=""
    LISTNAME=""
    DTOVERLAY=""
    SSET=""
    PARAMS1=""
    PARAMS2=""
    PARAMS3=""
    PARAMS4=""
    PARAMS5=""
    RPI_MODEL=""
    I=1
    while true; do
        eval TEST="\${TEXT${I}}"
        if [ x"$TEST" != x ]; then 
            eval TEXT${I}=""
            I=$((I+1))
        else
            break
        fi
    done
    }

Here is a sample DAC.conf file

[COMMON]
CARD="sndrpihifiberry"
LISTNAME="HiFiBerry DAC+ Light"
OUTPUT="hw:CARD=sndrpihifiberry"
ALSA_PARAMS="80:4::1"
RPI_MODEL="HAT_ALL"

[PCPAUDIOCORE]
GENERIC_CARD="ES9023"
DTOVERLAY="hifiberry-dacpluslight-es9023-audio"
TEXT1="<b>384k</b> will enable 352k and 384k sample rates."
TEXT2="<b>bclk_ratio_int_div</b> will use bclk_ratio=50 for 16/24bps and bclk_ratio=100 for 32bps media when sample rate is a multiple of 8kHz and less than 192kHz. Which causes the selection of the 19M2 OSC as the parent for the PCM clock with an integer divider, rather than PLLD with fractional divider and MASH noise shaping."
PARAMS1=""
PARAMS2="384k"
PARAMS3="bclk_ratio_int_div"
CONTROL_PAGE="soundcard_control.cgi"

[PCPCORE]
DTOVERLAY="hifiberry-dac"
CONTROL_PAGE=""

Please note that we are using ash #!/bin/sh as we are building on piCore which is using busybox

[amtssp]

Yes we would like to be able to only parse a specific part of the ini file (for other parts of our player) - so your additions are highly appreciated.

[amtssp]

Sorry - but the new shini script do not run on our system. This function stalls it all:

shini_regex_replace()
{
    if [ "${BASH_VERSINFO}" -ge 3 ] || [ -n "$ZSH_VERSION" ]; then
        [[ "$1" =~ $2 ]] && shini_retval=${BASH_REMATCH[1]} || shini_retval="$1"
        return 0
    fi

    shini_retval="$(printf '%s' "$1" | sed -E "s/$2/\1/")"  # If you have isses on older systems,
    # it may be the non-newer POSIX compliant sed.
    # -E should be enabling extended regex mode portably.
}

If I change it to this it runs, but ..:

shini_regex_replace()
{
#    if [ "${BASH_VERSINFO}" -ge 3 ] || [ -n "$ZSH_VERSION" ]; then
#        [[ "$1" =~ $2 ]] && shini_retval=${BASH_REMATCH[1]} || shini_retval="$1"
#        return 0
#    fi

    shini_retval="$(printf '%s' "$1" | sed -E "s/$2/\1/")"  # If you have isses on older systems,
    # it may be the non-newer POSIX compliant sed.
    # -E should be enabling extended regex mode portably.
}

[wallyhall]

Ah, I've not tested t under ash. Which version specifically are you using? I'll see if I can fix the incompatibility.

[wallyhall]

Ok. Could you let me know you shell version? (Specifically the shell version the script is being ran under.)

Sent from my iPhone

On 6 May 2017, at 13:43, amtssp notifications@github.com wrote:

Sorry - but the new shini script do wont run on our system. This function stalls it all:

shini_regex_replace() { if [ "${BASH_VERSINFO}" -ge 3 ] || [ -n "$ZSH_VERSION" ]; then [[ "$1" =~ $2 ]] && shini_retval=${BASH_REMATCH[1]} || shini_retval="$1" return 0 fi

shini_retval="$(printf '%s' "$1" | sed -E "s/$2/\1/")"  # If you have isses on older systems,
# it may be the non-newer POSIX compliant sed.
# -E should be enabling extended regex mode portably.

} If I change it to this it runs, but ..:

shini_regex_replace() {

if [ "${BASH_VERSINFO}" -ge 3 ] || [ -n "$ZSH_VERSION" ]; then

[[ "$1" =~ $2 ]] && shini_retval=${BASH_REMATCH[1]} || shini_retval="$1"

return 0

fi

shini_retval="$(printf '%s' "$1" | sed -E "s/$2/\1/")"  # If you have isses on older systems,
# it may be the non-newer POSIX compliant sed.
# -E should be enabling extended regex mode portably.

} — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[amtssp]

Sorry for the delay. Is this what you were asking for?

tc@piCorePlayer:~$ ash --help BusyBox v1.24.2 (2016-09-08 12:09:43 UTC) multi-call binary.

[amtssp]

Hi. Thanks for the changes now shini parse the file fine using busybox ash. It is still very slow (on a Raspberry pi Zerro Wireless) here is the result:

tc@piCorePlayer:~/www/cgi-bin$ time ash ./test_perf.sh > /dev/null
Command exited with non-zero status 1
real    2m 1.38s
user    0m 25.24s
sys     0m 52.27s

Do you think that you can improve the speed on ash as well? I noticed this discussion here: [http://stackoverflow.com/questions/21010882/how-to-match-regexp-with-ash]

I don't know if the last answer on that page will improve speed?

[wallyhall]

No problem, I realised there was a bug breaking the "fallback" functionality (which as you rightly notice, is extremely slow even on the best of hardware), which I've patched.

I too had read that article, but haven't had time to confirm it works yet. I'll give it a whirl this weekend and let you know. (I'm able to test locally against dash, which I believe is a fork of ash - and certainly seems to exhibit the same behaviour as you're seeing.)

I'll let you know ASAP. Thanks for your patience... :-)

[amtssp]

Hi there. Is there anything I can do to help you in the process of making shini faster using busybox or ash?

[wallyhall]

I've had an extensive think about the problem and tried to debug where the duration is being spent.

I suspected - and I believe have now proved beyond doubt - that the slowness comes from calling grep and sed (which I'm using as a fallback for non bash/zsh, which both have regex match and replacement built in. ksh has regex matching, but is relying on sed for replacements.)

Thus the performance is significantly faster in zsh and bash, slower in ksh and abysmal in anything else.

I'm not that familiar with ash, so if you can think of any way to do what I'm achieving with those grep and sed calls in a more "builtin" manner, that'd help more than a lot.

I've spent some time trying to use named pipes, so there's only one grep/sed process spawned at the start for each of the regexes used - but as you can imagine it's super hacky (you need around 22 named pipes and 11 children processes to get anywhere close).

Otherwise I'm afraid I'm out of ideas, except to suggest switching to another library which you could call via anything else you have available (Python/PHP) or knock something up in C. (I'm happy to give that a go if you're willing to compile+ship it with your project. Bit of an ache though!)

There might be some further micro-optimisations around the order of execution etc, but I don't think it'll make enough of a difference for as long as grep/sed are being used.

FYI - these are the performance gains by using builtins from other shells. Note it is still sped up by searching a specific section rather than the entire file - but still awfully slow in dash:

$ wc -l tests/php.ini 
1917 tests/php.ini

$ time zsh ./test_perf.sh > /dev/null
real    0m0.595s

$ time bash ./test_perf.sh > /dev/null
real    0m0.838s

$ time ksh ./test_perf.sh > /dev/null
real    0m2.901s

$ time zsh ./test_perf.sh opcache > /dev/null
real    0m0.237s

$ time bash ./test_perf.sh opcache > /dev/null
real    0m0.313s

$ time ksh ./test_perf.sh opcache > /dev/null
real    0m0.543s

$ time dash test_perf.sh > /dev/null
real    0m17.930s

$ time dash test_perf.sh opcache > /dev/null
real    0m5.839s

[amtssp]

Thanks for trying. I noticed this on the busybox web page: https://www.busybox.net/BusyBox.html

expr

    expr EXPRESSION

    Print the value of EXPRESSION to stdout

    EXPRESSION may be:

            ARG1 | ARG2     ARG1 if it is neither null nor 0, otherwise ARG2
            ARG1 & ARG2     ARG1 if neither argument is null or 0, otherwise 0
            ARG1 < ARG2     1 if ARG1 is less than ARG2, else 0. Similarly:
            ARG1 <= ARG2
            ARG1 = ARG2
            ARG1 != ARG2
            ARG1 >= ARG2
            ARG1 > ARG2
            ARG1 + ARG2     Sum of ARG1 and ARG2. Similarly:
            ARG1 - ARG2
            ARG1 * ARG2
            ARG1 / ARG2
            ARG1 % ARG2
            STRING : REGEXP         Anchored pattern match of REGEXP in STRING
            match STRING REGEXP     Same as STRING : REGEXP
            substr STRING POS LENGTH Substring of STRING, POS counted from 1
            index STRING CHARS      Index in STRING where any CHARS is found, or 0
            length STRING           Length of STRING
            quote TOKEN             Interpret TOKEN as a string, even if
                                    it is a keyword like 'match' or an
                                    operator like '/'
            (EXPRESSION)            Value of EXPRESSION

    Beware that many operators need to be escaped or quoted for shells. Comparisons are arithmetic if both ARGs are numbers, else lexicographical. Pattern matches return the string matched between \( and \) or null; if \( and \) are not used, they return the number of characters matched or 0.

So can the expr command be useful?

[wallyhall]

OK, with the greatest emphasis - please don't get overly excited because I've not extensively tested this (and haven't yet found a way to get myself easy access to a busybox install!)

But, if I've correctly implemented what you found, these are the new speeds on dash/ash (verses the 17s before):

$ time dash test.sh 
test.sh: 145: test.sh: cannot open tests/nonexistent.ini: No such file
real    0m0.707s

$ time bash test.sh 
./shini.sh: line 81: tests/nonexistent.ini: No such file or directory
real    0m0.187s

I'll commit and push the (experimental) change to a separate branch and post details here momentarily.

[wallyhall]

https://github.com/wallyhall/shini/compare/expr-experimental

Simple change, I had tried expr before but obviously wasn't doing it right. The documentation you referenced appears to work correctly in dash at least - so here's to hoping!

[amtssp]

tc@piCorePlayer:~/www/cgi-bin$ time ash ./test_perf.sh > /dev/null
./test_perf.sh: ./shini.sh: line 59: syntax error: bad substitution
Command exited with non-zero status 2
real    0m 0.02s
user    0m 0.01s
sys     0m 0.00s

If I comment that section again like this:

shini_regex_replace()
{
#    if [ -n "$BASH_VERSINFO" ] && [ "${BASH_VERSINFO}" -ge 3 ] || \
#       [ -n "$ZSH_VERSION" ]; then
#        [[ "$1" =~ $2 ]] && shini_retval=${BASH_REMATCH[1]} || shini_retval="$1"
#        return 0
#    fi

    shini_retval="$(printf '%s' "$1" | sed -E "s/$2/\1/")"  # If you have isses on older systems,
    # it may be the non-newer POSIX compliant sed.
    # -E should be enabling extended regex mode portably.
}

I get these results - so still a speed improvement compared to before but there is still room for improvement:

tc@piCorePlayer:~/www/cgi-bin$ time ash ./test_perf.sh > /dev/null
Command exited with non-zero status 1
real    1m 43.32s
user    0m 15.56s
sys     0m 32.17s

So this section needs further changes. I think it is this shini_retval=${BASH_REMATCH[1]} that upset busybox.

If I test this section I get this error:

tc@piCorePlayer:~/www/cgi-bin$ expr 'abc123' : '^abc[0-9]*'
expr: warning: '^abc[0-9]*': using '^' as the first character
of a basic regular expression is not portable; it is ignored
6

I managed to get the time down to "real 1m 27.95s" if I changed the script to this (t I think I then force it to use your new method?

shini_regex_match()
{
    # $KSH_VERSION (I'm told) only exists on ksh 93 and above, which supports regex matching.
    if [ -n "$BASH_VERSINFO" ] && [ "$BASH_VERSINFO" -ge 3 ] || \
       [ -n "$ZSH_VERSION" ] || \
       [ -n "$KSH_VERSION" ]; then
        [[ "$1" =~ $2 ]] && return 0 || return 1
    fi

    # ash/dash, and possibly others
#    if [ "$EXPR_REGEX" -eq 0 ]; then
        expr "$1" : "$2" > /dev/null 2>&1
        return $?
#    fi

    printf '%s' "$1" | grep -qe "$2"
    return $?
}

shini_regex_replace()
{
#    if [ -n "$BASH_VERSINFO" ] && [ "${BASH_VERSINFO}" -ge 3 ] || \
#       [ -n "$ZSH_VERSION" ]; then
#        [[ "$1" =~ $2 ]] && shini_retval=${BASH_REMATCH[1]} || shini_retval="$1"
#        return 0
#fi

shini_retval="$1"
#    shini_retval="$(printf '%s' "$1" | sed -E "s/$2/\1/")"  # If you have isses on older systems,
    # it may be the non-newer POSIX compliant sed.
    # -E should be enabling extended regex mode portably.
}

tc@piCorePlayer:~/www/cgi-bin$ time ash ./test_perf.sh > /dev/null
Command exited with non-zero status 1
real    1m 27.95s
user    0m 15.64s
sys     0m 33.83s

[wallyhall]

Regarding this issue:

shini_retval=${BASH_REMATCH[1]}

You are correct. It appears dash behaves differently, allowing the [1] syntax. busybox's ash refuses to process the script at all with it there. I'll have to figure a way around that.

I think when you "force" it to use the expr mode, all you've done (at least, testing on this busybox VM image I found!) is remove the if condition. So you're seeing a ~15s increase in performance just because there's no if statement.

I'm guessing the rasberrypi hardware (is that what you're running the test on?) is just woefully less powerful than the busybox VM I have (which completes in 3.8s).

The test_perf.sh script is running against a fairly large INI file (1917 lines), what kind of performance do you get if you replace /tests/php.ini with a smaller example (like the one you'll be using for your project)?

Assuming the if condition is running that slowly, you'll get further performance gains by completely stripping down that first function:

shini_regex_match()
{
        expr "$1" : "$2" > /dev/null 2>&1
        return $?
}

[amtssp]

Thanks for working on this - it is good you found a busybox environtment. I noticed that if I added a debug notice like this:

shini_regex_replace()
{
#    if [ -n "$BASH_VERSINFO" ] && [ "${BASH_VERSINFO}" -ge 3 ] || \
#       [ -n "$ZSH_VERSION" ]; then
#        [[ "$1" =~ $2 ]] && shini_retval=${BASH_REMATCH[1]} || shini_retval="$1"
#        return 0
#fi

echo "we are here 3"
    shini_retval="$(printf '%s' "$1" | sed -E "s/$2/\1/")"  # If you have isses on older systems,
    # it may be the non-newer POSIX compliant sed.
    # -E should be enabling extended regex mode portably.
}

The the terminal shows that this function get called two or three times for each variable?

tc@piCorePlayer:~/www/cgi-bin$ time ash ./test_perf.sh
..
..
..
[opcache] opcache.validate_permission = 0
we are here 3
we are here 3
[opcache] opcache.validate_root = 0
we are here 3
we are here 3
we are here 3
[curl] curl.cainfo =
we are here 3
we are here 3
we are here 3
[openssl] openssl.cafile =
we are here 3
we are here 3
[openssl] openssl.capath =
Command exited with non-zero status 1

[amtssp]

Hi Did you find time to look at this issue:

Regarding this issue:

shini_retval=${BASH_REMATCH[1]}

You are correct. It appears dash behaves differently, allowing the [1] syntax. busybox's ash refuses to process the script at all with it there. I'll have to figure a way around that.