events list returns with big delay

gabest11 commented 5 months ago

https://github.com/roleoroleo/yi-hack-MStar/blob/42d08f9ff709ffa8b6f179a21a53e143e61b9e30/src/www/httpd/cgi-bin/eventsdir.sh#L20

The repeated calls to date slows down this loop from 0.x seconds to 5-9 seconds, with about 60 directories. I tried removing datetime from the json output, that's how I compared it. It's only a 32GB sd card, about three days worth of recordings.

Maybe cut the number of date calls into half like this, could this work? I'm not sure about the -u switch, if it applies to the input or the output. It only brings it down to 3-4 seconds though.

        FS00="${f:0:4}-${f:5:2}-${f:8:2} ${f:11:2}:00"
        if [[ $(get_config EVENTS_TIME) == "autodetect" ]] ; then
            FS00E=$(date -u -d "$FS00" +"%s")
        elif [[ $(get_config EVENTS_TIME) == "local" ]] ; then
            FS00E=$(date -d "$FS00" +"%s")
        elif [[ $(get_config EVENTS_TIME) == "gmt" ]] ; then
            FS00E=$(date -u -d "$FS00" +"%s")
        fi
        FL=$(date +%YY%mM%dD%HH -d "@$FS00E")

        FS00="${f:0:4}-${f:5:2}-${f:8:2} ${f:11:2}:00"
        if [[ $(get_config EVENTS_TIME) == "autodetect" ]] ; then
            FL=$(date -u -d "$FS00" +%YY%mM%dD%HH)
        elif [[ $(get_config EVENTS_TIME) == "local" ]] ; then
            FL=$(date -d "$FS00" +%YY%mM%dD%HH)
        elif [[ $(get_config EVENTS_TIME) == "gmt" ]] ; then
            FL=$(date -u -d "$FS00" +%YY%mM%dD%HH)
        fi

I also use this script directly, not just in a browser, to programatically get the list the recorded files. The date is already in the directory/file name, so I don't even have any use for the datetime element. A bit troublesome to me since the last update.

My other idea, moving $(get_config EVENTS_TIME) outside the loop into a variable, has no noticable effect. It's just the date calls.

timelapse.sh has a similar problem.

roleoroleo commented 5 months ago

I think the problem is get_config function. Please try this code:

#!/bin/sh

CONF_FILE="etc/system.conf"
YI_HACK_PREFIX="/home/yi-hack"

get_config()
{
    key=$1
    grep -w $1 $YI_HACK_PREFIX/$CONF_FILE | cut -d "=" -f2
}

printf "Content-type: application/json\r\n\r\n"
printf "{\"records\":[\n"

ET=$(get_config EVENTS_TIME)
COUNT=`ls -r /tmp/sd/record | grep H -c`
IDX=1
for f in `ls -r /tmp/sd/record | grep H`; do
    if [ ${#f} == 14 ]; then
        FS00="${f:0:4}-${f:5:2}-${f:8:2} ${f:11:2}:00"
        if [[ "$ET" == "autodetect" ]] ; then
            FS00E=$(date -u -d "$FS00" +"%s")
        elif [[ "$ET" == "local" ]] ; then
            FS00E=$(date -d "$FS00" +"%s")
        elif [[ "$ET" == "gmt" ]] ; then
            FS00E=$(date -u -d "$FS00" +"%s")
        fi
        FL=$(date +%YY%mM%dD%HH -d "@$FS00E")
        printf "{\n"
        printf "\"%s\":\"%s\",\n" "datetime" "Date: ${FL:0:4}-${FL:5:2}-${FL:8:2} Time: ${FL:11:2}:00"
        printf "\"%s\":\"%s\"\n" "dirname" "$f"
        if [ "$IDX" == "$COUNT" ]; then
            printf "}\n"
        else
            printf "},\n"
        fi
        IDX=$(($IDX+1))
    fi
done

printf "]}\n"

gabest11 commented 4 months ago

Better, but not that much.

http://192.168.0.24:8080/cgi-bin/eventsdir.sh 0.51s
http://192.168.0.24:8080/cgi-bin/eventsdir_getconfig.sh 3.78s
http://192.168.0.24:8080/cgi-bin/eventsdir_original.sh 7.41s
http://192.168.0.24:8080/cgi-bin/eventsdir.sh 0.33s
http://192.168.0.24:8080/cgi-bin/eventsdir_getconfig.sh 4.26s
http://192.168.0.24:8080/cgi-bin/eventsdir_original.sh 10.25s
http://192.168.0.24:8080/cgi-bin/eventsdir.sh 0.41s
http://192.168.0.24:8080/cgi-bin/eventsdir_getconfig.sh 4.47s
http://192.168.0.24:8080/cgi-bin/eventsdir_original.sh 7.65s
http://192.168.0.24:8080/cgi-bin/eventsdir.sh 0.34s
http://192.168.0.24:8080/cgi-bin/eventsdir_getconfig.sh 4.03s
http://192.168.0.24:8080/cgi-bin/eventsdir_original.sh 7.60s
http://192.168.0.24:8080/cgi-bin/eventsdir.sh 0.35s
http://192.168.0.24:8080/cgi-bin/eventsdir_getconfig.sh 3.33s
http://192.168.0.24:8080/cgi-bin/eventsdir_original.sh 7.68s
http://192.168.0.24:8080/cgi-bin/eventsdir.sh 0.34s
http://192.168.0.24:8080/cgi-bin/eventsdir_getconfig.sh 3.31s
http://192.168.0.24:8080/cgi-bin/eventsdir_original.sh 6.58s

This is my version (eventsdir.sh above) with just the dir name to time conversion. It could be even faster with get_config removed from the loop, I mean, if it was used, but it's also commented out.

COUNT=`ls -r /tmp/sd/record | grep H -c`
IDX=1
for f in `ls -r /tmp/sd/record | grep H`; do
    if [ ${#f} == 14 ]; then
#        FS00="${f:0:4}-${f:5:2}-${f:8:2} ${f:11:2}:00"
#        if [[ $(get_config EVENTS_TIME) == "autodetect" ]] ; then
#            FS00E=$(date -u -d "$FS00" +"%s")
#        elif [[ $(get_config EVENTS_TIME) == "local" ]] ; then
#            FS00E=$(date -d "$FS00" +"%s")
#        elif [[ $(get_config EVENTS_TIME) == "gmt" ]] ; then
#            FS00E=$(date -u -d "$FS00" +"%s")
#        fi
#        FL=$(date +%YY%mM%dD%HH -d "@$FS00E")
        printf "{\n"
#        printf "\"%s\":\"%s\",\n" "datetime" "Date: ${FL:0:4}-${FL:5:2}-${FL:8:2} Time: ${FL:11:2}:00"
        printf "\"%s\":\"%s\",\n" "datetime" "Date: ${f:0:4}-${f:5:2}-${f:8:2} Time: ${f:11:2}:00"
        printf "\"%s\":\"%s\"\n" "dirname" "$f"
        if [ "$IDX" == "$COUNT" ]; then
            printf "}\n"
        else
            printf "},\n"
        fi
        IDX=$(($IDX+1))
    fi
done

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

roleoroleo / yi-hack-MStar

events list returns with big delay #543