imageRecentMax not used?

[peterb99]

The following section in the config.py file does not seen to be used in the .py code; imageRecentMax = 100 # 0=off Maintain specified number of most recent files in motionRecentDir imageRecentDir = "media/recent" # Default= "media/recent" save recent files directory path

My web page shows a huge number of files, so it takes quite long for a web page to build when initially accessed, Web Root is [ media ] Drive [ /dev/sda1 ] Mount_Point [ /home/pi/speed-camera ] Space_Used [ 23% 3.1G of 15G ] Space_Avail [ 11G ] SPEED-CAMERA Media Auto Refresh = 900 sec Listing All 14046 Files in /recent/

[pageauc]

A simple text search shows imageRecentMax variable used 5 times in speed-cam.py . This variable sets the number of recent entries to display in the default media/recent folder. when a new entry is created symbolic link the oldest entry is removed. You can increase or decrease value to suit. This will greatly speed up browsing recent images and I use it a lot unless I want to view something older. this value can be increase to 1000 - 2000 without too much performance penalty depending on the version of RPI.

The imageRecentDir is used 8 times in the speed-cam.py code. I would recommend you create subfolders for the speed images using the imageSubDirMaxFiles variable in config.py. I set mine to 2000. Once the max is reached a new subdirectory is created with a datetime folder name. This avoids having a single directory with 14000+ files in it that can be very slow to create a listing in the webserver. If you have problems with recent entries just delete all the symbolic links in the recent folder and a new ones will be created and managed up to the recent max setting.l

Hope this answers your question. Let me know Claude ...

On Wed, Jun 23, 2021 at 10:08 AM peterb99 @.***> wrote:

The following section in the config.py file does not seen to be used in the .py code; imageRecentMax = 100 # 0=off Maintain specified number of most recent files in motionRecentDir imageRecentDir = "media/recent" # Default= "media/recent" save recent files directory path

My web page shows a huge number of files, so it takes quite long for a web page to build when initially accessed, Web Root is [ media ] Drive [ /dev/sda1 ] Mount_Point [ /home/pi/speed-camera ] Space_Used [ 23% 3.1G of 15G ] Space_Avail [ 11G ] SPEED-CAMERA Media Auto Refresh = 900 sec Listing All 14046 Files in /recent/

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pageauc/speed-camera/issues/90, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNPKZBDX5SJFZUPSSMFZGDTUHTETANCNFSM47F4JZQQ .

-- YouTube Channel at https://www.youtube.com/user/pageaucp http://www.youtube.com/user/pageaucp GitHub Repository at https://github.com/pageauc

[peterb99]

Hello Claude

I need to dive into this a but deeper, it looks like the list just keeps growing including the number of symlinks.

Just upgrade from version 10 to 11 and noticed the following error messages,

speed_camera sqlite3 DB /home/pi/speed-camera/data/speed_cam.db 2021-06-23 15:38:06 ERROR speed_camera Failed: To INSERT Speed Data into TABLE speed 2021-06-23 15:38:06 ERROR speed_camera Err Msg: table speed has 29 columns but 27 values were supplied

Does this ring a bell? has the format of the database changed?

Peter

On 23 Jun 2021, at 15:48, Claude Pageau @.***> wrote:

A simple text search shows imageRecentMax variable used 5 times in speed-cam.py . This variable sets the number of recent entries to display in the default media/recent folder. when a new entry is created symbolic link the oldest entry is removed. You can increase or decrease value to suit. This will greatly speed up browsing recent images and I use it a lot unless I want to view something older. this value can be increase to 1000 - 2000 without too much performance penalty depending on the version of RPI.

The imageRecentDir is used 8 times in the speed-cam.py code. I would recommend you create subfolders for the speed images using the imageSubDirMaxFiles variable in config.py. I set mine to 2000. Once the max is reached a new subdirectory is created with a datetime folder name. This avoids having a single directory with 14000+ files in it that can be very slow to create a listing in the webserver. If you have problems with recent entries just delete all the symbolic links in the recent folder and a new ones will be created and managed up to the recent max setting.l

Hope this answers your question. Let me know Claude ...

On Wed, Jun 23, 2021 at 10:08 AM peterb99 @.***> wrote:

The following section in the config.py file does not seen to be used in the .py code; imageRecentMax = 100 # 0=off Maintain specified number of most recent files in motionRecentDir imageRecentDir = "media/recent" # Default= "media/recent" save recent files directory path

My web page shows a huge number of files, so it takes quite long for a web page to build when initially accessed, Web Root is [ media ] Drive [ /dev/sda1 ] Mount_Point [ /home/pi/speed-camera ] Space_Used [ 23% 3.1G of 15G ] Space_Avail [ 11G ] SPEED-CAMERA Media Auto Refresh = 900 sec Listing All 14046 Files in /recent/

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pageauc/speed-camera/issues/90, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNPKZBDX5SJFZUPSSMFZGDTUHTETANCNFSM47F4JZQQ .

-- YouTube Channel at https://www.youtube.com/user/pageaucp http://www.youtube.com/user/pageaucp GitHub Repository at https://github.com/pageauc — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pageauc/speed-camera/issues/90#issuecomment-866904008, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIOUIOPENZIEVFU4KP45KCDTUHX4BANCNFSM47F4JZQQ.

[pageauc]

Format of the DB has changed. I recommend you rename speed-camera folder to something else and perform a new install

eg

cd ~ mv speed-camera speed-camera-old

Then do a new curl install.

Claude ...

On Wed, Jun 23, 2021 at 11:12 AM peterb99 @.***> wrote:

Hello Claude

I need to dive into this a but deeper, it looks like the list just keeps growing including the number of symlinks.

Just upgrade from version 10 to 11 and noticed the following error messages,

speed_camera sqlite3 DB /home/pi/speed-camera/data/speed_cam.db 2021-06-23 15:38:06 ERROR speed_camera Failed: To INSERT Speed Data into TABLE speed 2021-06-23 15:38:06 ERROR speed_camera Err Msg: table speed has 29 columns but 27 values were supplied

Does this ring a bell? has the format of the database changed?

Peter

On 23 Jun 2021, at 15:48, Claude Pageau @.***> wrote:

A simple text search shows imageRecentMax variable used 5 times in speed-cam.py . This variable sets the number of recent entries to display in the default media/recent folder. when a new entry is created symbolic link the oldest entry is removed. You can increase or decrease value to suit. This will greatly speed up browsing recent images and I use it a lot unless I want to view something older. this value can be increase to 1000 - 2000 without too much performance penalty depending on the version of RPI.

The imageRecentDir is used 8 times in the speed-cam.py code. I would recommend you create subfolders for the speed images using the imageSubDirMaxFiles variable in config.py. I set mine to 2000. Once the max is reached a new subdirectory is created with a datetime folder name. This avoids having a single directory with 14000+ files in it that can be very slow to create a listing in the webserver. If you have problems with recent entries just delete all the symbolic links in the recent folder and a new ones will be created and managed up to the recent max setting.l

Hope this answers your question. Let me know Claude ...

On Wed, Jun 23, 2021 at 10:08 AM peterb99 @.***> wrote:

The following section in the config.py file does not seen to be used in the .py code; imageRecentMax = 100 # 0=off Maintain specified number of most recent files in motionRecentDir imageRecentDir = "media/recent" # Default= "media/recent" save recent files directory path

My web page shows a huge number of files, so it takes quite long for a web page to build when initially accessed, Web Root is [ media ] Drive [ /dev/sda1 ] Mount_Point [ /home/pi/speed-camera ] Space_Used [ 23% 3.1G of 15G ] Space_Avail [ 11G ] SPEED-CAMERA Media Auto Refresh = 900 sec Listing All 14046 Files in /recent/

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pageauc/speed-camera/issues/90, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABNPKZBDX5SJFZUPSSMFZGDTUHTETANCNFSM47F4JZQQ

.

-- YouTube Channel at https://www.youtube.com/user/pageaucp http://www.youtube.com/user/pageaucp GitHub Repository at https://github.com/pageauc — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/pageauc/speed-camera/issues/90#issuecomment-866904008>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AIOUIOPENZIEVFU4KP45KCDTUHX4BANCNFSM47F4JZQQ .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pageauc/speed-camera/issues/90#issuecomment-866924538, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNPKZAM73A4IAZEWD3XPGTTUH2XVANCNFSM47F4JZQQ .

-- YouTube Channel at https://www.youtube.com/user/pageaucp http://www.youtube.com/user/pageaucp GitHub Repository at https://github.com/pageauc

[peterb99]

Hi Claude

Problem found, when you enable option to add speed to image file name the purge mechanism does not work any more.

I made the following simple change( add ‘*’ in join) in speed-cam.py;

def deleteOldFiles(maxFiles, dirPath, prefix): """ Delete Oldest files gt or equal to maxfiles that match filename prefix """ try: fileList = sorted(glob.glob(os.path.join(dirPath, '' + prefix + '')), key=os.path.getmtime)

Regards Peter

On 23 Jun 2021, at 15:48, Claude Pageau @.***> wrote:

A simple text search shows imageRecentMax variable used 5 times in speed-cam.py . This variable sets the number of recent entries to display in the default media/recent folder. when a new entry is created symbolic link the oldest entry is removed. You can increase or decrease value to suit. This will greatly speed up browsing recent images and I use it a lot unless I want to view something older. this value can be increase to 1000 - 2000 without too much performance penalty depending on the version of RPI.

The imageRecentDir is used 8 times in the speed-cam.py code. I would recommend you create subfolders for the speed images using the imageSubDirMaxFiles variable in config.py. I set mine to 2000. Once the max is reached a new subdirectory is created with a datetime folder name. This avoids having a single directory with 14000+ files in it that can be very slow to create a listing in the webserver. If you have problems with recent entries just delete all the symbolic links in the recent folder and a new ones will be created and managed up to the recent max setting.l

Hope this answers your question. Let me know Claude ...

On Wed, Jun 23, 2021 at 10:08 AM peterb99 @.***> wrote:

The following section in the config.py file does not seen to be used in the .py code; imageRecentMax = 100 # 0=off Maintain specified number of most recent files in motionRecentDir imageRecentDir = "media/recent" # Default= "media/recent" save recent files directory path

My web page shows a huge number of files, so it takes quite long for a web page to build when initially accessed, Web Root is [ media ] Drive [ /dev/sda1 ] Mount_Point [ /home/pi/speed-camera ] Space_Used [ 23% 3.1G of 15G ] Space_Avail [ 11G ] SPEED-CAMERA Media Auto Refresh = 900 sec Listing All 14046 Files in /recent/

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pageauc/speed-camera/issues/90, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNPKZBDX5SJFZUPSSMFZGDTUHTETANCNFSM47F4JZQQ .

-- YouTube Channel at https://www.youtube.com/user/pageaucp http://www.youtube.com/user/pageaucp GitHub Repository at https://github.com/pageauc — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pageauc/speed-camera/issues/90#issuecomment-866904008, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIOUIOPENZIEVFU4KP45KCDTUHX4BANCNFSM47F4JZQQ.

[peterb99]

It seems image_max_files value needs to be>= imageRecentMax. If not file is delete while symlink remains, leaving symlink to point to non existing file and routine def deleteOldFiles(maxFiles, dirPath, prefix): to fail when trying to build a list of links in the recent directory

2021-06-24 15:21:36 INFO deleteOldFiles folder media/images/speed-20210623-1831 prefix speed- 2021-06-24 15:21:36 INFO deleteOldFiles maxFiles 200 list len 200 2021-06-24 15:21:36 INFO deleteOldFiles Removing oldest file media/images/speed-20210623-1831/40-speed-20210624-1132123.jpg 2021-06-24 15:21:36 ERROR deleteOldFiles Problem Reading Directory /home/pi/speed-camera/media/recent 2021-06-24 15:21:36 ERROR deleteOldFiles [Errno 2] No such file or directory: '/home/pi/speed-camera/media/recent/40-speed-20210624-1132123.jpg' 2021-06-24 15:21:36 ERROR deleteOldFiles Possibly symlink destination File Does Not Exist 2021-06-24 15:21:36 ERROR deleteOldFiles To Fix - Try Deleting All Files in recent folder /home/pi/speed-camera/media/recent

[peterb99]

Added the following code to speed_cam.py at line 220 """ PB Added Prevent image_max_files to be lower in value then to avoid having symlinks pointing to non excisting files """ if image_max_files < imageRecentMax: logging.warn(" image_max_file value ( %d) lower then imageRecentMax (%d), image_max_file set to (%d)",image_max_files,imageRecentMax,imageRecentMax) image_max_files = imageRecentMax

[peterb99]

Hi Claude Problem found, when you enable option to add speed to image file name the purge mechanism does not work any more. I made the following simple change( add ‘’ in join) in speed-cam.py; def deleteOldFiles(maxFiles, dirPath, prefix): """ Delete Oldest files gt or equal to maxfiles that match filename prefix """ try: fileList = sorted(glob.glob(os.path.join(dirPath, '' + prefix + '*')), key=os.path.getmtime) Regards Peter …

Would it be an idea to change the following code segment from: # Check if subdirectories configured

                                    # and create as required
                                    speed_path = subDirChecks(imageSubDirMaxHours,
                                                              imageSubDirMaxFiles,
                                                              image_path, image_prefix)
                                    # Create image file name prefix
                                    if image_filename_speed:
                                        speed_prefix = (str(int(round(ave_speed)))
                                                        + "-" + image_prefix)
                                    else:
                                        speed_prefix = image_prefix

to

\# Check if subdirectories configured
                                    # and create as required
                                    speed_path = subDirChecks(imageSubDirMaxHours,
                                                              imageSubDirMaxFiles,
                                                              image_path, image_prefix)
                                    # Create image file name prefix
                                    if image_filename_speed:
                                        image_prefix = (str(int(round(ave_speed)))
                                                        + "-" + image_prefix)
\ # Record log_time for use later in csv and sqlite
                                    log_time = datetime.datetime.now()
                                    # create image file name path
                                    filename = get_image_name(speed_path,
                                                              image_prefix)

[pageauc]

I pushed 11.05 of speed-cam.py and would like you to test the changes suggested above. Let me know if this resolves the problem.

[peterb99]

Hello Claude, Change works! Great, Can I suggest the following change to the code to address the max files values issue: At line 216 add; """ PB Added Prevent image_max_files to be lower in value then imageRecentMax to avoid having symlinks pointing to non existing files """ if image_max_files < imageRecentMax: logging.warn(" image_max_file value ( %d) lower then imageRecentMax (%d), image_max_file set to (%d)",image_max_files,imageRecentMax,imageRecentMax) image_max_files = imageRecentMax

[pageauc]

Sorry for the lack of testing and subsequent errors. I think I have solved the problem and my testing went OK. I now append ave_speed after the image_prefix if image_filename_speed is set. Recent folder is being maintained correctly per my testing. I did find another issue since file_format is hardcoded in subDirCheckMaxFiles I have made it check for all files not just .jpg since image_format can be png and also extensions can be upper or lower case. Hopefully this should resolve the issue.

Run menubox.sh UPGRADE menu pick to download the latest ver 11.07

Please let me know if problem is now resolve. Sorry for the previous lack of testing Claude ...

On Fri, Jun 25, 2021 at 4:23 AM peterb99 @.***> wrote:

Hello Claude, Change works! Great, Can I suggest the following change to the code to address the max files values issue: At line 216 add; """ PB Added Prevent image_max_files to be lower in value then imageRecentMax to avoid having symlinks pointing to non existing files """ if image_max_files < imageRecentMax: logging.warn(" image_max_file value ( %d) lower then imageRecentMax (%d), image_max_file set to (%d)",image_max_files,imageRecentMax,imageRecentMax) image_max_files = imageRecentMax

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pageauc/speed-camera/issues/90#issuecomment-868317374, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNPKZHIZR4DE7TAPPL5JIDTUQ4GTANCNFSM47F4JZQQ .

-- YouTube Channel at https://www.youtube.com/user/pageaucp http://www.youtube.com/user/pageaucp GitHub Repository at https://github.com/pageauc

[pageauc]

With regards to webserver pointing to non existent symlinks. This is a function of the web browser not being refreshed. Since new symlinks are created and older ones deleted the browser needs to be refreshed to read the new symlinks in recent and clear the removed sysmlinks. If there is a lot of activity the webserver refresh seconds would need to be reduced. Reducing it too much can affect speed camera performance since the recent folder structure would need to be read more regularly.

It would be possible to setup a buffer by setting the webserver file display shorter than the speed-cam recent max so that as new symlinks are created the deleted ones would not appear in the browser for a while. Eg speed-cam.py recent set to 100 and a webserver max recent set to eg 90. Since the web browser can display other folders the implementation might be a bit tricky.
I will think about this and see if there is a simpler solution

Please note the new speed-cam.py version is 11.07 and I believe will resolve the recent max issue when image_filename_speed is enabled.

Thanks for your work. Your feedback is appreciated. Claude ...

[pageauc]

Please let me know if this issue can now be closed.

[peterb99]

I have to back track on that it works, I was to soon with this. It looks like in python when you change a value of a global variable in a local subroutine the variable changed from global to local and as such the variable cannot be reference again global or local in the same subroutine before it is given a value. So I guess the code has to stay as it is. Also the file prefix using the speed always has a unique start name, so it cannot be used in the deleteOldFiles routine. Best solution to this still seems to be to make the change in deleteOldFiles routine like, fileList = sorted(glob.glob(os.path.join(dirPath, '' + prefix + '')), key=os.path.getmtime)

[peterb99]

Looks like our comments crossed in time, the issue with max recent is still valid. Issue arises when image_max_files is < imageRecentMax, and image files get deleted while symlink to them still there. when a link does not point to an existing file the code line calling sorted throws error, my guess is os.path.getmtime fails there. try: fileList = sorted(glob.glob(os.path.join(dirPath, '' + prefix + '')), key=os.path.getmtime)

[pageauc]

Please test speed-cam.py version 11.07 I am having problems duplicating your issue. Also delete all the symlinks in the recent folder. Deleting the symlinks will not delete the destination image files.

Note I think the problem is that some of your files are prefixed with the ave_speed-image_prefix-datetime. The new version has image_prefix-ave_speed-datetime as filename

If things are really messed up try a fresh install and test that. If there is still a problem send me the python error messages.

Thanks Claude .... .

On Fri, Jun 25, 2021 at 11:15 AM peterb99 @.***> wrote:

Looks like our comments crossed in time, the issue with max recent is still valid. Issue arises when image_max_files is < imageRecentMax, and image files get deleted while symlink to them still there. when a link does not point to an existing file the code line calling sorted throws error, my guess is os.path.getmtime fails there. try: fileList = sorted(glob.glob(os.path.join(dirPath, '' + prefix + '')), key=os.path.getmtime)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pageauc/speed-camera/issues/90#issuecomment-868572885, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNPKZDD2JV7VRCNKTY3WKDTUSMQFANCNFSM47F4JZQQ .

-- YouTube Channel at https://www.youtube.com/user/pageaucp http://www.youtube.com/user/pageaucp GitHub Repository at https://github.com/pageauc

[pageauc]

release 11.07 fixes issue