Crash when processing huge number of files already cached

[lklark]

When I reprocess directories with a lot of clips (>17000) I get this error after reading file lists and starting processing the file list:

Qt Concurrent has caught an exception thrown from a worker thread. This is not supported, exceptions thrown in worker threads must be caught before control returns to Qt Concurrent.Qt Concurrent has caught an exception thrown from a worker thread. This is not supported, exceptions thrown in worker threads must be caught before control returns to Qt Concurrent.OpenCV: terminate handler is called! The last OpenCV error is: OpenCV(3.4.7) Error: Assertion failed (!ssize.empty()) in resize, file E:\opencv-3.4.7\opencv-3.4.7\modules\imgproc\src\resize.cpp, line 3720

[theophanemayaud]

Thanks for the report !!! That’s weird, I have never seen this… I test et very time on a set of over 10 000 videos, I’ll try and see if I can replicate by doubling the number of videos to scan.

[theophanemayaud]

Haha my bad, I’m working on a fork of this project over at https://theophanemayaud.github.io/video-simili-duplicate-cleaner/, and I thought the issue you raised was for my fork 😬 Are you on windows ? If so, I can’t help you yet, I only have a working MacOS version, I’m still working on compiling it for windows !

[lklark]

Hi, I'm running it on windows. But thank you anyway!

Best, Klark

On Thu, Jun 10, 2021 at 4:17 PM Théophane Mayaud @.***> wrote:

Haha my bad, I’m working on a fork of this project over at https://theophanemayaud.github.io/video-simili-duplicate-cleaner/, and I thought the issue you raised was for my fork 😬 Are you on windows ? If so, I can’t help you yet, I only have a working MacOS version, I’m still working on compiling it for windows !

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kristiankoskimaki/vidupe/issues/14#issuecomment-858661348, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNXO7YVCLXUM7NCNQYMZFDTSDCODANCNFSM46OO2PNA .

[theophanemayaud]

Hi, if you still need this I finally managed to build it for windows, and you can checkout my project page to see if my version fixes the bug you had with this version 🤷‍♂️ https://theophanemayaud.github.io/video-simili-duplicate-cleaner/

[lklark]

Thank you. I'll try it out!

On Wed, Mar 16, 2022 at 2:19 PM Théophane Mayaud @.***> wrote:

Hi, if you still need this I finally managed to build it for windows, and you can checkout my project page to see if my version fixes the bug you had with this version 🤷‍♂️ https://theophanemayaud.github.io/video-simili-duplicate-cleaner/

— Reply to this email directly, view it on GitHub https://github.com/kristiankoskimaki/vidupe/issues/14#issuecomment-1069120225, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNXO73MGBKVES2KUFRNHWTVAHNVZANCNFSM46OO2PNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.***>

[jedmeister324]

I get the crash too, is it necessary to buy the software to see if it fixes the crash? My crash is a bit different though, i get the crash while scanning a large library, the event logs say...

Faulting application name: Vidupe.exe, version: 1.211.0.0, time stamp: 0x5d81af89 Faulting module name: libopencv_core347.dll, version: 3.4.7.0, time stamp: 0x5d6be248 Exception code: 0x40000015 Fault offset: 0x001a9066 Faulting process id: 0xbd4 Faulting application start time: 0x01d8397175a9f3c3 Faulting application path: C:\Users\xxx\Downloads\Vidupe-1.211_win32\Vidupe\Vidupe.exe Faulting module path: C:\Users\xxx\Downloads\Vidupe-1.211_win32\Vidupe\libopencv_core347.dll Report Id: cb686613-a55e-4527-8503-b1ebcf984f54 Faulting package full name: Faulting package-relative application ID:

So, looks like the opencv code is crashing. Happens only when scanning 3x2 or higher.

[theophanemayaud]

Well you can try to build my project from source but since I put a lot of effort in it, that’s why I put it on the store. If you still get the crash you can also ask for a refund from Microsoft I believe ;)

[theophanemayaud]

Also, if you still get the crash I’ll look into it and make updates to fix it.

[lklark]

Hi Théophane, Thank you very much! I meanwhile bought the windows version. Thanks for that! It is not crashing any more but I'm wondering if it creates a cache file, I couldn't find it after reading 7000 clips. And second pass seemed also to be quite slow. Where is the cache file typically saved to?

Thanks again and Best Regards, Lothar

On Sat, Mar 19, 2022 at 7:21 AM Théophane Mayaud @.***> wrote:

Also, if you still get the crash I’ll look into it and make updates to fix it.

— Reply to this email directly, view it on GitHub https://github.com/kristiankoskimaki/vidupe/issues/14#issuecomment-1072952094, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNXO73HTHPBDUQAYIULET3VAVXAHANCNFSM46OO2PNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.***>

[theophanemayaud]

Maybe we should open an issue or discussion on https://github.com/theophanemayaud/video-simili-duplicate-cleaner/discussions

I’ll look into where the cache is on windows, I’m not sure 🤔

[lklark]

Hi Théophane,

Thank you for your answer. I reviewed the source code. it uses "%1/cache.db". I think it is a read/write permission issue in the windows app directory (Install issue). I checked it compiling from source code in a different directory. It works. So either there must be a fix within the app installation or the cache file location should be customizable...

I think you should solve this as it may affect all windows installations having no cache function at all.

Thanks and BR Lothar

On Sun, Mar 20, 2022 at 12:53 PM Théophane Mayaud @.***> wrote:

Maybe we should open an issue or discussion on https://github.com/theophanemayaud/video-simili-duplicate-cleaner/discussions

I’ll look into where the cache is on windows, I’m not sure 🤔

— Reply to this email directly, view it on GitHub https://github.com/kristiankoskimaki/vidupe/issues/14#issuecomment-1073234646, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNXO77CUXXOCXMMC2PTBLDVA4GTBANCNFSM46OO2PNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.***>

[theophanemayaud]

I've finally had time to look at it, I'll look for a way around it, there's indeed a problem ! I'll post further updates on the dedicated issue : https://github.com/theophanemayaud/video-simili-duplicate-cleaner/issues/69

[lklark]

Awesome, thanks!

On Sat, Apr 9, 2022 at 10:35 AM Théophane Mayaud @.***> wrote:

I've finally had time to look at it, I'll look for a way around it, there's indeed a problem ! I'll post further updates on the dedicated issue : theophanemayaud/video-simili-duplicate-cleaner#69 https://github.com/theophanemayaud/video-simili-duplicate-cleaner/issues/69

— Reply to this email directly, view it on GitHub https://github.com/kristiankoskimaki/vidupe/issues/14#issuecomment-1093811977, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNXO7776453TFJ3Z2NE4V3VEE6L7ANCNFSM46OO2PNA . You are receiving this because you authored the thread.Message ID: @.***>

[lklark]

Maybe it makes sense to put the cache file into a LocalApp,AppData etc. Directory of the user...

On Sat, Apr 9, 2022 at 1:42 PM Lothar Klark @.***> wrote:

Awesome, thanks!

On Sat, Apr 9, 2022 at 10:35 AM Théophane Mayaud @.***> wrote:

I've finally had time to look at it, I'll look for a way around it, there's indeed a problem ! I'll post further updates on the dedicated issue : theophanemayaud/video-simili-duplicate-cleaner#69 https://github.com/theophanemayaud/video-simili-duplicate-cleaner/issues/69

— Reply to this email directly, view it on GitHub https://github.com/kristiankoskimaki/vidupe/issues/14#issuecomment-1093811977, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNXO7776453TFJ3Z2NE4V3VEE6L7ANCNFSM46OO2PNA . You are receiving this because you authored the thread.Message ID: @.***>

[faceleg]

Also the app doesn't seem to remember the custom cache location when it is restarted