Search for similar images breaks down

garlem-rid commented 5 months ago

Bug Description I'm using https://github.com/jlesage/docker-czkawka version of czkawka in docker. And I have a problem with similar images during the scanning stage (img below). The process starts, but after a few hashes it aborts and the application restarts in the container without the hashes saved. I can't catch the specific image that broke the process because it is being destroyed by a hash that is random to me. This could be a good 10 hashes and on the 11th the program will fail. And after reboot it may be the 40th or 14th. But the calculated hashes will be saved if I stop the process with the button

Steps to reproduce: Can't do explicit repetition. Im use default properties with 64 hash size (img). And I can’t catch specific images because there are more than 14,000 of them but I didn’t find the image hashing log. I tried 6.1, 5.1 and others between.

Terminal output (optional):

Debug log

``` czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:10.631 [DEBUG] czkawka_gui::connect_things::connect_button_search: clean_tree_view czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:10.631 [DEBUG] czkawka_gui::connect_things::connect_button_search: clean_tree_view: Done in 3.10µs czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:10.644 [INFO] czkawka_core::similar_images: find_similar_images czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:10.644 [DEBUG] czkawka_core::similar_images: check_for_similar_images czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:11.414 [DEBUG] czkawka_core::common: send_info_and_wait_for_ending_all_threads czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:11.430 [DEBUG] czkawka_core::common: send_info_and_wait_for_ending_all_threads: Done in 16.53ms czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:11.430 [DEBUG] czkawka_core::similar_images: check_for_similar_images: Done in 786.16ms czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:11.430 [DEBUG] czkawka_core::similar_images: hash_images czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:11.430 [DEBUG] czkawka_core::similar_images: hash_images_load_cache czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:11.431 [DEBUG] czkawka_core::similar_images: hash_images-load_cache - starting calculating diff czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:11.484 [DEBUG] czkawka_core::similar_images: hash_images_load_cache - completed diff between loaded and prechecked files, 14973(35.96 GiB) - non cached, 0(0 B) - already cached czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:11.484 [DEBUG] czkawka_core::similar_images: hash_images_load_cache: Done in 53.55ms czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:11.484 [DEBUG] czkawka_core::similar_images: hash_images - start hashing images czkawka_photo_deduplication-czkawka-1 | [app ] thread '' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rawloader-0.37.1/src/decoders/dng.rs:228:81: czkawka_photo_deduplication-czkawka-1 | [app ] called `Result::unwrap()` on an `Err` value: "ljpeg: predictor 7 not supported" czkawka_photo_deduplication-czkawka-1 | [app ] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. czkawka_photo_deduplication-czkawka-1 | [supervisor ] service 'app' exited (got signal SIGABRT). czkawka_photo_deduplication-czkawka-1 | [supervisor ] waiting termination of logger thread of service 'app'... czkawka_photo_deduplication-czkawka-1 | [supervisor ] started service 'app'. czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:29.652 [INFO] czkawka_core::common: Czkawka version: 6.1.0, was compiled with release mode czkawka_photo_deduplication-czkawka-1 | [app ] 17:08:30.184 [INFO] czkawka_gui: Set thread number to 1 ```

System

Czkawka version: 6.1.0, (and 5.1)
OS version in docker: alpine:3.17 (Linux 0d671f174dc0 4.4.302+ #69057 SMP Mon Nov 13 14:19:30 CST 2023 x86_64 Linux)
OS version main: synology (Linux MySynology 4.4.302+ #69057 SMP Mon Nov 13 14:19:30 CST 2023 x86_64 GNU/Linux synology_geminilake_220+)

docker-compose.yml

``` version: '3' services: czkawka: image: jlesage/czkawka ports: - "5800:5800" volumes: - "./config:/config:rw" - "./cache:/config/xdg/cache/czkawka" - "/volumeUSB1/usbshare/myphoto:/storage" cpuset: "0" environment: - SECURE_CONNECTION=0 - CONTAINER_DEBUG=1 - KEEP_APP_RUNNING=1 - RUST_BACKTRACE=full - RUST_BACKTRACE=1 - RUST_LOG=debug - USER_ID=1026 - GROUP_ID=100 - DARK_MODE=1 # - ENABLE_CJK_FONT=1 ```

garlem-rid commented 5 months ago

Also I have the same problems on my opi5 with kernel (Linux opi 5.10.160-legacy-rk35xx #1 SMP Tue Nov 28 02:45:16 UTC 2023 aarch64 GNU/Linux)

garlem-rid commented 5 months ago

I think I found the problem. Most likely the problem is in iPhone raw photos (DNG). I simply ran the script through the files, replacing dng ->tif after running the Bad Extention utility.

mehrexe commented 5 months ago

Also experiencing this issue. Also using the docker image. 19K images, default settings for similar images, mostly iPhone photos. I checked as @garlem-rid said but have no dng images in my bad extensions search.

qarmin commented 5 months ago

Looks like same issue like https://github.com/qarmin/czkawka/issues/1140 and should be fixed by https://github.com/qarmin/czkawka/pull/1102

qarmin / czkawka

Search for similar images breaks down #1179