Process fails after 9 images

shimizurei commented 6 years ago

Tested Systems: Win7x64, Win10

I had tested this on a small group of files and it worked beautifully, so I decided to go big (183 files):

Input

caesiumclt -q 0 -RS -o "C:\Users\ME\Downloads\ComicMangaNovel\[APPROVED]\Joou to Shitateya by Scarlet Beriko\compressed" "C:\Users\ME\Downloads\ComicMangaNovel\[APPROVED]\Joou to Shitateya by Scarlet Beriko\tocompress"

Output

[WARNING] 3: Empty input folder.
(1/183) 000.jpg -> C:\Users\ME\Downloads\ComicMangaNovel\[APPROVED]\Joou to Shitateya by Scarlet Beriko\compressed\[Scarlet Beriko] Joou to Shitateya ch. 1 [Eng]\000.jpg
1.48 MB -> 1.30 MB [-12.29%]
(2/183) 0000.png -> C:\Users\ME\Downloads\ComicMangaNovel\[APPROVED]\Joou to Shitateya by Scarlet Beriko\compressed\[Scarlet Beriko] Joou to Shitateya ch. 1 [Eng]\0000.png
239.57 KB -> 177.59 KB [-25.87%]
(3/183) 001.jpg -> C:\Users\ME\Downloads\ComicMangaNovel\[APPROVED]\Joou to Shitateya by Scarlet Beriko\compressed\[Scarlet Beriko] Joou to Shitateya ch. 1 [Eng]\001.jpg
2.03 MB -> 1.85 MB [-9.30%]
(4/183) 002.jpg -> C:\Users\ME\Downloads\ComicMangaNovel\[APPROVED]\Joou to Shitateya by Scarlet Beriko\compressed\[Scarlet Beriko] Joou to Shitateya ch. 1 [Eng]\002.jpg
853.61 KB -> 709.89 KB [-16.84%]
(5/183) 003.jpg -> C:\Users\ME\Downloads\ComicMangaNovel\[APPROVED]\Joou to Shitateya by Scarlet Beriko\compressed\[Scarlet Beriko] Joou to Shitateya ch. 1 [Eng]\003.jpg
40.46 KB -> 7.81 KB [-80.69%]
(6/183) 004.jpg -> C:\Users\ME\Downloads\ComicMangaNovel\[APPROVED]\Joou to Shitateya by Scarlet Beriko\compressed\[Scarlet Beriko] Joou to Shitateya ch. 1 [Eng]\004.jpg
85.14 KB -> 47.87 KB [-43.77%]
(7/183) 005.jpg -> C:\Users\ME\Downloads\ComicMangaNovel\[APPROVED]\Joou to Shitateya by Scarlet Beriko\compressed\[Scarlet Beriko] Joou to Shitateya ch. 1 [Eng]\005.jpg
69.26 KB -> 32.88 KB [-52.52%]
(8/183) 006.jpg -> C:\Users\ME\Downloads\ComicMangaNovel\[APPROVED]\Joou to Shitateya by Scarlet Beriko\compressed\[Scarlet Beriko] Joou to Shitateya ch. 1 [Eng]\006.jpg
534.81 KB -> 491.10 KB [-8.17%]
(9/183) 007.png -> C:\Users\ME\Downloads\ComicMangaNovel\[APPROVED]\Joou to Shitateya by Scarlet Beriko\compressed\[Scarlet Beriko] Joou to Shitateya ch. 1 [Eng]\007.png

Then caesium-clt crashes. Every time on the 9th file. On Win 10, it just silently stops at file 9.

Could it be the length of the path name? - Maybe not, since I moved it closer to my user profile folder and it still crashed on file 9.

I used 2 computers, moved directories, made sure it wasn't my antivirus. Not sure what else to check. Also, what up with the [WARNING] 3: Empty input folder error?

Lymphatus commented 6 years ago

I ran a bunch of tests with the new releases and seems everything ok. Can you please try again using this version and tell me if the problem persists?

Thanks.

shimizurei commented 6 years ago

Still there :( It just silently stops working. "Dry run" prints out all the files though.

Is my usage wrong in some way? I eliminated the white space thinking that might be why.

PS C:\Users\ME> caesiumclt -q 0 -RS -o "C:\Users\ME\Downloads\install\compr" "C:\Users\ME\Downloads\install\ff"
Collecting files...
[WARNING] 3: Empty input folder.
(1/117) Katajikenai_000a.jpg -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_000a.jpg
1.36 MB -> 1.32 MB [-2.81%]
(2/117) Katajikenai_000b.jpg -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_000b.jpg
1.36 MB -> 1.32 MB [-2.75%]
(3/117) Katajikenai_000c.jpg -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_000c.jpg
953.47 KB -> 919.44 KB [-3.57%]
(4/117) Katajikenai_000d.jpg -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_000d.jpg
2.70 MB -> 2.65 MB [-2.05%]
(5/117) Katajikenai_000e.png -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_000e.png
1.79 KB -> 940.00 B [-48.86%]
(6/117) Katajikenai_001.jpg -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_001.jpg
607.93 KB -> 591.79 KB [-2.65%]
(7/117) Katajikenai_002.jpg -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_002.jpg
1.01 MB -> 1011.35 KB [-2.53%]
(8/117) Katajikenai_003.jpg -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_003.jpg
771.25 KB -> 750.97 KB [-2.63%]
(9/117) Katajikenai_004.jpg -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_004.jpg
1.03 MB -> 1.01 MB [-2.70%]
(10/117) Katajikenai_005.jpg -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_005.jpg
596.07 KB -> 579.10 KB [-2.85%]
(11/117) Katajikenai_006-007.jpg -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_006-007.jpg
2.34 MB -> 2.28 MB [-2.45%]
(12/117) Katajikenai_008.jpg -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_008.jpg
461.49 KB -> 449.48 KB [-2.60%]
(13/117) Katajikenai_009.png -> C:\Users\ME\Downloads\install\compr\Katajikenai_byKOIKE_Keiichi\Katajikenai_009.png

Lymphatus commented 6 years ago

This may be related to the -RS option, then. I'm gonna run some tests and see if this might be the issue.

Lymphatus commented 6 years ago

I just noticed from your logs that stops always with a PNG file. Is that true for all cases you tested or just a coincidence?

shimizurei commented 6 years ago

Interesting...

Test 1: no PNG files ==> CRASHES

PS C:\Users\ME> caesiumclt -q 0 -RS -o C:\Users\ME\Downloads\www\compr C:\Users\ME\Downloads\www\ff
Collecting files...
[WARNING] 3: Empty input folder.
(1/21) 001.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\001.jpg
669.34 KB -> 601.77 KB [-10.09%]
(2/21) 002.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\002.jpg
71.07 KB -> 31.81 KB [-55.23%]
(3/21) 003.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\003.jpg
45.69 KB -> 8.79 KB [-80.75%]
(4/21) 004.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\004.jpg
289.46 KB -> 233.97 KB [-19.17%]
(5/21) 005.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\005.jpg
418.45 KB -> 343.94 KB [-17.81%]
(6/21) 006.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\006.jpg
509.49 KB -> 437.79 KB [-14.07%]
(7/21) 007.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\007.jpg
589.76 KB -> 512.67 KB [-13.07%]
(8/21) 008.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\008.jpg
445.24 KB -> 373.58 KB [-16.09%]
(9/21) 009.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\009.jpg
696.74 KB -> 615.47 KB [-11.66%]
(10/21) 010.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\010.jpg
645.73 KB -> 545.73 KB [-15.49%]
(11/21) 011.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\011.jpg
719.05 KB -> 634.36 KB [-11.78%]
(12/21) 012.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\012.jpg
572.35 KB -> 496.55 KB [-13.24%]
(13/21) 013.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\013.jpg
543.43 KB -> 469.19 KB [-13.66%]
(14/21) 014.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\014.jpg
807.44 KB -> 719.74 KB [-10.86%]
(15/21) 015.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\015.jpg
555.37 KB -> 480.90 KB [-13.41%]
(16/21) 016.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\016.jpg
364.28 KB -> 299.37 KB [-17.82%]
(17/21) 017.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\017.jpg
276.83 KB -> 218.13 KB [-21.20%]
(18/21) 018.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\018.jpg
===> CRASH

Test 2: random PNG file inserted into same files from before ==> no crash

PS C:\Users\ME> caesiumclt -q 0 -RS -o C:\Users\ME\Downloads\www\compr C:\Users\ME\Downloads\www\ff
Collecting files...
[WARNING] 3: Empty input folder.
(1/22) 001.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\001.jpg
669.34 KB -> 601.77 KB [-10.09%]
(2/22) 002.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\002.jpg
71.07 KB -> 31.81 KB [-55.23%]
(3/22) 003.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\003.jpg
45.69 KB -> 8.79 KB [-80.75%]
(4/22) 003a.png -> C:\Users\ME\Downloads\www\compr\500NengoDJ\003a.png
446.06 KB -> 421.21 KB [-5.57%]
(5/22) 004.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\004.jpg
289.46 KB -> 233.97 KB [-19.17%]
(6/22) 005.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\005.jpg
418.45 KB -> 343.94 KB [-17.81%]
(7/22) 006.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\006.jpg
509.49 KB -> 437.79 KB [-14.07%]
(8/22) 007.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\007.jpg
589.76 KB -> 512.67 KB [-13.07%]
(9/22) 008.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\008.jpg
445.24 KB -> 373.58 KB [-16.09%]
(10/22) 009.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\009.jpg
696.74 KB -> 615.47 KB [-11.66%]
(11/22) 010.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\010.jpg
645.73 KB -> 545.73 KB [-15.49%]
(12/22) 011.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\011.jpg
719.05 KB -> 634.36 KB [-11.78%]
(13/22) 012.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\012.jpg
572.35 KB -> 496.55 KB [-13.24%]
(14/22) 013.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\013.jpg
543.43 KB -> 469.19 KB [-13.66%]
(15/22) 014.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\014.jpg
807.44 KB -> 719.74 KB [-10.86%]
(16/22) 015.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\015.jpg
555.37 KB -> 480.90 KB [-13.41%]
(17/22) 016.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\016.jpg
364.28 KB -> 299.37 KB [-17.82%]
(18/22) 017.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\017.jpg
276.83 KB -> 218.13 KB [-21.20%]
(19/22) 018.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\018.jpg
45.69 KB -> 8.79 KB [-80.75%]
(20/22) 019.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\019.jpg
2.74 MB -> 2.49 MB [-9.21%]
(21/22) 020.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\020.jpg
116.14 KB -> 76.14 KB [-34.44%]
(22/22) 021.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\021.jpg
694.43 KB -> 624.31 KB [-10.10%]
-------------------------------
Compression completed in 0m47s
12.04 MB -> 10.46 MB [-13.19% | -1.59 MB]

Test 3: PNG file moved to right before program usually crashes ==> no crash

PS C:\Users\ME> caesiumclt -q 0 -RS -o C:\Users\ME\Downloads\www\compr C:\Users\ME\Downloads\www\ff
Collecting files...
[WARNING] 3: Empty input folder.
(1/22) 001.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\001.jpg
669.34 KB -> 601.77 KB [-10.09%]
(2/22) 002.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\002.jpg
71.07 KB -> 31.81 KB [-55.23%]
(3/22) 003.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\003.jpg
45.69 KB -> 8.79 KB [-80.75%]
(4/22) 004.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\004.jpg
289.46 KB -> 233.97 KB [-19.17%]
(5/22) 005.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\005.jpg
418.45 KB -> 343.94 KB [-17.81%]
(6/22) 006.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\006.jpg
509.49 KB -> 437.79 KB [-14.07%]
(7/22) 007.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\007.jpg
589.76 KB -> 512.67 KB [-13.07%]
(8/22) 008.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\008.jpg
445.24 KB -> 373.58 KB [-16.09%]
(9/22) 009.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\009.jpg
696.74 KB -> 615.47 KB [-11.66%]
(10/22) 010.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\010.jpg
645.73 KB -> 545.73 KB [-15.49%]
(11/22) 011.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\011.jpg
719.05 KB -> 634.36 KB [-11.78%]
(12/22) 012.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\012.jpg
572.35 KB -> 496.55 KB [-13.24%]
(13/22) 013.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\013.jpg
543.43 KB -> 469.19 KB [-13.66%]
(14/22) 014.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\014.jpg
807.44 KB -> 719.74 KB [-10.86%]
(15/22) 015.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\015.jpg
555.37 KB -> 480.90 KB [-13.41%]
(16/22) 016.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\016.jpg
364.28 KB -> 299.37 KB [-17.82%]
(17/22) 017.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\017.jpg
276.83 KB -> 218.13 KB [-21.20%]
(18/22) 017a.png -> C:\Users\ME\Downloads\www\compr\500NengoDJ\017a.png
446.06 KB -> 421.21 KB [-5.57%]
(19/22) 018.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\018.jpg
45.69 KB -> 8.79 KB [-80.75%]
(20/22) 019.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\019.jpg
2.74 MB -> 2.49 MB [-9.21%]
(21/22) 020.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\020.jpg
116.14 KB -> 76.14 KB [-34.44%]
(22/22) 021.jpg -> C:\Users\ME\Downloads\www\compr\500NengoDJ\021.jpg
694.43 KB -> 624.31 KB [-10.10%]
-------------------------------
Compression completed in 0m47s
12.04 MB -> 10.46 MB [-13.19% | -1.59 MB]

On the sets with the PNG inserted the program hangs for about 20 sec on it, then completes work on the remaining images.

Lymphatus commented 6 years ago

That's indeed interesting. Could you please share with me the set of pictures? Or any set where the crash occurs. It seems random at first sight, but I want to run some more tests to see if there's something with the images or there's some kind of memory leak involved.

shimizurei commented 6 years ago

Here's the zip with the JPG images (that leads to the crash) and the random PNG that stops the crash (??). testo.zip

Lymphatus commented 6 years ago

I just tested on my Windows 10 VM and the compression goes well for all the cases (strange indeed...). But I just noticed the picture you provided me were already compressed. Do you still have the original uncompressed images? I just want to be sure and try my best to reproduce the issue. Thanks!

shimizurei commented 6 years ago

Oh sorry! testo2.zip

Lymphatus commented 6 years ago

Ok, I could reproduce the bug with this set of images. I will work on finding the cause and on a fix. Thanks a lot!

Lymphatus commented 6 years ago

Just a quick update: I found the problem, I'm working on a fix. :)

Lymphatus commented 6 years ago

Can you please run some tests with the version attached? It's working on my VM. clt.zip

shimizurei commented 6 years ago

It worked on the first set of files I gave you, but not this larger set I tried to throw at it (had to split it up since it's too big for github; I created an empty folder for you to throw all the files into in p1: kata2ke71-p1.zip kata2ke71-p2.zip kata2ke71-p3.zip kata2ke71-p4.zip kata2ke71-p5.zip

Lymphatus / caesium-clt