search

tjko / jpegoptim

jpegoptim - utility to optimize/compress JPEG files
http://www.iki.fi/tjko/projects.html
GNU General Public License v3.0
1.56k stars 116 forks source link

multithread results in garbled output #123

Closed catharsis71 closed 1 year ago

catharsis71 commented 1 year ago

Single thread (no output because all files are already optimized and I'm using grep to exclude "skipped" messages):

root@uk2:/var/www/fanart/unsorted/1# jpegoptim-test -w 1 -n --nofix *.jpg | grep -v skipped
root@uk2:/var/www/fanart/unsorted/1#

Multi thread (same directory, same files, output varies randomly from run to run):

root@uk2:/var/www/fanart/unsorted/1# jpegoptim-test -w 4 -n --nofix *.jpg | grep -v skipped
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1353979094592.jpg 555x660 24b.
.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
pped.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
d.
141082d.
d.
root@uk2:/var/www/fanart/unsorted/1#

Leaving off the | grep -v skipped, here are a few examples of garbling issues in the full output:

1361402666946.jpg 966x972 24bit N Exif IPTC XMP ICC JFIF  [OK] 736368 --> 736368 bytes (0.00%), skipped.
1361795560423.jpg 500x376 24bit N JFIF  [OK] 124340 --> 124340 bytes (0.00%), skipped.
1361826677668.jpg 1024x681 24bit N IPTC JFIF  [OK] 312987 --> 312987 bytes (0.00%), skipped.
pped.
it N Exif JFIF  [OK] 75034 --> 75034 bytes (0.00%), skipped.
1354139056440.jpg 800x600 24bit N JFIF  [OK] 67998 --> 67998 bytes (0.00%), skipped.
1354559075660.jpg 1000x669 24bit N JFIF  [OK] 124224 --> 124224 bytes (0.00%), skipped.
1389120_552542361499300_2018231742_n.jpg 612x612 24bit P IPTC JFIF  [OK] 137243 --> 137243 bytes (0.00%), skipped.
1389737855089.jpg 854x1200 24bit N JFIF  [OK] 402579 --> 402579 bytes (0.00%), skipped.
1389746192882.jpg 558x700 24bit N Exif JFIF  [OK] 135010 --> 135010 bytes (0.00%), skipped.
d.
d.
1386490743082.jpg 1404x1183 24bit N Exif IPTC XMP ICC JFIF  [OK] 1222057 --> 1222057 bytes (0.00%), skipped.
1390345060382.jpg 750x1000 24bit N JFIF  [OK] 146750 --> 146750 bytes (0.00%), skipped.
d.
1386490743082.jpg 1404x1183 24bit N Exif IPTC XMP ICC JFIF  [OK] 1222057 --> 1222057 bytes (0.00%), skipped.
1390345060382.jpg 750x1000 24bit N JFIF  [OK] 146750 --> 146750 bytes (0.00%), skipped.
1390197501771.jpg 750x1000 24bit N JFIF  [OK] 519670 --> 519670 bytes (0.00%), skipped.
d.
1386490743082.jpg 1404x1183 24bit N Exif IPTC XMP ICC JFIF  [OK] 1222057 --> 1222057 bytes (0.00%), skipped.
d.
1386490743082.jpg 1404x1183 24bit N Exif IPTC XMP ICC JFIF  [OK] 1222057 --> 1222057 bytes (0.00%), skipped.
1390345060382.jpg 750x1000 24bit N JFIF  [OK] 146750 --> 146750 bytes (0.00%), skipped.
1390197501771.jpg 750x1000 24bit N JFIF  [OK] 519670 --> 519670 bytes (0.00%), skipped.
1390945325358.jpg 960x1280 24bit N JFIF  [OK] 262432 --> 262432 bytes (0.00%), skipped.
1392052696415.jpg 960x768 24bit P ICC JFIF  [OK] 118420 --> 118420 bytes (0.00%), skipped.
d.
1386490743082.jpg 1404x1183 24bit N Exif IPTC XMP ICC JFIF  [OK] 1222057 --> 1222057 bytes (0.00%), skipped.
1390345060382.jpg 750x1000 24bit N JFIF  [OK] 146750 --> 146750 bytes (0.00%), skipped.
1390197501771.jpg 750x1000 24bit N JFIF  [OK] 519670 --> 519670 bytes (0.00%), skipped.
1390945325358.jpg 960x1280 24bit N JFIF  [OK] 262432 --> 262432 bytes (0.00%), skipped.
1392052696415.jpg 960x768 24bit P ICC JFIF  [OK] 118420 --> 118420 bytes (0.00%), skipped.
1392409005424.jpg 1024x768 24bit N JFIF  [OK] 206229 --> 206229 bytes (0.00%), skipped.
d.
1386490743082.jpg 1404x1183 24bit N Exif IPTC XMP ICC JFIF  [OK] 1222057 --> 1222057 bytes (0.00%), skipped.
1390345060382.jpg 750x1000 24bit N JFIF  [OK] 146750 --> 146750 bytes (0.00%), skipped.
1390197501771.jpg 750x1000 24bit N JFIF  [OK] 519670 --> 519670 bytes (0.00%), skipped.
1390945325358.jpg 960x1280 24bit N JFIF  [OK] 262432 --> 262432 bytes (0.00%), skipped.
tjko commented 1 year ago

I tried this on Linux and couldn't reproduce the problem... Were you using v1.5.0? Does it happen if you compile the latest version from github (v1.5.1beta) ?

catharsis71 commented 1 year ago

I tested on two systems:

  1. Ubuntu Server 22.04.1 ARM on an Oracle Cloud flex VPS, 4 core
  2. Ubuntu WSL 22.04.1 on Windows 10, 12 core / 24 thread

I downloaded the 1.5.0 source for both but obviously had to compile separately since one is ARM and one is AMD.

Note that the number of garbled lines is proportionally small so you may only see this when processing a large number of files

On system 1, w=1 is clean, then I start seeing garbled lines when using w=2. Moving to w=3, the number of garbled lines actually decreases for some reason, then increases again when using w=4

root@uk2:/var/www/fanart/unsorted/1# jpegoptim-test -w 1 -n --nofix *.jpg | grep -v skipped | wc -l
0
root@uk2:/var/www/fanart/unsorted/1# jpegoptim-test -w 1 -n --nofix *.jpg | grep -v skipped | wc -l
0
root@uk2:/var/www/fanart/unsorted/1# jpegoptim-test -w 2 -n --nofix *.jpg | grep -v skipped | wc -l
85
root@uk2:/var/www/fanart/unsorted/1# jpegoptim-test -w 2 -n --nofix *.jpg | grep -v skipped | wc -l
86
root@uk2:/var/www/fanart/unsorted/1# jpegoptim-test -w 3 -n --nofix *.jpg | grep -v skipped | wc -l
44
root@uk2:/var/www/fanart/unsorted/1# jpegoptim-test -w 3 -n --nofix *.jpg | grep -v skipped | wc -l
45
root@uk2:/var/www/fanart/unsorted/1# jpegoptim-test -w 4 -n --nofix *.jpg | grep -v skipped | wc -l
91
root@uk2:/var/www/fanart/unsorted/1# jpegoptim-test -w 4 -n --nofix *.jpg | grep -v skipped | wc -l
90
root@uk2:/var/www/fanart/unsorted/1#

On system 2, I don't see garbled lines until w=4 after which the number of garbled lines varies unpredictably:

:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 1 -n --nofix *.jpg | grep -v skipped | wc -l
0
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 1 -n --nofix *.jpg | grep -v skipped | wc -l
0
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 2 -n --nofix *.jpg | grep -v skipped | wc -l
0
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 2 -n --nofix *.jpg | grep -v skipped | wc -l
0
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 3 -n --nofix *.jpg | grep -v skipped | wc -l
0
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 3 -n --nofix *.jpg | grep -v skipped | wc -l
0
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 4 -n --nofix *.jpg | grep -v skipped | wc -l
44
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 4 -n --nofix *.jpg | grep -v skipped | wc -l
44
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 6 -n --nofix *.jpg | grep -v skipped | wc -l
93
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 6 -n --nofix *.jpg | grep -v skipped | wc -l
93
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 8 -n --nofix *.jpg | grep -v skipped | wc -l
46
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 8 -n --nofix *.jpg | grep -v skipped | wc -l
46
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 10 -n --nofix *.jpg | grep -v skipped | wc -l
46
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 10 -n --nofix *.jpg | grep -v skipped | wc -l
46
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 12 -n --nofix *.jpg | grep -v skipped | wc -l
91
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 12 -n --nofix *.jpg | grep -v skipped | wc -l
46
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 24 -n --nofix *.jpg | grep -v skipped | wc -l
90
:~/backup/www/fanart/unsorted/1$ jpegoptim-150 -w 24 -n --nofix *.jpg | grep -v skipped | wc -l
44

Even though these are two separate systems, in the examples above, they're running on the same set of files (1861 JPEGs totaling 329MB) as I synchronized the two systems with rsync before testing.

catharsis71 commented 1 year ago

I will try compiling from master tomorrow

tjko commented 1 year ago

I recall there was some fixes to output since 1.5.0, that may have "fixed" this as well...