search

DaveStrickland / AstroPhotography

Python workflow for processing astrophotography images from either digital cameras and/or amateur telescopes.
GNU General Public License v3.0
22 stars 4 forks source link

Imageio output issues within dksraw: slow or lossy #6

Closed DaveStrickland closed 3 days ago

DaveStrickland commented 3 years ago

Writing a PNG format greyscale from the test image:

2021-03-10 06:39:09,385 | DEBUG | AstroPhotography | Greyscale image generated.
2021-03-10 06:39:09,472 | INFO | AstroPhotography | Writing 2602x3906 uint16 image to test_linear.png
2021-03-10 06:39:54,606 | DEBUG | AstroPhotography | Wrote 7.059 MB file successfully in 45.135 seconds (0.156 MB/s)

Writing jpg forces conversion down to 8 bits:

2021-03-10 06:47:41,385 | DEBUG | AstroPhotography | Greyscale image generated.
2021-03-10 06:47:41,441 | INFO | AstroPhotography | Writing 2602x3906 uint16 image to test_linear.jpg
Lossy conversion from uint16 to uint8. Losing 8 bits of resolution. Convert image to uint8 prior to saving to suppress this warning.
2021-03-10 06:47:41,763 | DEBUG | AstroPhotography | Wrote 0.120 MB file successfully in 0.322 seconds (0.373 MB/s)

Writing TIFF is fast though:

2021-03-10 06:54:17,079 | DEBUG | AstroPhotography | Greyscale image generated.
2021-03-10 06:54:17,137 | INFO | AstroPhotography | Writing 2602x3906 uint16 image to test_linear.tiff
2021-03-10 06:54:17,593 | DEBUG | AstroPhotography | Wrote 20.327 MB file successfully in 0.456 seconds (44.566 MB/s)

Suggested solutions:

DaveStrickland commented 3 days ago

These performance numbers need to be re-evaluated with the current version(s) of imageio. It might also be related to which plugin is being used for each format.

DaveStrickland commented 3 days ago

Running commands of the form dksraw grey test/AstroPhotography/data/IMG_0672.CR2 -o test.png -l DEBUG --renormalize with v0.6.0 I get the following results (this example with png as the file name suffix):

PNG:

2024-11-27 21:27:22,714 | DEBUG | AstroPhotography | Greyscale image of shape (2602, 3906) generated.
2024-11-27 21:27:22,719 | INFO | AstroPhotography | Writing 2602x3906 uint16 image to test.png
2024-11-27 21:27:23,321 | DEBUG | AstroPhotography | Wrote 17.479 MB file successfully in 0.603 seconds (29.006 MB/s)
2024-11-27 21:27:23,321 | WARNING | AstroPhotography | Writing EXIF metadata not yet supported.
2024-11-27 21:27:23,321 | INFO | AstroPhotography | Completed grey command in 2.22 seconds.

JPEG

2024-11-27 21:30:58,558 | DEBUG | AstroPhotography | Greyscale image of shape (2602, 3906) generated.
2024-11-27 21:30:58,563 | INFO | AstroPhotography | Writing 2602x3906 uint16 image to test.jpg
Traceback (most recent call last):
  File "/home/dks/miniconda3/envs/ap-env/lib/python3.10/site-packages/PIL/JpegImagePlugin.py", line 650, in _save
    rawmode = RAWMODE[im.mode]
KeyError: 'I;16'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/dks/miniconda3/envs/ap-env/bin/dksraw", line 33, in <module>
    sys.exit(load_entry_point('AstroPhotography', 'console_scripts', 'dksraw')())
  File "/home/dks/git/AstroPhotography/AstroPhotography/cli.py", line 69, in main
    command(**args)
  File "/home/dks/git/AstroPhotography/AstroPhotography/api/grey.py", line 42, in main
    file_writer(output, grey_im, exif_dict)
  File "/home/dks/git/AstroPhotography/AstroPhotography/util/file_writer.py", line 145, in file_writer
    imageio.imsave(uri=out_file, im=data_array, format=our_format)
  File "/home/dks/miniconda3/envs/ap-env/lib/python3.10/site-packages/imageio/v2.py", line 396, in imwrite
    with imopen(uri, "wi", **imopen_args) as file:
  File "/home/dks/miniconda3/envs/ap-env/lib/python3.10/site-packages/imageio/core/v3_plugin_api.py", line 367, in __exit__
    self.close()
  File "/home/dks/miniconda3/envs/ap-env/lib/python3.10/site-packages/imageio/plugins/pillow.py", line 144, in close
    self._flush_writer()
  File "/home/dks/miniconda3/envs/ap-env/lib/python3.10/site-packages/imageio/plugins/pillow.py", line 485, in _flush_writer
    primary_image.save(self._request.get_file(), **self.save_args)
  File "/home/dks/miniconda3/envs/ap-env/lib/python3.10/site-packages/PIL/Image.py", line 2439, in save
    save_handler(self, fp, filename)
  File "/home/dks/miniconda3/envs/ap-env/lib/python3.10/site-packages/PIL/JpegImagePlugin.py", line 653, in _save
    raise OSError(msg) from e
OSError: cannot write mode I;16 as JPEG

So no silent down-conversion to 8-bits, but at the cost of a full blown exception and traceback.

Jpeg2000:

2024-11-27 21:32:30,801 | DEBUG | AstroPhotography | Greyscale image of shape (2602, 3906) generated.
2024-11-27 21:32:30,806 | INFO | AstroPhotography | Writing 2602x3906 uint16 image to test.jp2
Lossy conversion from uint16 to uint8. Losing 8 bits of resolution. Convert image to uint8 prior to saving to suppress this warning.
2024-11-27 21:32:31,488 | DEBUG | AstroPhotography | Wrote 2.032 MB file successfully in 0.682 seconds (2.978 MB/s)
2024-11-27 21:32:31,488 | WARNING | AstroPhotography | Writing EXIF metadata not yet supported.
2024-11-27 21:32:31,488 | INFO | AstroPhotography | Completed grey command in 2.28 seconds.

TIFF

2024-11-27 21:32:02,060 | DEBUG | AstroPhotography | Greyscale image of shape (2602, 3906) generated.
2024-11-27 21:32:02,066 | INFO | AstroPhotography | Writing 2602x3906 uint16 image to test.tiff
2024-11-27 21:32:02,085 | DEBUG | AstroPhotography | Wrote 20.327 MB file successfully in 0.019 seconds (1043.404 MB/s)
2024-11-27 21:32:02,085 | WARNING | AstroPhotography | Writing EXIF metadata not yet supported.
2024-11-27 21:32:02,085 | INFO | AstroPhotography | Completed grey command in 1.64 seconds.

Other file info:

identify test.tiff test.jp2 test.png test.jpg
test.tiff TIFF 3906x2602 3906x2602+0+0 16-bit Grayscale Gray 19.3854MiB 0.000u 0:00.000
test.jp2 JP2 3906x2602 3906x2602+0+0 8-bit Grayscale Gray 0.000u 0:00.000
test.png PNG 3906x2602 3906x2602+0+0 16-bit Grayscale Gray 16.6689MiB 0.000u 0:00.000
identify: insufficient image data in file `test.jpg' @ error/jpeg.c/ReadOneJPEGImage/1140.
ls -l --si test.tiff test.jp2 test.png test.jpg
-rw-r--r--. 1 dks dks 2.1M Nov 27 21:32 test.jp2
-rw-r--r--. 1 dks dks    0 Nov 27 21:30 test.jpg
-rw-r--r--. 1 dks dks  18M Nov 27 21:27 test.png
-rw-r--r--. 1 dks dks  21M Nov 27 21:32 test.tiff

So PNG, Jp2 and TIFF are all around 2 seconds for the test image, with TIFF still being fastest. However the JP2 was silently down-converted, so 2 seconds for a much smaller byte size is actually much worse performance.

Findings: The current versions of imageio (2.33.1) and Pillow (10.2.0, which imageio is likely using under the hood) do not have slow PNG output. Jpeg output is still problematiic, TIFF still works well.

I think it is safe to close this ticket.