Compression 4 is not supported

arogozhnikov commented 5 years ago

(I am new to this data format, so blindly trying to iterate over continuous chunks of data). Using latest czifile:

import czifile
f = czifile.CziFile('/Users/axelr/Downloads/Brightfield+RAC/2019_02_19__11_16__SL000768.czi')
for block in f.subblocks():
    print(block.shape)
    print(block.compression)
    print(block.data())
    break

Result:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-6-a57de3b0f14e> in <module>
      6     print(block.shape)
      7     print(block.compression)
----> 8     print(block.data())
      9     break

~/system1/envs/pipeline/lib/python3.7/site-packages/czifile/czifile.py in data(self, raw, resize, order)
    615                 fh.seek(self.data_offset)
    616                 data = fh.read(self.data_size)
--> 617             data = DECOMPRESS[de.compression](data)
    618             if de.compression == 2:
    619                 # LZW

KeyError: 4

arogozhnikov commented 5 years ago

And this looks like complete nonsense for me:

> arr = file.asarray()

> arr.shape
(1, 9, 6, 8, 68409, 125571, 1)

> arr.dtype
dtype('uint16')

> arr.nbytes
7421921169696

> arr.strides
(7421921169696, 824657907744, 137442984624, 17180373078, 251142, 2, 2)

cgohlke commented 5 years ago

KeyError: 4

Newer versions of czifile are using the imagecodecs package to decode LZW, JPEG, and JXR compressed image data.

And this looks like complete nonsense

Why?

arogozhnikov commented 5 years ago

Hi @cgohlke, unfortunately I don't quite get it - am I supposed to install imagecodecs?
(I'm trying to do this on mac, but if it is expected to work on ubuntu, this will also work for me)

And this looks like complete nonsense

because even strides are larger than my whole RAM, and it looks like a usual numpy.array. Did you use some kind of subclassing magic with custom access to elements?

cgohlke commented 5 years ago

if it is expected to work on ubuntu

On Ubuntu/Debian, install the build dependencies for imagecodecs before pip install imagecodecs (works for me):

$ sudo apt-get install build-essential python3-dev cython3 python3-setuptools python3-pip python3-wheel python3-numpy libz-dev libblosc-dev liblzma-dev liblz4-dev libzstd-dev libpng-dev libwebp-dev libbz2-dev libopenjp2-7-dev libjpeg62-turbo-dev libjxr-dev liblcms2-dev libtiff-dev

strides are larger than my whole RAM

What, not even 8 TB RAM?

Try to filter SubBlockSegments for small areas of interests. Czifile does not currently offer convenience functions for that. Python-bioformats or pylibczi might be better suited for such use cases.

arogozhnikov commented 5 years ago

Bioformats fail with 'java heap', ~~and increasing java heap size didn't help~~ (edit: it finally seemed to work)
I wan't able to iterate though scene with pylibczi (or get any scene, actually)
wasn't able to use imageformats in ubuntu 16:04 - no libjpeg62-turbo-dev, used just libjpeg62-dev you can see the log below

root@2392e33769e0:/# pip install imagecodecs
Collecting imagecodecs
  Using cached https://files.pythonhosted.org/packages/8e/64/3b839f8bf3c72be752b3a607f4b48109f41d895e3bf71e22de5f319c7050/imagecodecs-2019.2.20.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting numpy>=1.16.1 (from imagecodecs)
  Using cached https://files.pythonhosted.org/packages/ad/15/690c13ae714e156491392cdbdbf41b485d23c285aa698239a67f7cfc9e0a/numpy-1.16.1-cp35-cp35m-manylinux1_x86_64.whl
Building wheels for collected packages: imagecodecs
  Building wheel for imagecodecs (PEP 517) ... error
  Complete output from command /usr/bin/python3 /usr/local/lib/python3.5/dist-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpvb492fjp:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.5
  creating build/lib.linux-x86_64-3.5/imagecodecs
  copying imagecodecs/__init__.py -> build/lib.linux-x86_64-3.5/imagecodecs
  copying imagecodecs/__main__.py -> build/lib.linux-x86_64-3.5/imagecodecs
  copying imagecodecs/imagecodecs.py -> build/lib.linux-x86_64-3.5/imagecodecs
  creating build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-zfp -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-lcms2 -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-libwebp -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-bzip2 -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-libtiff -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-lz4 -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-libjpeg-turbo -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-liblzma -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-lzf -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-charls -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-zstd -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-jxrlib -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-blosc -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-openjpeg -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-bitshuffle -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-fastlz -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-zlib -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-libpng -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-jpg_0xc3 -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  copying imagecodecs/licenses/LICENSE-jpeg -> build/lib.linux-x86_64-3.5/imagecodecs/licenses
  running build_ext
  skipping 'imagecodecs/_imagecodecs.c' Cython extension (up-to-date)
  building 'imagecodecs._imagecodecs' extension
  creating build/temp.linux-x86_64-3.5
  creating build/temp.linux-x86_64-3.5/imagecodecs
  creating build/temp.linux-x86_64-3.5/liblzf-3.6
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DOPJ_HAVE_LIBLCMS2=1 -I/tmp/pip-build-env-8ztozem9/overlay/lib/python3.5/site-packages/numpy/core/include -Iimagecodecs -I/usr/include/jxrlib -I/usr/include/openjpeg-2.3 -Iliblzf-3.6 -I/usr/include/python3.5m -c imagecodecs/imagecodecs.c -o build/temp.linux-x86_64-3.5/imagecodecs/imagecodecs.o
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DOPJ_HAVE_LIBLCMS2=1 -I/tmp/pip-build-env-8ztozem9/overlay/lib/python3.5/site-packages/numpy/core/include -Iimagecodecs -I/usr/include/jxrlib -I/usr/include/openjpeg-2.3 -Iliblzf-3.6 -I/usr/include/python3.5m -c imagecodecs/opj_color.c -o build/temp.linux-x86_64-3.5/imagecodecs/opj_color.o
  imagecodecs/opj_color.c: In function 'sycc444_to_rgb':
  imagecodecs/opj_color.c:125:20: warning: implicit declaration of function 'opj_image_data_alloc' [-Wimplicit-function-declaration]
       d0 = r = (int*)opj_image_data_alloc(sizeof(int) * max);
                      ^
  imagecodecs/opj_color.c:125:14: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
       d0 = r = (int*)opj_image_data_alloc(sizeof(int) * max);
                ^
  imagecodecs/opj_color.c:126:14: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
       d1 = g = (int*)opj_image_data_alloc(sizeof(int) * max);
                ^
  imagecodecs/opj_color.c:127:14: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
       d2 = b = (int*)opj_image_data_alloc(sizeof(int) * max);
                ^
  imagecodecs/opj_color.c:142:5: warning: implicit declaration of function 'opj_image_data_free' [-Wimplicit-function-declaration]
       opj_image_data_free(img->comps[0].data);
       ^
  imagecodecs/opj_color.c:148:24: error: 'OPJ_CLRSPC_SRGB' undeclared (first use in this function)
       img->color_space = OPJ_CLRSPC_SRGB;
                          ^
  imagecodecs/opj_color.c:148:24: note: each undeclared identifier is reported only once for each function it appears in
  imagecodecs/opj_color.c: In function 'sycc422_to_rgb':
  imagecodecs/opj_color.c:177:14: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
       d0 = r = (int*)opj_image_data_alloc(sizeof(int) * max);
                ^
  imagecodecs/opj_color.c:178:14: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
       d1 = g = (int*)opj_image_data_alloc(sizeof(int) * max);
                ^
  imagecodecs/opj_color.c:179:14: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
       d2 = b = (int*)opj_image_data_alloc(sizeof(int) * max);
                ^
  imagecodecs/opj_color.c:236:24: error: 'OPJ_CLRSPC_SRGB' undeclared (first use in this function)
       img->color_space = OPJ_CLRSPC_SRGB;
                          ^
  imagecodecs/opj_color.c: In function 'sycc420_to_rgb':
  imagecodecs/opj_color.c:265:14: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
       d0 = r = (int*)opj_image_data_alloc(sizeof(int) * max);
                ^
  imagecodecs/opj_color.c:266:14: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
       d1 = g = (int*)opj_image_data_alloc(sizeof(int) * max);
                ^
  imagecodecs/opj_color.c:267:14: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
       d2 = b = (int*)opj_image_data_alloc(sizeof(int) * max);
                ^
  imagecodecs/opj_color.c:394:24: error: 'OPJ_CLRSPC_SRGB' undeclared (first use in this function)
       img->color_space = OPJ_CLRSPC_SRGB;
                          ^
  imagecodecs/opj_color.c: In function 'color_sycc_to_rgb':
  imagecodecs/opj_color.c:406:28: error: 'OPJ_CLRSPC_GRAY' undeclared (first use in this function)
           img->color_space = OPJ_CLRSPC_GRAY;
                              ^
  imagecodecs/opj_color.c: In function 'color_apply_icc_profile':
  imagecodecs/opj_color.c:525:21: error: 'OPJ_CLRSPC_SRGB' undeclared (first use in this function)
           new_space = OPJ_CLRSPC_SRGB;
                       ^
  imagecodecs/opj_color.c:605:26: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               in = inbuf = (unsigned char*)opj_image_data_alloc(nr_samples);
                            ^
  imagecodecs/opj_color.c:606:28: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               out = outbuf = (unsigned char*)opj_image_data_alloc(nr_samples);
                              ^
  imagecodecs/opj_color.c:643:26: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               in = inbuf = (unsigned short*)opj_image_data_alloc(nr_samples);
                            ^
  imagecodecs/opj_color.c:644:28: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               out = outbuf = (unsigned short*)opj_image_data_alloc(nr_samples);
                              ^
  imagecodecs/opj_color.c:684:26: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               in = inbuf = (unsigned char*)opj_image_data_alloc(nr_samples);
                            ^
  imagecodecs/opj_color.c:685:28: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               out = outbuf = (unsigned char*)opj_image_data_alloc(nr_samples);
                              ^
  imagecodecs/opj_color.c:686:17: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               g = (int*)opj_image_data_alloc((size_t)max * sizeof(int));
                   ^
  imagecodecs/opj_color.c:687:17: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               b = (int*)opj_image_data_alloc((size_t)max * sizeof(int));
                   ^
  imagecodecs/opj_color.c:744:26: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               in = inbuf = (unsigned short*)opj_image_data_alloc(nr_samples);
                            ^
  imagecodecs/opj_color.c:745:28: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               out = outbuf = (unsigned short*)opj_image_data_alloc(nr_samples);
                              ^
  imagecodecs/opj_color.c:746:17: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               g = (int*)opj_image_data_alloc((size_t)max * sizeof(int));
                   ^
  imagecodecs/opj_color.c:747:17: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
               b = (int*)opj_image_data_alloc((size_t)max * sizeof(int));
                   ^
  imagecodecs/opj_color.c: In function 'color_cielab_to_rgb':
  imagecodecs/opj_color.c:882:21: error: 'OPJ_CLRSPC_SRGB' undeclared (first use in this function)
           new_space = OPJ_CLRSPC_SRGB;
                       ^
  imagecodecs/opj_color.c:912:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
           red = dst0 = (int*)opj_image_data_alloc(max * sizeof(int));
                        ^
  imagecodecs/opj_color.c:913:24: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
           green = dst1 = (int*)opj_image_data_alloc(max * sizeof(int));
                          ^
  imagecodecs/opj_color.c:914:23: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
           blue = dst2 = (int*)opj_image_data_alloc(max * sizeof(int));
                         ^
  imagecodecs/opj_color.c: In function 'color_cmyk_to_rgb':
  imagecodecs/opj_color.c:1041:26: error: 'OPJ_CLRSPC_SRGB' undeclared (first use in this function)
       image->color_space = OPJ_CLRSPC_SRGB;
                            ^
  imagecodecs/opj_color.c: In function 'color_esycc_to_rgb':
  imagecodecs/opj_color.c:1125:26: error: 'OPJ_CLRSPC_SRGB' undeclared (first use in this function)
       image->color_space = OPJ_CLRSPC_SRGB;
                            ^
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for imagecodecs
  Running setup.py clean for imagecodecs
Failed to build imagecodecs
Could not build wheels for imagecodecs which use PEP 517 and cannot be installed directly

cgohlke commented 5 years ago

* wasn't able to use `imageformats` in ubuntu 16:04

Just tried with 16.04 LTS. The libraries are too out of date...

AllenCellModeling / czifile

Compression 4 is not supported #6