Kakadu causes defunct fcgi

[MichaelTran262]

Hello,

iipsrv children keep gradually going into defunct state, I traced logs with verbosity 10 and found this.

Kakadu :: error: Kakadu Core Error:
Kakadu :: error: Expected to find EPH marker following packet header.  Found Kakadu :: error: 0xKakadu :: error: 0Kakadu :: error:  instead.
Unsupported HTTP status code: 1

What exactly causes it? Bad image file or Kakadu itself? How can we fix this? Should I switch to OpenJpeg?

[MichaelTran262]

After a bit of more digging, I've noticed the defunct state is caused by one specific image with IIIF endpoint /full/max/0/default.jpg. However if I make an IIIF call on this image with downscaling /full/,1000/0/default.jpg, then it loads image just fine and the child will not go defunct.

[MichaelTran262]

I also found out, that endpoint /full/,1400/0/default.jpg is the boundary where image can still load. If I try /full/,1401/0/default.jpg then it fails and the child goes defunct.

[ruven]

From what's written in the log file, it looks like a problem with the image itself. But iipsrv should not go into a defunct state even if the image is corrupt. If it only happens with 1 particular image, could you share this image for me to test?

[MichaelTran262]

The image is here.

https://drive.google.com/drive/folders/14Y-DNRXJhRXOUvod-Pt3UHiyeMuzFPeq?usp=sharing

Thanks for help, hope it works out.

[ruven]

Your image works fine for me. Which version of Kakadu are you using exactly? Which version of iipsrv? And can you also try restarting your iipsrv instance, run the request which causes and defunct process and copy and paste all of what appears in the log file.

[MichaelTran262]

We are using KDU-7.10.7, iipsrv version 1.2. The logs are bellow, then it causes defunct state. Verbosity is set to 10.

Full Request is iiif=/mrln/797/bbf/797bbfd6447b46098adc2c31f5f54d46/2293ac5d-b7e9-4e76-b79b-28bda4752a00.jp2/full/max/0/default.jpg
[1/1]: Command / Argument is iiif : /mrln/797/bbf/797bbfd6447b46098adc2c31f5f54d46/2293ac5d-b7e9-4e76-b79b-28bda4752a00.jp2/full/max/0/default.jpg
IIIF handler reached
IIIF :: URL decoded to /mrln/797/bbf/797bbfd6447b46098adc2c31f5f54d46/2293ac5d-b7e9-4e76-b79b-28bda4752a00.jp2/full/max/0/default.jpg
FIF handler reached
FIF :: URL decoding/filtering: /mrln/797/bbf/797bbfd6447b46098adc2c31f5f54d46/2293ac5d-b7e9-4e76-b79b-28bda4752a00.jp2 => /mrln/797/bbf/797bbfd6447b46098adc2c31f5f54d46/2293ac5d-b7e9-4e76-b79b-28bda4752a00.jp2
FIF :: Image cache initialization
FIF :: JPEG2000 image detected
FIF :: Created image
FIF :: Image dimensions are 4293 x 5908
FIF :: Image contains 3 channels with 8 bits per channel
FIF :: Image timestamp: Fri, 08 Dec 2023 11:35:46 GMT
FIF :: Total command time 2335 microseconds
IIIF :: Requested Region (x, y, w. h): 0, 0, 4293, 5908 (ratios: 0, 0, 1, 1)
IIIF :: Requested Size: 4293x5908
IIIF :: Requested Rotation: 0 degrees
IIIF :: Requested Quality: default with format: jpg
IIIF :: image request for /mrln/797/bbf/797bbfd6447b46098adc2c31f5f54d46/2293ac5d-b7e9-4e76-b79b-28bda4752a00.jp2 with arguments: scaled region: 0,0,4293,5908; size: 4293x5908; rotation: 0; mirroring: 0
CVT handler reached
CVT :: Using resolution 5 with size 4293x5908
CVT :: No view port set
CVT :: Requested scaled region size is 4293x5908. Nearest existing resolution is 5 which has region with size 4293x5908
TileManager getRegion :: requesting region directly from image
Kakadu :: error: Kakadu Core Error:
Kakadu :: error: Expected to find EPH marker following packet header.  Found Kakadu :: error: 0xKakadu :: error: 0Kakadu :: error:  instead.
Unsupported HTTP status code: 1

It's running in Fedora 35 Docker contianer with these environment variables and dependencies:

ENV LOGFILE=/tmp/iipsrv.log \
    CHILDREN=6 \
    PORT=9000 \
    FILESYSTEM_PREFIX=/data/ \
    VERBOSITY=2 \
    JPEG_QUALITY=90 \
    MAX_IMAGE_CACHE_SIZE=10 \
    MAX_CVT=30000 \
    MAX_LAYERS=10 \
    LD_LIBRARY_PATH=/usr/local/lib

RUN dnf -y install libtool gcc-c++ libjpeg-turbo-devel libtiff-devel spawn-fcgi git cmake wget 
# required for iipsrv build
RUN dnf install -y libwebp-devel libzstd-devel jbigkit-devel zlib-devel libpng-devel
RUN dnf install -y memcached-devel libmemcached-awesome-devel
# required for kakadu
RUN dnf install -y java-11-openjdk-devel.x86_64
ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.17.0.8-2.fc35.x86_64/

[ruven]

I managed to track down the problem to a change in the Kakadu API in version 7.5, which was causing a crash if an exception occurs. iipsrv should no longer crash on this kind of corrupted image. Try out the fix here: https://github.com/ruven/iipsrv/commit/b2d5944376f3d01215d63971ce0b6368d34ecef4 and let me know if you still have any problems

[MichaelTran262]

It's fixed, a crash is not happening anymore, but the image won't be loaded at all. Is that intended? It's alright if it is intended, we will try to fix the image.

[ruven]

The image has a problem, so, yes iipsrv will intentionally return an error. (Note that OpenJPEG also fails on this image).

However, if you really need this image to work, Kakadu is able to do a certain amount of error correction on corrupted code-streams. But, this kind of error-correction makes decoding much slower, so this is not activated by default. If you set the KAKADU_READMODE startup environment variable to 2 (KAKADU_READMODE=2), this will activate Kakadu's "resilient" mode and your image will load correctly. Of course, the best option, if possible, would be to simply fix the image.