esmero / archipelago-docker-images

This holds our master docker images used in Archipelago deployment
GNU General Public License v3.0
2 stars 3 forks source link

Add GrokProcessor to esmero-cantaloupe #30

Open aksm opened 3 years ago

aksm commented 3 years ago

What?

GrokProcessor is a nice middle ground between OpenJPEG and Kakadu (see benchmarks). Unfortunately, despite the reported bugs being resolved, it's still being held in Debian sid (unstable), and compiling from source isn't possible because of compiler requirements. Precompiled binaries for 9.2.0 are only partially working with some images but showing the following errors for certain JP2 sources:

15:51:55.761 [qtp572868060-152] DEBUG e.i.l.c.r.ImageRepresentation - write(): Unsupported source format: JPEG2000
edu.illinois.library.cantaloupe.processor.SourceFormatException: Unsupported source format: JPEG2000
    at edu.illinois.library.cantaloupe.processor.GrokProcessor.process(GrokProcessor.java:406)
    at edu.illinois.library.cantaloupe.resource.ImageRepresentation.process(ImageRepresentation.java:205)
    at edu.illinois.library.cantaloupe.resource.ImageRepresentation.copyOrProcess(ImageRepresentation.java:167)
    at edu.illinois.library.cantaloupe.resource.ImageRepresentation.write(ImageRepresentation.java:130)
    at edu.illinois.library.cantaloupe.resource.ImageRequestHandler.handle(ImageRequestHandler.java:405)
    at edu.illinois.library.cantaloupe.resource.iiif.v2.ImageResource.doGET(ImageResource.java:128)
    at edu.illinois.library.cantaloupe.resource.HandlerServlet.handle(HandlerServlet.java:97)
    at edu.illinois.library.cantaloupe.resource.HandlerServlet.doGet(HandlerServlet.java:35)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
    at java.base/java.lang.Thread.run(Thread.java:832)

Some debugging/testing will be required. Maybe trying a later version? Compiling with different flags?

DiegoPino commented 3 years ago

@aksm could you share your full cmake command? Thanks!

aksm commented 3 years ago

@DiegoPino Sorry, forgot to mention that it was the release build that generated those errors. I'll try building it manually today.

aksm commented 3 years ago

@DiegoPino Ok, images seem to be loading now after building, but I still see the above in the logs. Thoughts? Should I open an issue with Cantaloupe?

DiegoPino commented 3 years ago

@aksm that means it can not process JPEG2000 which is bad. There is probably a missing dependency in the make file. Let me check

DiegoPino commented 3 years ago

Ok, that was quick. Since I do not know how you are building it, but I do know that this here works, can you compare notes?

https://github.com/cantaloupe-project/cantaloupe/blob/develop/docker/Linux-JDK16/Dockerfile#L1-L67

Also, JDK-16... which one are we using??

There are other JDK examples there. But in all Grok passes compile... give it a try

aksm commented 3 years ago

@DiegoPino I'm just building locally with default flags and then copying the binaries into the container. I was trying to go for something from 9.2+ since the benchmarks are for 9.2 and since the buffer overflow bug seems to have been fixed from that point forward, but maybe 7.6.5 is not so much worse in performance? In any case, I'll give this a go and see.

We're on JDK-15.

DiegoPino commented 3 years ago

Comment I left on slack:

https://github.com/GrokImageCompression/grok/releases/download/v9.2.0/grokj2k-tools_9.2.0-1_amd64.deb

aksm commented 3 years ago

Ok, I tried with a base image of Ubuntu 21.10 and the official apt package and ran into the same exact error so something else is going on. I'll try building from source with a newer version of grok since this base box should have all the dependencies.

DiegoPino commented 3 years ago

Ok, it may be one of the dependencies then. Which is good because we do not need to change (once you figure out which one) the current base box