HTTP Compression not working with quarkus-amazon-lambda-rest dependency

[itzsuman]

Describe the bug

No http compression happening in my rest api if I add quarkus-amazon-lambda-rest maven dependency.

Expected behavior

Http compression should occur

Actual behavior

No response

How to Reproduce?

Reproduce :

gzip-reactive.zip

1. mvn clean compile quarkus:dev
2. open a browser
3. open devtools / network tab
4. visit http://localhost:8080/hello/2 in devtools, you can see that no compression used for the response

Output of uname -a or ver

Microsoft Windows [Version 10]

Output of java -version

OpenJDK Runtime Environment Temurin-17.0.5

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.1.2.Final

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

I have followed this link https://quarkus.io/guides/resteasy-reactive#http-compression

[quarkus-bot[bot]]

/cc @matejvasek (amazon-lambda), @patriot1burke (amazon-lambda)

[patriot1burke]

Did it ever work for you? I ask this because I removed transfer encoding support as chunked was screwing things up.

[itzsuman]

yes its working if I not add quarkus-amazon-lambda-rest maven dependency. If you remove the dependency from the attached project then it should work.

[patriot1burke]

No, I mean did it ever work with Lambda. We might be at the mercy of the AWS API Gateway

[itzsuman]

I have a scenario where lambda may return more than 6 mb of data . Since knowing the limit of AWS lambda, we cant give back more than 6MB. And if we enable API gateway encoding, it will not fulfill our scenario as we get lambda timeout for more than 6mb data .. hence we want compression at the lambda level.. But, quarkus-amazon-lambda-rest blocking the gzip/compression.. I have tried in my local, compression only works if I do not include quarkus-amazon-lambda-rest dependency.

[itzsuman]

@patriot1burke Did you get any chance to look into this issue. Please let me know if you have any solution.

[itzsuman]

@geoand @patriot1burke @matejvasek Please help me on this.

[Serkan80]

@mkouba

I can confirm that this also doesnt work when you have resteasy-reactive-jackson in your pom.xml (using Quarkus 3.2.1.Final)

[mkouba]

@mkouba

I can confirm that this also doesnt work when you have resteasy-reactive-jackson in your pom.xml (using Quarkus 3.2.1.Final)

Unfortunately, I have no idea how quarkus-amazon-lambda-rest maps the JAX-RS resources to AWS and vice versa.

[itzsuman]

@mkouba I can confirm that this also doesnt work when you have resteasy-reactive-jackson in your pom.xml (using Quarkus 3.2.1.Final)

Unfortunately, I have no idea how quarkus-amazon-lambda-rest maps the JAX-RS resources to AWS and vice versa.

I have tried with WriterInterceptor to add Content-Encoding gzip using the below mentioned code. But getting QuarkusErrorHandler java.lang.RuntimeException: java.lang.NoSuchMethodException in lambda.

    @Compress
    @RegisterForReflection
    @Provider
    public class GzipFilter implements WriterInterceptor {

        @Override
        public void aroundWriteTo(WriterInterceptorContext context)
                throws IOException, WebApplicationException {
            MultivaluedMap<String, Object> headers = context.getHeaders();
            headers.add(HttpHeaders.CONTENT_ENCODING, "gzip");
            final OutputStream outputStream = context.getOutputStream();
            context.setOutputStream(new GZIPOutputStream(outputStream));
            context.proceed();
        }
    }

@compress annotation is the name binding annotation

@NameBinding
@retention(RetentionPolicy.RUNTIME)
public @interface Compress {}

[itzsuman]

Fixed the above issue with

@RegisterForReflection(targets = {Resource.class})
public class YourReflectionConfig {}

Now, My GZIP’d/compressed response looks like: {J?J??t??`$?@??????....

However when I try to consume this gzip'd response using quarkus rest client and for decompress using ReaderInterceptor/ GzipInputStream

  @Override
  public Object aroundReadFrom(ReaderInterceptorContext context)
      throws IOException, WebApplicationException 
      {
        final InputStream originalInputStream = context.getInputStream();
        context.setInputStream(new GZIPInputStream(originalInputStream));
        return context.proceed();
      }

getting ' Not a gzip format' error.

Any thoughts or suggestions @mkouba @geoand @Serkan80 @patriot1burke @Postremus

[geoand]

@itzsuman what the entire stacktrace?

[itzsuman]

@itzsuman what the entire stacktrace?

ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /test/v1/resource failed, error id: 86d95716-2ee1-4eb1-936a-098e5e657777-1: jakarta.ws.rs.ProcessingException: Not in GZIP format
        at org.jboss.resteasy.reactive.client.impl.InvocationBuilderImpl.unwrap(InvocationBuilderImpl.java:223)
        at org.jboss.resteasy.reactive.client.impl.InvocationBuilderImpl.method(InvocationBuilderImpl.java:324)
        at com.test.rest.client.Client$$QuarkusRestClientInterface.getList(Unknown Source)
        at com.test.rest.client.Client$$CDIWrapper.getList(Unknown Source)
        at com.test.rest.client.Client$$CDIWrapper_ClientProxy.getList(Unknown Source)
        at com.test.bags.resource.TestResource.getLoad(TestResource.java:73)
        at com.test.bags.resource.TestResource$quarkusrestinvoker$getLoad_6cace0981d7af1a0ee81fd1133d6095f83f1a64e.invoke(Unknown Source)
        at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
        at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:145)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:576)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.zip.ZipException: Not in GZIP format
        at java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:166)
        at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:80)
        at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:92)
        at com.test.config.GZIPReadInterceptor.aroundReadFrom(GZIPReadInterceptor.java:23)
        at org.jboss.resteasy.reactive.client.impl.ClientReaderInterceptorContextImpl.proceed(ClientReaderInterceptorContextImpl.java:133)
        at org.jboss.resteasy.reactive.client.impl.ClientSerialisers.invokeClientReader(ClientSerialisers.java:160)
        at org.jboss.resteasy.reactive.client.impl.RestClientRequestContext.readEntity(RestClientRequestContext.java:199)
        at org.jboss.resteasy.reactive.client.handlers.ClientResponseCompleteRestHandler.mapToResponse(ClientResponseCompleteRestHandler.java:104)
        at org.jboss.resteasy.reactive.client.handlers.ClientResponseCompleteRestHandler.handle(ClientResponseCompleteRestHandler.java:35)
        at org.jboss.resteasy.reactive.client.handlers.ClientResponseCompleteRestHandler.handle(ClientResponseCompleteRestHandler.java:31)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.invokeHandler(AbstractResteasyReactiveContext.java:229)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:145)
        at org.jboss.resteasy.reactive.client.impl.RestClientRequestContext$1.lambda$execute$0(RestClientRequestContext.java:304)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:246)
        at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:43)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        ... 2 more

Compressed using this filter code. @geoand

public void filter(ContainerRequestContext requestContext,
      ContainerResponseContext responseContext) throws IOException {
    String encoding = requestContext.getHeaderString(HttpHeaders.ACCEPT_ENCODING);
    if (encoding != null && encoding.contains("gzip")) {
      responseContext.getHeaders().put(HttpHeaders.CONTENT_ENCODING, Arrays.asList("gzip"));
      OutputStream outputStream = responseContext.getEntityStream();
      responseContext.setEntityStream(new GZIPOutputStream(outputStream));
    }
  }

[geoand]

That looks very weird as it includes JAX-RS Client elements.

[itzsuman]

is there anything wrong in compression logic.

[geoand]

Is the reproducer attached in the description up to date? Can I use it to see the stacktrace you mentioned?

[itzsuman]

Is the reproducer attached in the description up to date? Can I use it to see the stacktrace you mentioned?

No, interceptor one is not attached . Attached one was quarkus HTTP Compression https://quarkus.io/guides/resteasy-reactive#http-compression but with aws-lambda dependency that was not working . Hence tried with Interceptor .

[geoand]

Do you have that experiment anywhere I can try?

[itzsuman]

yes , @geoand you can check the reproducer attached one. If you remove the lambda dependency from pom then only compression happen.With lambda dependency it won't work.

[itzsuman]

Do you have that experiment anywhere I can try?

@geoand Please let me know if you want any update on attached reproducer . Reproducer one based on quarkus HTTP Compression but not working with aws lambda.

[geoand]

I have not had to test it unfortunately, it will likely have to wait until next week.

[Serkan80]

I just want to remind that compression also doesn’t work when resteasy-reactive-jackson is on the classpath.

[itzsuman]

I have not had to test it unfortunately, it will likely have to wait until next week.

@geoand Did you get any chance to check the issue.

[geoand]

Nope, as I was out for a conference last week and as a result I have a large backlog

On Tue, Oct 10, 2023, 21:24 itzsuman @.***> wrote:

I have not had to test it unfortunately, it will likely have to wait until next week.

@geoand https://github.com/geoand Did you get any chance to check the issue.

— Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/34472#issuecomment-1755994164, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBMDP36Z5EYPH6P7YR4SYDX6WHGJAVCNFSM6AAAAAAZ4G5IH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJVHE4TIMJWGQ . You are receiving this because you were mentioned.Message ID: @.***>

[geoand]

I just want to remind that compression also doesn’t work when resteasy-reactive-jackson is on the classpath.

Do you have a sample for this?

cc @mkouba

[mkouba]

Like I mentioned earlier, I have no idea how quarkus-amazon-lambda-rest works internally but it seems that in the dev mode, Quarkus starts a mock AWS Lambda server, so if you do HTTP GET http://localhost:8080/hello/2 then the request hits the mock server, is converted to a JSON message that is consumed by the "Quarkus Lambda Poll loop"; i.e. not by the regular HTTP server. So the question is how the mock server handles the requests... :shrug:

[geoand]

I think what @Serkan80 was saying that the problem manifests without Amazon Lambda

[mkouba]

I think what @Serkan80 was saying that the problem manifests without Amazon Lambda

I see, we'll need a dedicated reproducer then...

[itzsuman]

@geoand @mkouba For the time being , can we have any workaround with quarkus-amazon-lambda-rest

[geoand]

That would likely require input from @patriot1burke

[Alexander-Krause-Glau]

I am using Quarkus 3.2.9.Final and io.quarkus:quarkus-resteasy-reactive-jackson (among others) and face the same problem. Any updates?