mock-server / mockserver

MockServer enables easy mocking of any system you integrate with via HTTP or HTTPS with clients written in Java, JavaScript and Ruby. MockServer also includes a proxy that introspects all proxied traffic including encrypted SSL traffic and supports Port Forwarding, Web Proxying (i.e. HTTP proxy), HTTPS Tunneling Proxying (using HTTP CONNECT) and SOCKS Proxying (i.e. dynamic port forwarding).
http://mock-server.com
Apache License 2.0
4.6k stars 1.07k forks source link

Blocking/deadlock/response not received exceptions when creating expectations from response callbacks #577

Closed ininrobu closed 5 years ago

ininrobu commented 5 years ago

First off, thanks for quickly fixing issue #570! Unfortunately it looks like the fix for that issue introduced another - when creating expectations from within client-side response callbacks I'm now seeing requests time out and/or netty blocking exceptions being thrown. Here's an example of the blocking exception, which seems to be the most common:

14:15:14.620 ERROR o.m.c.n.w.WebSocketClientHandler - web socket client caught exception
org.mockserver.client.netty.websocket.WebSocketException: Exception while receiving web socket message
    at org.mockserver.client.netty.websocket.WebSocketClient.receivedTextWebSocketFrame(WebSocketClient.java:105)
    at org.mockserver.client.netty.websocket.WebSocketClientHandler.channelRead0(WebSocketClientHandler.java:79)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: org.mockserver.client.netty.websocket.WebSocketException: Exception while starting web socket client
    at org.mockserver.client.netty.websocket.WebSocketClient.<init>(WebSocketClient.java:77)
    at org.mockserver.client.ForwardChainExpectation.initWebSocketClient(ForwardChainExpectation.java:180)
    at org.mockserver.client.ForwardChainExpectation.respond(ForwardChainExpectation.java:81)
    at com.test.robu.MockServerIssue.createCallbackExpectation(MockServerIssue.java:20)
    at com.test.robu.MockServerIssue.lambda$testExpectations$1(MockServerIssue.java:35)
    at org.mockserver.client.netty.websocket.WebSocketClient.receivedTextWebSocketFrame(WebSocketClient.java:92)
    ... 22 common frames omitted
Caused by: io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@7c08a754(incomplete)
    at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:395)
    at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:159)
    at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:225)
    at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:131)
    at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:30)
    at io.netty.util.concurrent.DefaultPromise.sync(DefaultPromise.java:337)
    at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:119)
    at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:30)
    at org.mockserver.client.netty.websocket.WebSocketClient.<init>(WebSocketClient.java:47)

Sometimes requests simply time out though, with an error logged like so: org.mockserver.client.netty.SocketCommunicationException: Response was not received from MockServer after 20000 milliseconds, to make the proxy wait longer please use "mockserver.maxSocketTimeout" system property or ConfigurationProperties.maxSocketTimeout(long milliseconds) These requests take only milliseconds so I'm sure it's simply the same blocking/deadlock issue manifesting slightly differently.

These issues are only present in version 5.5.1. Everything works as expected on 5.5.0.

Here is an example class that reproduces the issue: https://gist.github.com/ininrobu/c14d2dd53879f0e81e505c836e81caa7

jamesdbloom commented 5 years ago

I'm not sure this is an issue I can solve without adding considerable overhead. The crux of the issue is that you are re-using the same MockServerClient instance inside a web socket call back handled by the MockServerClient. The problem is that these share EventLoopGroup but also have Futures, etc to ensure the right things occur in the correct order. This means that is a Future is blocked waiting on a task on the same EventLoop you have a deadlock because that task won't get scheduled because of the blocking call to the Future. I may be able to unpick this but initial basic attempts don't look promising unless I can remove all futures and replace them with callbacks. That is particularly hard because inside the MockServer everything is asynchronous but the MockServerClient API isn't. I'll try but in the mean time the simple solution is as follows:

import org.mockserver.client.MockServerClient;
import org.mockserver.integration.ClientAndServer;
import org.mockserver.mock.action.ExpectationResponseCallback;
import org.mockserver.model.HttpRequest;
import org.mockserver.model.HttpResponse;

import java.net.HttpURLConnection;
import java.net.URL;
import java.util.UUID;

import static org.mockserver.model.HttpRequest.request;
import static org.mockserver.model.HttpResponse.response;

public class MockServerIssue {

    public static void main(String[] args) throws Exception {
        new MockServerIssue().testExpectations();
    }

    private final MockServerClient mockServerClient = new ClientAndServer(8080);

    private void createCallbackExpectation(final String path,
                                           final String method) {
        new MockServerClient("localhost", 8080)
//        mockServerClient
                .when(request().withMethod(method).withPath(path))
                .respond(new ExpectationResponseCallback() {
                    @Override
                    public HttpResponse handle(HttpRequest request) {
                        // Actual logic omitted, typically some sort of serde/bookkeeping
                        return response().withBody(String.format("Contrived payload from %s %s", method, path));
                    }
                });
    }

    /**
     * Call this to reproduce the issue
     */
    private void testExpectations() throws Exception {
        mockServerClient
                .when(request().withMethod("GET").withPath("/reproduce"))
                .respond(new ExpectationResponseCallback() {
                    @Override
                    public HttpResponse handle(HttpRequest request) {
                        final String id = UUID.randomUUID().toString();
                        final String path = String.format("/resources/%s", id);
                        createCallbackExpectation(path, "GET");
                        createCallbackExpectation(path, "PUT");
                        createCallbackExpectation(path, "PATCH");
                        createCallbackExpectation(path, "DELETE");
                        return response().withBody(id);
                    }
                });

        final URL url = new URL("http://localhost:8080/reproduce");
        final HttpURLConnection connection = (HttpURLConnection) url.openConnection();
        connection.setRequestMethod("GET");
        System.out.println("Got response code: " + connection.getResponseCode());
    }
}
jamesdbloom commented 5 years ago

It won't be possible to improve this without adding too much overhead by creating too many thread pools (one per object callback). Therefore I've added a meaningful error about not reusing the MockServerClient. That way it is up to the user to use multiple MockServerClient instances (each one has a thread pool) as necessary. I'm going to close this ticket as this is the best possible without other negative impacts.