Invoke blocks and do not time out if active fails and no server left

[mathieucarbou]

We start integrating galvan in m&m with @anthonydahanne and I have the SAME issue than with passthrough: invoke calls are bloquant.

1. I connect to a server, ask to create an entity
2. entity creation fails, server is killed
3. test tear down method: management entity client sends a message to the server entity (which is closed)
4. it blocks indefinitely in the invoke.

"main" #1 prio=5 os_prio=31 tid=0x00007fb8e1802000 nid=0xe03 in Object.wait() [0x000070000f86f000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x000000076eb18a48> (a org.terracotta.passthrough.PassthroughConnectionState)
    at java.lang.Object.wait(Object.java:502)
    at org.terracotta.passthrough.PassthroughConnectionState.sendNormal(PassthroughConnectionState.java:59)
    - locked <0x000000076eb18a48> (a org.terracotta.passthrough.PassthroughConnectionState)
    at org.terracotta.passthrough.PassthroughConnection.invokeAndWait(PassthroughConnection.java:155)
    at org.terracotta.passthrough.PassthroughConnection.invokeActionAndWaitForAcks(PassthroughConnection.java:147)
    at org.terracotta.passthrough.PassthroughInvocationBuilder.invoke(PassthroughInvocationBuilder.java:113)
    at org.terracotta.voltron.proxy.client.VoltronProxyInvocationHandler.invoke(VoltronProxyInvocationHandler.java:103)
    at com.sun.proxy.$Proxy18.pushNotification(Unknown Source)

It is the exact same issue as this one: https://github.com/Terracotta-OSS/tc-passthrough-testing/issues/70, but with Galvan, and not in a failover case this time: only 1 active, that fails.

threaddump-1481905905864.tdump.txt

[ljacomet]

I may be missing something but the fact that you have a PassthroughConnection in that thread dump seems fishy!

[mathieucarbou]

We only use Galvan dependencies and we test our sample entity @ljacomet: here is our first simple test.

<dependency>
      <groupId>org.terracotta</groupId>
      <artifactId>galvan-support</artifactId>
      <version>${galvan.version}</version>
      <scope>test</scope>
    </dependency>

We even see all the server logs flowing correctly, until an exception (on our side when creating the sample server entity), which kills the server.

public class SimpleGalvanIT {

  private static final String OFFHEAP_RESOURCE = "primary-server-resource";

  private static final String RESOURCE_CONFIG =
      "<config xmlns:ohr='http://www.terracotta.org/config/offheap-resource'>"
          + "<ohr:offheap-resources>"
          + "<ohr:resource name=\"" + OFFHEAP_RESOURCE + "\" unit=\"MB\">64</ohr:resource>"
          + "</ohr:offheap-resources>" +
          "</config>\n";

  @ClassRule
  public static Cluster CLUSTER =
      new BasicExternalCluster(new File("target/galvan"), 1, emptyList(), "", RESOURCE_CONFIG, "");

  @BeforeClass
  public static void waitForActive() throws Exception {
    CLUSTER.getClusterControl().waitForActive();
  }
  @Test
  public void simpleTest_one_active() throws Exception {
    StatisticConfiguration statisticConfiguration = new StatisticConfiguration()
        .setAverageWindowDuration(1, TimeUnit.MINUTES)
        .setHistorySize(100)
        .setHistoryInterval(1, TimeUnit.SECONDS)
        .setTimeToDisable(5, TimeUnit.SECONDS);
    CacheFactory cacheFactory = new CacheFactory(CLUSTER.getConnectionURI().resolve("/pif"), statisticConfiguration);
    cacheFactory.init();
    try {
      cacheFactory.getCache("paf"); // entity creation is there, it fails on server
    } finally {
      cacheFactory.close();
    }
  }
}