filibuster test suite flaky/broken, 2.0 regression

[cmeiklejohn]

this needs to be fixed today

if you run the WorldTest in isolation, it's fine.

however, if you run it with other tests, it's failing -- passing test executions will fail because wasFaultInjected returns true when it should be false. this is regression introduced by 2.0 and could be causing problems elsewhere.

to reproduce, run the entire suite with the debugger, but break on wasFaultInjected = true in the passing branch. I don't know why it's running byzantine fault injection code -- this code doesn't have an exception handler and shouldn't be run which seems to indicate state leakage from a previous BFI test.

Aug 08, 2023 1:01:21 PM cloud.filibuster.instrumentation.instrumentors.FilibusterServerInstrumentor lambda$beforeInvocation$0
INFO: making call to filibuster server, update with body: {"target_service_name":"world","execution_index":"[[\"V1-[TestScope+0+DEFAULT]-aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d-308d1419e1ba0da4af15c810881dec2f4c11dba9-21b91ff4ab27dc03fa7f8acfcc0d03add457ceb8-b47b124da5826697c7722dd3c7f2dabf088ca246\", 1]]","instrumentation_type":"request_received","generated_id":"5"}
Aug 08, 2023 1:01:21 PM cloud.filibuster.instrumentation.instrumentors.FilibusterServerInstrumentor lambda$beforeInvocation$0
INFO: URI: http://localhost:5005/
Aug 08, 2023 1:01:21 PM cloud.filibuster.junit.server.core.FilibusterCore endInvocation
INFO: [FILIBUSTER-CORE]: endInvocation called
Aug 08, 2023 1:01:21 PM cloud.filibuster.junit.server.core.FilibusterCore endInvocation
INFO: [FILIBUSTER-CORE]: endInvocation called, distributedExecutionIndex: [["V1-[TestScope+0+DEFAULT]-aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d-308d1419e1ba0da4af15c810881dec2f4c11dba9-21b91ff4ab27dc03fa7f8acfcc0d03add457ceb8-b47b124da5826697c7722dd3c7f2dabf088ca246", 1]]
13:01:21.526 [armeria-common-worker-nio-2-12] WARN com.linecorp.armeria.server.DefaultServerErrorHandler - [sreqId=7c839958, chanId=f99ba9a6, raddr=127.0.0.1:52274, laddr=127.0.0.1:5005][h2c://tx17169q4v/filibuster/update#POST] Unhandled exception from a service:
cloud.filibuster.exceptions.filibuster.FilibusterFaultInjectionException: [FILIBUSTER-CORE]: scheduleByzantineFault, payload missing module or method: {
    "target_service_name": "world",
    "execution_index": "[[\"V1-[TestScope+0+DEFAULT]-aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d-308d1419e1ba0da4af15c810881dec2f4c11dba9-21b91ff4ab27dc03fa7f8acfcc0d03add457ceb8-b47b124da5826697c7722dd3c7f2dabf088ca246\", 1]]",
    "instrumentation_type": "request_received",
    "generated_id": "5"
}
    at cloud.filibuster.junit.server.core.FilibusterCore.generateByzantineAndTransformerFaults(FilibusterCore.java:420)
    at cloud.filibuster.junit.server.core.FilibusterCore.scheduleByzantineAndTransformerFaults(FilibusterCore.java:413)
    at cloud.filibuster.junit.server.core.FilibusterCore.endInvocation(FilibusterCore.java:380)
    at cloud.filibuster.junit.server.local.FilibusterServer$2.update(FilibusterServer.java:47)
    at com.linecorp.armeria.internal.server.annotation.AnnotatedService.invoke(AnnotatedService.java:394)
    at com.linecorp.armeria.internal.server.annotation.AnnotatedService.lambda$serve0$4(AnnotatedService.java:343)
    at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
    at com.linecorp.armeria.common.util.UnmodifiableFuture.doComplete(UnmodifiableFuture.java:124)
    at com.linecorp.armeria.common.util.UnmodifiableFuture.lambda$wrap$0(UnmodifiableFuture.java:100)
    at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
    at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
    at com.linecorp.armeria.common.stream.StreamMessageCollector.onComplete(StreamMessageCollector.java:67)
    at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:257)
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent0(DefaultStreamMessage.java:291)
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:283)
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:420)
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:363)
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber(DefaultStreamMessage.java:319)
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.addObjectOrEvent(DefaultStreamMessage.java:305)
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.close(DefaultStreamMessage.java:427)
    at com.linecorp.armeria.server.Http2RequestDecoder.onDataRead(Http2RequestDecoder.java:237)
    at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onDataRead(Http2FrameListenerDecorator.java:36)

[cmeiklejohn]

[cmeiklejohn]

branch failures: https://github.com/filibuster-testing/filibuster-java-instrumentation/pull/241

[cmeiklejohn]

this code should not be throwing here, because request_received will not have a method or module set, only a DEI, IIRC. this could be contributing to the cause, I don't know. it could be that a non-200 is going back to the client because of the unhandled exception and that is causing something later to fail, it's unclear.

[cmeiklejohn]

WorldTest needs to be fixed, we have to identify the root cause because it's a clear indication of a bug that is preventing merging of PRs.

[IMichael-ops]

On it!

[cmeiklejohn]

on that branch:

[IMichael-ops]

Closed for now since tests are passing