search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.68k stars 1.79k forks source link

[BUG] Netty Transport test failing with large responses #1847

Closed saratvemulapalli closed 1 year ago

saratvemulapalli commented 2 years ago

Coming from https://github.com/opensearch-project/OpenSearch/pull/1834#issuecomment-1005098940

REPRODUCE WITH: ./gradlew ':modules:transport-netty4:test' --tests "org.opensearch.http.netty4.Netty4HttpServerTransportTests.testLargeCompressedResponse" -Dtests.seed=49B97FF7A35F7849 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sl -Dtests.timezone=Europe/Uzhgorod -Druntime.java=15

org.opensearch.http.netty4.Netty4HttpServerTransportTests > testLargeCompressedResponse FAILED
    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=273, name=Thread-5, state=RUNNABLE, group=TGRP-Netty4HttpServerTransportTests]

        Caused by:
        java.lang.AssertionError
            at __randomizedtesting.SeedInfo.seed([49B97FF7A35F7849]:0)
            at org.opensearch.http.netty4.Netty4HttpServerTransportTests$3.dispatchRequest(Netty4HttpServerTransportTests.java:357)

    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=275, name=Thread-6, state=RUNNABLE, group=TGRP-Netty4HttpServerTransportTests]

        Caused by:
        java.lang.AssertionError
            at __randomizedtesting.SeedInfo.seed([49B97FF7A35F7849]:0)
            at org.opensearch.http.netty4.Netty4HttpServerTransportTests$3.dispatchRequest(Netty4HttpServerTransportTests.java:357)
meghasaik commented 2 years ago

I ran this command for reproducing this flaky test: for i in $(seq 0 250) ; do echo "Iteration: $i" && ./gradlew clean > /dev/null 2>&1 && ./gradlew ':modules:transport-netty4:test' --tests "org.opensearch.http.netty4.Netty4HttpServerTransportTests.testLargeCompressedResponse" -Dtests.seed=49B97FF7A35F7849 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sl -Dtests.timezone=Europe/Uzhgorod -Druntime.java=15 >> /Users/kavmegha/documents/test-output-2022-01-11.txt 2>&1 ; done

I tried running it in iterations of 50, 100, 150, 250 and 350. Still, this issue was not reproducible. The Results:

Screen Shot 2022-01-12 at 11 57 20 AM

As mentioned here: running this in isolation may not yield much in terms of reproducibility.

owaiskazi19 commented 1 year ago

Tried running the test 400 times in isolation and it passed


./gradlew ':modules:transport-netty4:test' --tests "org.opensearch.http.netty4.Netty4HttpServerTransportTests.testLargeCompressedResponse" -Dtests.seed=49B97FF7A35F7849 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sl -Dtests.timezone=Europe/Uzhgorod -Druntime.java=15 -Dtests.iters=400

> Configure project :qa:os
Cannot add task 'destructiveDistroTest.docker' as a task with that name already exists.
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.6
  OS Info               : Linux 5.11.0-1020-aws (amd64)
  Runtime JDK Version   : 15 (Oracle JDK)
  Runtime java.home     : /usr/lib/jvm/java-15/jdk-15
  Gradle JDK Version    : 17 (Oracle JDK)
  Gradle java.home      : /usr/lib/jvm/java-17/jdk-17.0.1
  Random Testing Seed   : 49B97FF7A35F7849
  In FIPS 140 mode      : false
=======================================

BUILD SUCCESSFUL in 16m 56s
dblock commented 1 year ago

Let's close until we see it again.