search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.43k stars 1.72k forks source link

[CI] too many disjunctions to expand failure in IntervalQueryBuilderTests.testMustRewrite #3321

Open nknize opened 2 years ago

nknize commented 2 years ago

Caught on #3318 the following failure is reproducible:

./gradlew ':server:test' --tests "org.opensearch.index.query.IntervalQueryBuilderTests.testMustRewrite" -Dtests.seed=A6236EB12532F6E5 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=America/Rosario -Druntime.java=17
org.opensearch.index.query.IntervalQueryBuilderTests > testMustRewrite FAILED
    java.lang.IllegalArgumentException: Too many disjunctions to expand
        at __randomizedtesting.SeedInfo.seed([A6236EB12532F6E5:12CC1D8C5D65F065]:0)
        at org.apache.lucene.queries.intervals.Disjunctions.pullUp(Disjunctions.java:45)
        at org.apache.lucene.queries.intervals.OrderedIntervalsSource.pullUpDisjunctions(OrderedIntervalsSource.java:91)
        at org.apache.lucene.queries.intervals.FilteredIntervalsSource.maxGaps(FilteredIntervalsSource.java:33)
        at org.apache.lucene.queries.intervals.Intervals.maxgaps(Intervals.java:302)
        at org.opensearch.index.query.IntervalBuilder.combineSources(IntervalBuilder.java:173)
        at org.opensearch.index.query.IntervalsSourceProvider$Combine.getSource(IntervalsSourceProvider.java:468)

Looks related to #2103. A large test query is being created that is resulting in a 1152 disjunctions which exceeds IndexSearcher.maxClauseCount (1024). With the addition of the unordered non-overlapping intervals support we should probably make the test less aggressive here. Is this something you can look at @mattweber?

Test query:

{
  "intervals" : {
    "mapped_string" : {
      "all_of" : {
        "mode" : "ORDERED",
        "max_gaps" : 3,
        "intervals" : [
          {
            "match" : {
              "query" : "ഭകധള൤൛a",
              "max_gaps" : 0,
              "mode" : "UNORDERED_NO_OVERLAP",
              "analyzer" : "simple"
            }
          },
          {
            "any_of" : {
              "intervals" : [
                {
                  "any_of" : {
                    "intervals" : [
                      {
                        "match" : {
                          "query" : "ᥭᥙᥕᥨ᥹ᥧ᥾᥮ᥨᥒᥩᥛᥭ᥼ᥔa ᱳᱸᱧᱵᱻᱡᱧᱦᱹᱪ᱾᱔ᱹᱰa ᰑᰩᰊa \uD800\uDDD2\uD800\uDDFDaa \uD801\uDCA3\uD801\uDC99\uD801\uDC88\uD801\uDC9D\uD801\uDC95\uD801\uDCACa",
                          "max_gaps" : 3,
                          "mode" : "UNORDERED_NO_OVERLAP",
                          "analyzer" : "whitespace"
                        }
                      },
                      {
                        "match" : {
                          "query" : "㆟㆟㆜a",
                          "max_gaps" : 0,
                          "mode" : "ORDERED",
                          "analyzer" : "keyword"
                        }
                      },
                      {
                        "match" : {
                          "query" : "\uD802\uDF74\uD802\uDF6A\uD802\uDF6B\uD802\uDF68\uD802\uDF72\uD802\uDF6D\uD802\uDF7Eaa ㍬㍞㎗㍠㌺㎵㎕㏲a ▎▇▀▒▏▇▒▜▁▌▊▘▍▖a \uDBCD\uDEC4aa",
                          "max_gaps" : 3,
                          "mode" : "UNORDERED",
                          "analyzer" : "whitespace"
                        }
                      },
                      {
                        "match" : {
                          "query" : "₽₴₲⃎⃁⃉₥a ⡩⢺⣔⣒⡐⢵⡥⠉⠾⢭a",
                          "max_gaps" : 4,
                          "mode" : "ORDERED",
                          "analyzer" : "whitespace"
                        }
                      },
                      {
                        "match" : {
                          "query" : "꤫ꤩꤒꤔ꤅ꤐꤔa ՉԺժնԵճՓԵa \uD808\uDD91\uD808\uDF21\uD808\uDE8C\uD808\uDC6B\uD808\uDF65\uD808\uDDD9aa \uD808\uDDA5\uD808\uDC73aa",
                          "max_gaps" : 4,
                          "mode" : "UNORDERED",
                          "analyzer" : "keyword"
                        }
                      }
                    ]
                  }
                },
                {
                  "match" : {
                    "query" : "᭿ᬋ᭝ᬽᬹa ꥕ꥀ꥞꥟ꥌ꥘꥚ꥌ꥝ꤸ꥙ꥐa",
                    "max_gaps" : 4,
                    "mode" : "ORDERED",
                    "analyzer" : "simple"
                  }
                },
                {
                  "match" : {
                    "query" : "\uD84B\uDCAE\uD84D\uDD7C\uD841\uDFB3\uD847\uDD0D\uD865\uDE21\uD867\uDFDC\uD865\uDE22a ﹈︸﹇﹃﹉﹍︵︴︷﹂﹊﹊﹅﹋a",
                    "max_gaps" : 4,
                    "mode" : "UNORDERED",
                    "analyzer" : "simple"
                  }
                },
                {
                  "match" : {
                    "query" : "ᚭᛖᚠᚦᚥa \uD802\uDE64\uD802\uDE79\uD802\uDE68\uD802\uDE69\uD802\uDE64\uD802\uDE7A\uD802\uDE7Aaa ⺹⺤⺤⺗⺳⻝⻅⺾a 鳩肍稶鈆箤纠齩鹹偷儣捒a c7\u0014D)Z% \u001C\bg\u001B.cAa",
                    "max_gaps" : 1,
                    "mode" : "UNORDERED_NO_OVERLAP",
                    "analyzer" : "whitespace"
                  }
                }
              ]
            }
          },
          {
            "any_of" : {
              "intervals" : [
                {
                  "match" : {
                    "query" : "᪇ᨯᩆ᪭᪓ᩓ᪑ᩀᨽᩢᨧ᪢᪑ᨰᨷ᩼᪩᩻a ␚␳␏␩␇␻␞␳␐␈␿␎␩a ꩍꨘꨅ꩓a",
                    "max_gaps" : 1,
                    "mode" : "UNORDERED",
                    "analyzer" : "simple"
                  }
                },
                {
                  "any_of" : {
                    "intervals" : [
                      {
                        "match" : {
                          "query" : "Ꙅ꙼ꙀꚀꙏꙍꚅꚏꙥꚉꙁꚁa a ᝽ᝩ᝽ᝠᝧ᝵ᝫ᝺ᝦ᝿a ⴌⴃⴢⴕⴤⴢⴖⴣⴋ⴫ⴂⴈa ኚዏ፭ሁጇዦቑ፽ኤቓጌፎጕዌሬa",
                          "max_gaps" : 3,
                          "mode" : "UNORDERED",
                          "analyzer" : "keyword"
                        }
                      },
                      {
                        "match" : {
                          "query" : "ýٖŒáíÁò‹†úܟçáÌa \uD803\uDE73\uD803\uDE6A\uD803\uDE6C\uD803\uDE60a ݗݸݷݜݻݔݴݬݛݺݤݤݵݾa \uD800\uDCE9\uD800\uDCB8\uD800\uDCF2\uD800\uDC93\uD800\uDCF9\uD800\uDCC7a",
                          "max_gaps" : 3,
                          "mode" : "ORDERED",
                          "analyzer" : "whitespace"
                        }
                      },
                      {
                        "match" : {
                          "query" : "ፉቃርኖ዁፪ቭኌኒጱፄa ㇠㇝㇋㇂㇘㇘㇚㇎㇋㇨a ꚎꚉꙭꚍꙟꚈꙚꙉꚞa",
                          "max_gaps" : -1,
                          "mode" : "UNORDERED",
                          "analyzer" : "simple"
                        }
                      },
                      {
                        "match" : {
                          "query" : " ᚊᚒᚇᚌ᚜ᚃᚃᚈᚖa ﴵﰄﯶﲩ﴿ﲄﲇ﷛ﳸﲚﮃﰝ﷓ﯚﮡﲱﶝﱎ﷙a",
                          "max_gaps" : 1,
                          "mode" : "ORDERED",
                          "analyzer" : "keyword"
                        }
                      }
                    ]
                  }
                },
                {
                  "match" : {
                    "query" : "⸝⸱⹵a ᭑᭲ᬤ᭺᭹ᬱ᭟ᬷᬓ᭟᭝᭮a",
                    "max_gaps" : 1,
                    "mode" : "UNORDERED_NO_OVERLAP",
                    "analyzer" : "simple"
                  }
                },
                {
                  "match" : {
                    "query" : "a",
                    "max_gaps" : 1,
                    "mode" : "UNORDERED",
                    "analyzer" : "whitespace"
                  }
                },
                {
                  "match" : {
                    "query" : "ㄧㄟㄆㄜㄤㄎㄅㄅㄘ㄄ㄏㄐㄘㄋ㄁ㄩㄤa",
                    "max_gaps" : 3,
                    "mode" : "UNORDERED_NO_OVERLAP",
                    "analyzer" : "simple"
                  }
                }
              ]
            }
          },
          {
            "match" : {
              "query" : "ᇱᆵᄡᅔᄖᇎᆌᆶᅋᇽᄨᆿᅥa ᅭᆕᄲᇳᅆᄬᅈᄍᇖᅾᄊᇴᆔᇾᄡᇈa ꙳ꙬꙐꙣꙂꙺꙘꚍꚆꙟꙃꚄꚍꙁꚞꙨa るぉろほゃひぅもぷゟゟしべばごんa \uD87E\uDC2E\uD87E\uDD26\uD87E\uDCE5\uD87E\uDDD2\uD87E\uDDFB\uD87E\uDCD7aa",
              "max_gaps" : -1,
              "mode" : "UNORDERED",
              "analyzer" : "whitespace"
            }
          },
          {
            "all_of" : {
              "mode" : "ORDERED",
              "max_gaps" : 2,
              "intervals" : [
                {
                  "match" : {
                    "query" : "꒽꓋꒞꒑꓇꒭꒞꒹꒵a a ꡾ꡤ꡾꡴ꡖꡝꡫ꡼ꡣꡋ꡺꡿ꡖꡌ꡼ꡟꡕꡡꡫa ⦓⧯⦓a",
                    "max_gaps" : 1,
                    "mode" : "UNORDERED",
                    "analyzer" : "simple"
                  }
                },
                {
                  "all_of" : {
                    "mode" : "ORDERED",
                    "max_gaps" : 4,
                    "intervals" : [
                      {
                        "match" : {
                          "query" : "ㇴㇳㇷㇱㇷㇲㇴㇽㇲㇲㇼa \uD802\uDE5D\uD802\uDE5D\uD802\uDE4E\uD802\uDE23\uD802\uDE52\uD802\uDE11\uD802\uDE5B\uD802\uDE55\uD802\uDE57aa",
                          "max_gaps" : 1,
                          "mode" : "UNORDERED_NO_OVERLAP",
                          "analyzer" : "simple"
                        }
                      },
                      {
                        "match" : {
                          "query" : "令﨩契兩菉廉節憐﫯煮滛郎﫳祈黎a ᙋᗴᓭ᙮ᖽᒌᘒᓋᗣᙗᘺᙞᒩa ు౻ణ౜౸ఔే౗థేa",
                          "max_gaps" : 0,
                          "mode" : "UNORDERED_NO_OVERLAP",
                          "analyzer" : "whitespace"
                        }
                      }
                    ]
                  }
                },
                {
                  "match" : {
                    "query" : "ඎ෥෕ඨඹළඉජ෽අට෽ඐනa ᤟ᤁ᥌᥇ᤗᤁᤣᤝᤙᤡ᤺a",
                    "max_gaps" : 2,
                    "mode" : "UNORDERED",
                    "analyzer" : "keyword"
                  }
                },
                {
                  "all_of" : {
                    "mode" : "ORDERED",
                    "max_gaps" : 4,
                    "intervals" : [
                      {
                        "match" : {
                          "query" : "طڡأ٬ڹڼ٨۩a ⹶⸫⸌⹥⹋⸈a ︶﹍︲︰﹃︵︸﹉︿﹇﹆︼﹌︲︽︿a a ㄲㄷㅠㄴㄾㅷㅥㅡa",
                          "max_gaps" : 3,
                          "mode" : "UNORDERED_NO_OVERLAP",
                          "analyzer" : "simple"
                        }
                      },
                      {
                        "match" : {
                          "query" : "ⴅⴑⴜ⴪ⴇⴗⴠⴍⴌⴊⴥⴄⴚ⴮ⴔⴝa ᱴ᱒᱖᱐ᱟᱽᱫᱲᱼᱫᱬᱰ᱙ᱡᱲa ೠು್೫ೲ೹೔ಹರಕುರಌ್೒೦ಟ೪ೀa ᜟᜎ᜗ᜄᜓᜅ᜛᜔ᜃᜓᜆ᜙ᜏᜀᜌ᜔᜘ᜍ᜕a ပဌႛၰအၿ၁ၰa",
                          "max_gaps" : 2,
                          "mode" : "UNORDERED_NO_OVERLAP",
                          "analyzer" : "keyword"
                        }
                      },
                      {
                        "match" : {
                          "query" : "ಲಊ೜ಮಀa ⟵⟽⟳⟵⟺⟶⟻⟲⟱⟻a \uD834\uDF42\uD834\uDF04\uD834\uDF3Baa",
                          "max_gaps" : 4,
                          "mode" : "UNORDERED",
                          "analyzer" : "keyword"
                        }
                      }
                    ]
                  }
                }
              ]
            }
          }
        ]
      },
      "boost" : 0.33333334,
      "_name" : "YBYknk0"
    }
  }
}
ccook-ibm commented 1 year ago

A similar instance of the Too many disjunctions to expand exception, this time failing a different test in the same suite: https://github.com/opensearch-project/OpenSearch/pull/8823#issuecomment-1646218392


Build failed due to flaky test:

Tests with failures:
 - org.opensearch.index.query.IntervalQueryBuilderTests.testToQuery
REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.query.IntervalQueryBuilderTests.testToQuery" -Dtests.seed=A1E5FA5E6C37D034 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-ES -Dtests.timezone=Africa/Gaborone -Druntime.java=20

org.opensearch.index.query.IntervalQueryBuilderTests > testToQuery FAILED
    java.lang.IllegalArgumentException: Too many disjunctions to expand
        at __randomizedtesting.SeedInfo.seed([A1E5FA5E6C37D034:561EF8601DB415DE]:0)
        at org.apache.lucene.queries.intervals.Disjunctions.pullUp(Disjunctions.java:45)
        at org.apache.lucene.queries.intervals.UnorderedIntervalsSource.pullUpDisjunctions(UnorderedIntervalsSource.java:90)
        at org.apache.lucene.queries.intervals.FilteredIntervalsSource.maxGaps(FilteredIntervalsSource.java:33)
        at org.apache.lucene.queries.intervals.Intervals.maxgaps(Intervals.java:302)
        at org.opensearch.index.query.IntervalBuilder.combineSources(IntervalBuilder.java:173)
        at org.opensearch.index.query.IntervalsSourceProvider$Combine.getSource(IntervalsSourceProvider.java:444)
        at org.opensearch.index.query.IntervalQueryBuilder.doToQuery(IntervalQueryBuilder.java:164)
        at org.opensearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:117)
        at org.opensearch.test.AbstractQueryTestCase.testToQuery(AbstractQueryTestCase.java:448)
        at java.****/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.****/java.lang.reflect.Method.invoke(Method.java:578)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.****/java.lang.Thread.run(Thread.java:1623)
peternied commented 9 months ago

Reproduced on PR, gradle check log.