search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.09k stars 24.84k forks source link

[CI] ActiveDirectorySessionFactoryTests classMethod failing #105401

Open edsavage opened 9 months ago

edsavage commented 9 months ago

Seems docker related

Build scan: https://gradle-enterprise.elastic.co/s/gkulbnzrdipfg/tests/:x-pack:qa:third-party:active-directory:test/org.elasticsearch.xpack.security.authc.ldap.ActiveDirectorySessionFactoryTests

Reproduction line:

null

Applicable branches: main

Reproduces locally?: Didn't try

Failure history: Failure dashboard for org.elasticsearch.xpack.security.authc.ldap.ActiveDirectorySessionFactoryTests#classMethod&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('classMethod'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.xpack.security.authc.ldap.ActiveDirectorySessionFactoryTests'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure excerpt:

org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=<resolving>, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@27332164)

  at __randomizedtesting.SeedInfo.seed([7FA184586C7298D8]:0)
  at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1367)
  at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:674)
  at org.elasticsearch.test.fixtures.testcontainers.DockerEnvironmentAwareTestContainer.start(DockerEnvironmentAwareTestContainer.java:67)
  at org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1085)
  at org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:28)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)

  Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not build image: failed to get digest sha256:cc95a733c5d8e97e125081874caaaf90cc1e1bb5d68b2af165a989cc7673b7aa: open /var/lib/docker/image/overlay2/imagedb/content/sha256/cc95a733c5d8e97e125081874caaaf90cc1e1bb5d68b2af165a989cc7673b7aa: no such file or directory

    at com.github.dockerjava.api.command.BuildImageResultCallback.getImageId(BuildImageResultCallback.java:78)
    at com.github.dockerjava.api.command.BuildImageResultCallback.awaitImageId(BuildImageResultCallback.java:50)
    at org.testcontainers.images.builder.ImageFromDockerfile.resolve(ImageFromDockerfile.java:159)
    at org.testcontainers.images.builder.ImageFromDockerfile.resolve(ImageFromDockerfile.java:40)
    at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
    at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
    at org.testcontainers.shaded.com.google.common.util.concurrent.Futures$1.get(Futures.java:536)
    at org.testcontainers.images.RemoteDockerImage.getImageName(RemoteDockerImage.java:130)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:67)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:28)
    at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
    at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1365)
    at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:674)
    at org.elasticsearch.test.fixtures.testcontainers.DockerEnvironmentAwareTestContainer.start(DockerEnvironmentAwareTestContainer.java:67)
    at org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1085)
    at org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:28)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1583)
elasticsearchmachine commented 9 months ago

Pinging @elastic/es-search (Team:Search)

benwtrent commented 9 months ago

Not sure why this was tagged as :search/search

The failure was due to a docker image failing to download and if it wasn't due to that, it the test is located within the security plugin.

Tagging delivery & security.

elasticsearchmachine commented 9 months ago

Pinging @elastic/es-security (Team:Security)

elasticsearchmachine commented 9 months ago

Pinging @elastic/es-delivery (Team:Delivery)

mark-vieira commented 9 months ago

So far only a single occurrence. We'll keep an eye on this.

benwtrent commented 8 months ago

eerily similar failure here: https://gradle-enterprise.elastic.co/s/zlqmdntkflpbs

active-directory test failure due to docker failure.

slobodanadamovic commented 6 months ago

There have been only 2 failures in the past 2 months and in the past 1 month 0 failures. Hence I marked this issue as low-risk, since this feels like a temporary issue.

mark-vieira commented 6 months ago

Closing since there have been no failures in the last month.

slobodanadamovic commented 6 months ago

Just found two recent failures with the same issue but for different tests:

slobodanadamovic commented 3 weeks ago

We still see occasional failures for GroupMappingIT and ActiveDirectoryRunAsIT integration test:

This still looks like a transient error, but it might as well be a race condition!? 

GroupMappingIT > classMethod FAILED
    org.testcontainers.containers.ContainerLaunchException: Container startup failed for image es-smb-fixture:latest
        at __randomizedtesting.SeedInfo.seed([ABF161D102ABA71E]:0)
        at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:362)
        at app//org.testcontainers.containers.GenericContainer.start(GenericContainer.java:333)
        at app//org.elasticsearch.test.fixtures.testcontainers.DockerEnvironmentAwareTestContainer.start(DockerEnvironmentAwareTestContainer.java:68)
        at app//org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1085)
        at app//org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:28)
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)

        Caused by:
        org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=<resolving>, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@1c350e95)
            at app//org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1367)
            at app//org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:674)
            at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:344)
            ... 21 more

            Caused by:
            com.github.dockerjava.api.exception.DockerClientException: Could not build image: failed to get digest sha256:a543b761e853ed2141667038347abfe5b0683c620183d5ed320595cc83b254ed: open /var/lib/docker/image/overlay2/imagedb/content/sha256/a543b761e853ed2141667038347abfe5b0683c620183d5ed320595cc83b254ed: no such file or directory
                at app//com.github.dockerjava.api.command.BuildImageResultCallback.getImageId(BuildImageResultCallback.java:78)
                at app//com.github.dockerjava.api.command.BuildImageResultCallback.awaitImageId(BuildImageResultCallback.java:50)
                at app//org.testcontainers.images.builder.ImageFromDockerfile.resolve(ImageFromDockerfile.java:159)
                at app//org.testcontainers.images.builder.ImageFromDockerfile.resolve(ImageFromDockerfile.java:40)
                at app//org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
                at app//org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
                at app//org.testcontainers.shaded.com.google.common.util.concurrent.Futures$1.get(Futures.java:536)
                at app//org.testcontainers.images.RemoteDockerImage.getImageName(RemoteDockerImage.java:130)
                at app//org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:67)
                at app//org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:28)
                at app//org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
                at app//org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
                at app//org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1365)
                ... 23 more