[AUTOCUT] Integration Test failed for cross-cluster-replication: 2.13.0

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/7981/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9616/linux/arm64/deb/test-results/7981/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8051/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9634/linux/x64/rpm/test-results/8051/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8054/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9634/windows/x64/zip/test-results/8054/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8055/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9647/windows/x64/zip/test-results/8055/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8060/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9647/linux/arm64/deb/test-results/8060/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8058/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9647/linux/x64/rpm/test-results/8058/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8059/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9647/linux/arm64/rpm/test-results/8059/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8061/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9647/linux/x64/deb/test-results/8061/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8056/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9647/linux/x64/tar/test-results/8056/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8067/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9651/linux/x64/deb/test-results/8067/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8064/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9651/linux/arm64/rpm/test-results/8064/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8066/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9651/linux/arm64/deb/test-results/8066/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8065/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9651/linux/x64/rpm/test-results/8065/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8068/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9651/windows/x64/zip/test-results/8068/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8074/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9652/linux/x64/deb/test-results/8074/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8072/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9652/linux/arm64/rpm/test-results/8072/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8075/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9652/linux/arm64/deb/test-results/8075/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8070/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9652/windows/x64/zip/test-results/8070/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8073/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9652/linux/x64/rpm/test-results/8073/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8082/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9658/linux/x64/deb/test-results/8082/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8084/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9658/linux/arm64/deb/test-results/8084/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8083/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9658/linux/x64/rpm/test-results/8083/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8085/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9658/linux/arm64/rpm/test-results/8085/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8090/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9659/linux/x64/deb/test-results/8090/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8091/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9659/linux/arm64/deb/test-results/8091/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8089/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9659/linux/arm64/rpm/test-results/8089/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8088/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9659/linux/x64/rpm/test-results/8088/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8092/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9659/windows/x64/zip/test-results/8092/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8081/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9658/windows/x64/zip/test-results/8081/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[nisgoel-amazon]

In deb Distribution, I have investigated by running the remoteIntegTest through opensearch-build package. There is some issue while creating multiple clusters. Clusters are not forming and that is why we can see connection refused error in test failures. I have created 2 cluster using docker image and then try to run the integration test, All the test are working fine

Docker File

``` # docker-compose.yml version: '3' services: opensearch-leader: image: opensearchstaging/opensearch:2.13.0.9652 container_name: opensearch-leader environment: - cluster.name=opensearch-leader - node.name=opensearch-leader1 - discovery.seed_hosts=opensearch-leader1 - cluster.initial_master_nodes=opensearch-leader1 - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123@!" - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems hard: 65536 volumes: - opensearch-data1:/usr/share/opensearch/data ports: - 9200:9200 - 9600:9600 # required for Performance Analyzer networks: - opensearch-net opensearch-follower: image: opensearchstaging/opensearch:2.13.0.9652 container_name: opensearch-follower environment: - cluster.name=opensearch-follower - node.name=opensearch-follower1 - discovery.seed_hosts=opensearch-follower1 - cluster.initial_master_nodes=opensearch-follower1 - bootstrap.memory_lock=true - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123@!" - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 volumes: - opensearch-data2:/usr/share/opensearch/data ports: - 9201:9200 - 9601:9600 # required for Performance Analyzer networks: - opensearch-net volumes: opensearch-data1: opensearch-data2: networks: opensearch-net: ```

Command to run remote Integration Test

./gradlew integTestRemote  -Dleader.http_host=localhost:9200 -Dfollower.http_host=localhost:9201 -Dfollower.transport_host=172.18.0.3:9300  -Dleader.transport_host=172.18.0.2:9300  -Dsecurity_enabled=true -Duser=admin -Dpassword=myStrongPassword123! -PnumNodes=1

[nisgoel-amazon]

@monusingh-1 Can you take a look why clusters are not forming correctly from opensearch-build package.

[khushbr]

Issue tracked in https://github.com/opensearch-project/cross-cluster-replication/issues/1361

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8098/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9662/linux/x64/deb/test-results/8098/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8096/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9662/linux/x64/rpm/test-results/8096/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8099/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9662/linux/arm64/deb/test-results/8099/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: rpm
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8097/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9662/linux/arm64/rpm/test-results/8097/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8100/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9662/windows/x64/zip/test-results/8100/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[opensearch-ci-bot]

The integration test failed at distribution level for component cross-cluster-replication
Version: 2.13.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8101/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.13.0/9662/windows/x64/zip/test-results/8101/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

[ankitkala]

Related to https://github.com/opensearch-project/opensearch-build/issues/4610

[dblock]

Old autocut, closing.