[BUG] Intermittent test failures in IT

[saikaranam-amazon]

Describe the bug

• stop replication in restoring state with multiple shards

  REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.replication.integ.rest.StopReplicationIT.test stop without replication in progress" -Dtests.seed=C38276E892FB81C3 -Dtests.security.manager=true -Dtests.locale=en-CA -Dtests.timezone=Atlantic/Faroe -Druntime.java=14
  2021-09-23T09:22:45.2711484Z     org.junit.ComparisonFailure: expected:<[fals]e> but was:<[tru]e>
  2021-09-23T09:22:45.2713097Z         at jdk.internal.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)

• Start Replication force merge issue

  org.opensearch.replication.integ.rest.StartReplicationIT > test forcemerge on leader during replication bootstrap FAILED
  2021-09-23T16:02:07.3191334Z     org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:39143], URI [/_plugins/_replication/follower_index/_stop], status line [HTTP/1.1 503 Service Unavailable]
  2021-09-23T16:02:07.3195427Z     {"error":{"root_cause":[{"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (add-block) within 30s"}],"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (add-block) within 30s"},"status":503}
  2021-09-23T16:02:07.3198657Z         at org.opensearch.client.RestClient.convertResponse(RestClient.java:339)
  2021-09-23T16:02:07.3200412Z         at org.opensearch.client.RestClient.performRequest(RestClient.java:309)

[saikaranam-amazon]

• Local build succeeds for

  
  9801a7b33fa3:cross-cluster-replication karanas$ ./gradlew ':integTest' --tests "org.opensearch.replication.integ.rest.StartReplicationIT.test follower stats" 
  =======================================
  OpenSearch Build Hamster says Hello!
  Gradle Version        : 6.6.1
  OS Info               : Mac OS X 10.14.6 (x86_64)
  JDK Version           : 14 (OpenJDK)
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home
  Random Testing Seed   : 28D2600DB48D5B29
  In FIPS 140 mode      : false
  =======================================

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 2m 27s 7 actionable tasks: 1 executed, 6 up-to-date

CI is failing on this test

[krishna-ggk]

We still see test failures - https://github.com/opensearch-project/cross-cluster-replication/actions/workflows/build.yml?page=4

However it is not clear if those are due to this.

Will track this in https://github.com/opensearch-project/cross-cluster-replication/issues/369