opensearch-project / opensearch-hadoop

Apache License 2.0
29 stars 22 forks source link

Fixing integration test starting #477

Open asalamon74 opened 3 weeks ago

asalamon74 commented 3 weeks ago

Description

When I want to run the integration tests with ./gradlew check I've got an error message:

> Could not resolve all files for configuration ':qa:kerberos:downloadHadoop#3.3.2'.
   > Could not resolve hadoop.common:hadoop-3.3.2:hadoop-3.3.2.
     Required by:
         project :qa:kerberos
      > Could not resolve hadoop.common:hadoop-3.3.2:hadoop-3.3.2.
         > Could not get resource 'https://repo.spring.io/plugins-release-local/hadoop/common/hadoop-3.3.2/hadoop-3.3.2/hadoop-3.3.2-hadoop-3.3.2.pom'.
            > Could not GET 'https://repo.spring.io/plugins-release-local/hadoop/common/hadoop-3.3.2/hadoop-3.3.2/hadoop-3.3.2-hadoop-3.3.2.pom'. Received status code 401 from server:
      > Could not resolve hadoop.common:hadoop-3.3.2:hadoop-3.3.2.
         > Could not get resource 'https://artifacts.opensearch.org/snapshots/hadoop/common/hadoop-3.3.2/hadoop-3.3.2/hadoop-3.3.2-hadoop-3.3.2.pom'.
            > Could not GET 'https://artifacts.opensearch.org/snapshots/hadoop/common/hadoop-3.3.2/hadoop-3.3.2/hadoop-3.3.2-hadoop-3.3.2.pom'. Received status code 403 from server: Forbidden
      > Could not resolve hadoop.common:hadoop-3.3.2:hadoop-3.3.2.
         > Could not get resource 'https://snapshots.opensearch.org/releases/core/opensearch/hadoop-3.3.2/hadoop-3.3.2-min-hadoop-3.3.2.xml'.
            > Could not GET 'https://snapshots.opensearch.org/releases/core/opensearch/hadoop-3.3.2/hadoop-3.3.2-min-hadoop-3.3.2.xml'.
               > snapshots.opensearch.org: nodename nor servname provided, or not known
      > Could not resolve hadoop.common:hadoop-3.3.2:hadoop-3.3.2.
         > Could not get resource 'https://artifacts.opensearch.org/releases/core/opensearch/hadoop-3.3.2/hadoop-3.3.2-min-hadoop-3.3.2.xml'.
            > Could not GET 'https://artifacts.opensearch.org/releases/core/opensearch/hadoop-3.3.2/hadoop-3.3.2-min-hadoop-3.3.2.xml'. Received status code 403 from server: Forbidden

and the test are not started.

Issues Resolved

The error message is a bit misleading, we are trying to download artifacts from https://apache.osuosl.org/ but the required versions are not available anymore. After updating the version numbers I've got new errors:

> Could not resolve opensearch-distribution-snapshot:opensearch:2.13.0-SNAPSHOT.
         > Could not get resource 'https://artifacts.opensearch.org/snapshots/core/opensearch/2.13.0-SNAPSHOT/opensearch-min-2.13.0-SNAPSHOT-linux-x64-latest.tar.gz'.
            > Could not HEAD 'https://artifacts.opensearch.org/snapshots/core/opensearch/2.13.0-SNAPSHOT/opensearch-min-2.13.0-SNAPSHOT-linux-x64-latest.tar.gz'.
               > The server may not support the client's requested TLS protocol versions: (TLSv1.2, TLSv1.3). You may need to configure the client to allow other protocols to be used. For more on this, please refer to https://docs.gradle.org/8.5/userguide/build_environment.html#sec:gradle_system_properties in the Gradle documentation.
                  > Received fatal alert: handshake_failure

I can fix this with a gradle.properties change.

The next error message is keytab releated

 What went wrong:
A problem was found with the configuration of task ':qa:kerberos:hadoopFixture#namenode.configure' (type 'DefaultTestClustersTask').
  - In plugin 'org.opensearch.gradle.JdkDownloadPlugin' type 'org.opensearch.gradle.testclusters.DefaultTestClustersTask' property 'clusters.integTest$0.nodes.$0.extraConfigFiles.opensearch.keytab$0.file' specifies file '/home/systest/opensearch-hadoop/qa/kerberos/build/fixtures/kdcFixture/opensearch.keytab' which doesn't exist.

We need to create keytabs at the beginning. I've also fixed this.

Finally I've got the following error message:

Caused by: org.opensearch.hadoop.rest.OpenSearchHadoopRemoteException: no handler found for uri [/_security/user/client] and method [POST]
        at org.opensearch.hadoop.rest.ErrorExtractor.extractError(ErrorExtractor.java:108)
        at org.opensearch.hadoop.rest.RestClient.checkResponse(RestClient.java:473)
        ... 7 more

I'm not too familiar with this API but I have the feeling that this is some ancient code from elasticsearch and opensearch have a different API.

Can someone with opensearch experience help here?

-Was it possible to run these tests ever? Some of my fixes indicates that it's not working for a long time. -Is it worth committing this half-solution? At least we can start the tests and some of them are working with this. -I think it would be better to fix this no handler for uri error. Can someone help me what is the correct API here?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.