[BUG] unable to build source using gradlew distZip

FelixFeliciant commented 1 year ago

What is the bug?

A clear and concise description of the bug. unable to build source using gradlew distZip

How can one reproduce the bug?

run gradlew distZip

What is the expected behavior?

To create a distributable zip, run gradlew distZip from the command line; once completed you will find the jar in build/libs

What is your host/environment?

OS Name: Microsoft Windows 10 Enterprise OS Version: 10.0.19045 N/A Build 19045

Do you have any additional context?

When trying to gradlew distZip I get an error:

An exception occurred applying plugin request [id: 'com.github.johnrengelman.shadow', version: '6.1.0']
> Failed to apply plugin class 'com.github.jengelman.gradle.plugins.shadow.ShadowBasePlugin'.
   > This version of Shadow supports Gradle 8.0+ only. Please upgrade.

this is due to gradle/wrapper/gradle-wrapper.properties distributionUrl=https\://services.gradle.org/distributions/gradle-7.6.1-bin.zip I have changed it to distributionUrl=https\://services.gradle.org/distributions/gradle-8.1-bin.zip and tried to run gradlew distZip again and this time it failed again Execution failed for task ':opensearch-hadoop-hive:javadoc'.

> Error while evaluating property 'javadocTool' of task ':opensearch-hadoop-hive:javadoc'.
   > Failed to calculate the value of task ':opensearch-hadoop-hive:javadoc' property 'javadocTool'.
      > The configured executable does not exist (C:\Program Files\Java\jdk1.8.0_202\bin\javadoc)

is it possible that you can produce a dependency in maven repo please?

Thank you.

harshavamsi commented 1 year ago

Hi, this is a bug with the the gradle shadow plugin and requires that we update gradle to 8.1.1. I have it working in this PR. Currently working on some other fixes but should get gradle builds to work soon.

harshavamsi commented 1 year ago

@FelixFeliciant Can you build from source with the latest changes? I upgraded gradle to 8.1.1 and fixed some issues around plugin management. Thanks!

FelixFeliciant commented 1 year ago

Hi @harshavamsi , just updated source code with latest changes and I still get the same error, when running gradlew distZip: full log: ` 10:01:33: Executing 'distZip'...

Configuration on demand is an incubating feature.

Task :buildSrc:compileJava UP-TO-DATE Task :buildSrc:compileGroovy UP-TO-DATE Task :buildSrc:writeVersionProperties UP-TO-DATE Task :buildSrc:processResources UP-TO-DATE Task :buildSrc:classes UP-TO-DATE Task :buildSrc:jar UP-TO-DATE

Configure project :

OpenSearch-Hadoop Build Hamster says Hello!

Min runtime: 1.8 Building version [1.0.0-SNAPSHOT] Testing against OpenSearch [3.0.0-SNAPSHOT] with Lucene [9.5.0-snapshot-a4ef70f] Using Gradle [8.1.1] Using Apache Hadoop on YARN [3.1.2]

OpenSearch Build Hamster says Hello! Gradle Version : 8.1.1 OS Info : Windows 10 10.0 (amd64) JDK Version : 17 (Azul Zulu JDK) JAVA_HOME : C:\Program Files\Zulu\zulu-17 Random Testing Seed : 8FC090CD23DD40E1 In FIPS 140 mode : false

Task :opensearch-spark:processResources NO-SOURCE Task :dist:processResources NO-SOURCE Task :opensearch-hadoop-hive:processResources NO-SOURCE Task :compileJava NO-SOURCE Task :thirdparty:compileJava NO-SOURCE Task :thirdparty:processResources NO-SOURCE Task :opensearch-spark:processSpark20scala210Resources NO-SOURCE Task :thirdparty:classes UP-TO-DATE Task :processResources NO-SOURCE Task :classes UP-TO-DATE Task :opensearch-spark:processSpark20scala211Resources NO-SOURCE Task :opensearch-hadoop-mr:generateGitHash Task :opensearch-spark:processSpark30scala212Resources NO-SOURCE Task :opensearch-spark:processSpark30scala213Resources NO-SOURCE Task :opensearch-spark-20:processSpark20scala210Resources Task :opensearch-spark-30:processSpark30scala213Resources Task :jar Task :javadoc NO-SOURCE Task :opensearch-spark-20:processItestSpark20scala210Resources Task :opensearch-hadoop-mr:processResources Task :javadocJar Task :opensearch-spark-20:processSpark20scala211Resources Task :opensearch-spark-30:processItestSpark30scala213Resources Task :sourcesJar Task :pack Task :opensearch-spark-20:processItestSpark20scala211Resources Task :opensearch-spark-30:processResources Task :opensearch-spark-30:processTestSpark30scala213Resources NO-SOURCE Task :opensearch-spark-20:processResources Task :opensearch-spark-20:processTestSpark20scala210Resources NO-SOURCE Task :opensearch-spark-20:processTestSpark20scala211Resources NO-SOURCE Task :thirdparty:shadowJar

Task :opensearch-hadoop-mr:compileJava C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\util\ecs\HostData.java:34: warning: [removal] AccessController in java.security has been deprecated and marked for removal import java.security.AccessController; ^ C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\security\JdkUserProvider.java:32: warning: [removal] AccessControlContext in java.security has been deprecated and marked for removal import java.security.AccessControlContext; ^ C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\security\JdkUserProvider.java:33: warning: [removal] AccessController in java.security has been deprecated and marked for removal import java.security.AccessController; ^ C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\util\ecs\HostData.java:93: warning: [removal] AccessController in java.security has been deprecated and marked for removal return AccessController.doPrivileged(new GetPropAction(prop)); ^ C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\rest\RestClient.java:374: warning: [unchecked] unchecked cast Map<String, Object> fieldInfo = (Map<String, Object>) foundFields.get("_source"); ^ required: Map<String,Object> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\rest\RestClient.java:581: warning: [unchecked] unchecked cast Map<String, Object> totalObject = (Map<String, Object>) total; ^ required: Map<String,Object> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\rest\RestClient.java:616: warning: [unchecked] unchecked cast Map<String, Object> aliases = (Map<String, Object>) get(query, null); ^ required: Map<String,Object> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\serialization\dto\NodeInfo.java:58: warning: [unchecked] unchecked cast List roles = (List) map.get("roles"); ^ required: List found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\serialization\dto\NodeInfo.java:62: warning: [unchecked] unchecked cast Map<String, Object> httpMap = (Map<String, Object>) map.get("http"); ^ required: Map<String,Object> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\mr\MultiOutputFormat.java:62: warning: [unchecked] unchecked call to write(K,V) as a member of the raw type RecordWriter writer.write(key, value); ^ where K,V are type-variables: K extends Object declared in class RecordWriter V extends Object declared in class RecordWriter C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\mr\MultiOutputFormat.java:93: warning: [unchecked] unchecked call to write(K,V) as a member of the raw type RecordWriter writer.write(key, value); ^ where K,V are type-variables: K extends Object declared in interface RecordWriter V extends Object declared in interface RecordWriter C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\mr\MultiOutputFormat.java:267: warning: [unchecked] Possible heap pollution from parameterized vararg type Class<? extends OutputFormat> public static void addOutputFormat(Configuration cfg, Class<? extends OutputFormat>... formats) { ^ C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\serialization\dto\IndicesAliases.java:86: warning: [unchecked] unchecked cast final Map<String, Object> metadata = (Map<String, Object>) index.getValue(); ^ required: Map<String,Object> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\serialization\dto\IndicesAliases.java:87: warning: [unchecked] unchecked cast final Map<String, Map<String, Object> > aliases = (Map<String, Map<String, Object> >) metadata.get("aliases"); ^ required: Map<String,Map<String,Object>> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\serialization\dto\IndicesAliases.java:105: warning: [unchecked] unchecked cast filter = (Map<String, Object>) aliasMetadata.get("filter"); ^ required: Map<String,Object> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\rest\request\GetAliasesRequestBuilder.java:78: warning: [unchecked] unchecked cast return new Response((Map<String, Object>) client.get(path.toString(), null)); ^ required: Map<String,Object> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\rest\query\RawQueryBuilder.java:68: warning: [unchecked] unchecked cast this((Map<String, Object>) MAPPER.readValue(value, HashMap.class), isQuery); ^ required: Map<String,Object> found: HashMap C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\security\JdkUserProvider.java:49: warning: [removal] AccessControlContext in java.security has been deprecated and marked for removal AccessControlContext acc = AccessController.getContext(); ^ C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\security\JdkUserProvider.java:49: warning: [removal] AccessController in java.security has been deprecated and marked for removal AccessControlContext acc = AccessController.getContext(); ^ C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\security\JdkUserProvider.java:50: warning: [removal] getSubject(AccessControlContext) in Subject has been deprecated and marked for removal Subject subject = Subject.getSubject(acc); ^ C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\serialization\bulk\AbstractBulkFactory.java:162: warning: [unchecked] unchecked call to write(T,Generator) as a member of the raw type ValueWriter ValueWriter.Result writeResult = valueWriter.write(value, generator); ^ where T is a type-variable: T extends Object declared in interface ValueWriter C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\serialization\dto\mapping\FieldParser.java:83: warning: [unchecked] unchecked cast Map<String, Object> mappingsObject = (Map<String, Object>) indexToMappings.getValue(); ^ required: Map<String,Object> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\serialization\dto\mapping\FieldParser.java:89: warning: [unchecked] unchecked cast Map<String, Object> mappingEntries = (Map<String, Object>) mappingsObject.get("mappings"); ^ required: Map<String,Object> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\serialization\dto\mapping\FieldParser.java:131: warning: [unchecked] unchecked cast Map<String, Object> content = (Map<String, Object>) value; ^ required: Map<String,Object> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\serialization\dto\mapping\FieldParser.java:180: warning: [unchecked] unchecked cast Map<String,Object> fieldValueAsMap = ((Map<String, Object>)fieldValue); ^ required: Map<String,Object> found: Object C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\mr\src\main\java\org\opensearch\hadoop\serialization\dto\mapping\MappingUtils.java:207: warning: [unchecked] unchecked cast Map<String, Object> geoShape = (Map<String, Object>) parsedContent; ^ required: Map<String,Object> found: Object Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. 26 warnings

Task :opensearch-hadoop-mr:classes Task :opensearch-spark:compileSpark20scala210Java NO-SOURCE

Task :opensearch-hadoop-hive:compileJava

Task :opensearch-hadoop-hive:classes Task :opensearch-hadoop-hive:javadoc FAILED C:\Users\ffeliciant\source\repos\ffeliciant\opensearch-project\opensearch-hadoop\hive\src\main\java\org\opensearch\hadoop\hive\OpenSearchHiveInputFormat.java:136: warning: [unchecked] getRecordReader(InputSplit,JobConf,Reporter) in OpenSearchHiveInputFormat implements getRecordReader(InputSplit,JobConf,Reporter) in InputFormat public AbstractWritableOpenSearchInputRecordReader getRecordReader(InputSplit split, JobConf job, Reporter reporter) { ^ return type requires unchecked conversion from AbstractWritableOpenSearchInputRecordReader to RecordReader<Text,Writable> where K,V are type-variables: K extends Object declared in interface InputFormat V extends Object declared in interface InputFormat Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. 1 warning Task :opensearch-hadoop-mr:jar Task :opensearch-spark:compileSpark20scala210Scala 24 actionable tasks: 19 executed, 5 up-to-date

FAILURE: Build failed with an exception.

What went wrong: Execution failed for task ':opensearch-hadoop-hive:javadoc'.

Error while evaluating property 'javadocTool' of task ':opensearch-hadoop-hive:javadoc'. Failed to calculate the value of task ':opensearch-hadoop-hive:javadoc' property 'javadocTool'. The configured executable does not exist (C:\Program Files\Java\jdk1.8.0_202\bin\javadoc)
Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
Get more help at https://help.gradle.org

BUILD FAILED in 28s 10:02:02: Execution finished 'distZip'.

`

FelixFeliciant commented 1 year ago

Hi, I was able to build the project after adding these lines to settings.gradle:

startParameter.excludedTaskNames.add(':opensearch-hadoop-mr:javadoc')
startParameter.excludedTaskNames.add(':opensearch-hadoop-hive:javadoc')
startParameter.excludedTaskNames.add(':opensearch-spark-20:javadoc')
startParameter.excludedTaskNames.add(':dist:javadoc')
startParameter.excludedTaskNames.add(':opensearch-spark-30:javadoc')
startParameter.excludedTaskNames.add(':opensearch-spark-20:spark20scala210Javadoc')
startParameter.excludedTaskNames.add(':opensearch-spark-30:spark30scala213Javadoc')
startParameter.excludedTaskNames.add(':opensearch-spark-20:spark20scala211Javadoc')

opensearch-project / opensearch-hadoop