h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
http://h2o.ai
Apache License 2.0
6.93k stars 2k forks source link

In some environments build does not detect python binary properly #11138

Open exalate-issue-sync[bot] opened 1 year ago

exalate-issue-sync[bot] commented 1 year ago

Nidhi reported a problem when build was using python2 instead of python3 which was defined already in environment.

The problem is that Gradle/JVM Exec/ProcessBuilder does not take into account PATH property.

exalate-issue-sync[bot] commented 1 year ago

Nidhi Mehta commented: everdark reports build fails if python2 request package is missing ,even when using python3 - https://github.com/h2oai/deepwater/issues/36#issuecomment-293323574

Reproed- looks like rest bindings are failing and depends on python2's requests package Here is the stack trace-

{code:java} :h2o-assemblies:main:copyJar :h2o-bindings:runGenerateRESTAPIBindingsSrc Traceback (most recent call last): File "gen_all.py", line 5, in import gen_csharp File "/Users/nidhimehta/h2o-3/h2o-bindings/bin/gen_csharp.py", line 4, in import bindings as bi File "/Users/nidhimehta/h2o-3/h2o-bindings/bin/bindings.py", line 25, in import requests ImportError: No module named requests :h2o-bindings:runGenerateRESTAPIBindingsSrc FAILED

FAILURE: Build failed with an exception.

exalate-issue-sync[bot] commented 1 year ago

Michal Malohlava commented: The problem is how we are launching python:

"python".execute() will just fork a new python process however fork does not honor PATH of parent process (PATH is variable expose by shell, not by glibc providing fork/exec functions).

In this context, our command line launchers are wrong.

exalate-issue-sync[bot] commented 1 year ago

Michal Malohlava commented: Cannot reproduce on any machine except Nidhi's. FOr now closing and waiting for other reports.

exalate-issue-sync[bot] commented 1 year ago

Pavel Pscheidl commented: I've experienced the very same problem after a clean installation of Ubuntu 16 LTS. I'm building with

{code:java} ./gradlew build -x test --debug {code}

Tried with:

Python local installations

There are two files included. One default gradle output, other with --debug option enabled. (./gradlew build -x test --debug).

[^simple.log] [^debug.log]

Here is the log with --stacktrace option {code:java} pavel@pavel-G750JM:~/Projects/h2o-3$ ./gradlew build -x test --stacktrace WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.gradle.internal.reflect.JavaMethod (file:/home/pavel/.gradle/wrapper/dists/gradle-3.3-all/55gk2rcmfc6p2dg9u9ohc3hw9/gradle-3.3/lib/gradle-base-services-3.3.jar) to method java.lang.ClassLoader.getPackages() WARNING: Please consider reporting this to the maintainers of org.gradle.internal.reflect.JavaMethod WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release :buildSrc:compileJava UP-TO-DATE :buildSrc:compileGroovy UP-TO-DATE :buildSrc:processResources UP-TO-DATE :buildSrc:classes UP-TO-DATE :buildSrc:jar UP-TO-DATE :buildSrc:assemble UP-TO-DATE :buildSrc:compileTestJava UP-TO-DATE :buildSrc:compileTestGroovy UP-TO-DATE :buildSrc:processTestResources UP-TO-DATE :buildSrc:testClasses UP-TO-DATE :buildSrc:test UP-TO-DATE :buildSrc:check UP-TO-DATE :buildSrc:build UP-TO-DATE The project project ':h2o-persist-s3' needs CI for running tests! You can pass -PdoCI=true to force CI behaviour. The Task.leftShift(Closure) method has been deprecated and is scheduled to be removed in Gradle 5.0. Please use Task.doLast(Action) instead. at org.gradle.api.internal.AbstractTask.leftShift(AbstractTask.java:513) at org.gradle.api.Task$leftShift$1.call(Unknown Source) at build_8kuxx5rcw56xzie7or4da6vfw.run(/home/pavel/Projects/h2o-3/h2o-py/build.gradle:20) at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:90) at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl$2.run(DefaultScriptPluginFactory.java:176) at org.gradle.configuration.ProjectScriptTarget.addConfiguration(ProjectScriptTarget.java:77) at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:181) at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:39) at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26) at org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34) at org.gradle.configuration.project.LifecycleProjectEvaluator.doConfigure(LifecycleProjectEvaluator.java:70) at org.gradle.configuration.project.LifecycleProjectEvaluator.access$000(LifecycleProjectEvaluator.java:33) at org.gradle.configuration.project.LifecycleProjectEvaluator$1.execute(LifecycleProjectEvaluator.java:53) at org.gradle.configuration.project.LifecycleProjectEvaluator$1.execute(LifecycleProjectEvaluator.java:50) at org.gradle.internal.Transformers$4.transform(Transformers.java:169) at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:106) at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:61) at org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:50) at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:628) at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:129) at org.gradle.execution.TaskPathProjectEvaluator.configure(TaskPathProjectEvaluator.java:35) at org.gradle.execution.TaskPathProjectEvaluator.configureHierarchy(TaskPathProjectEvaluator.java:62) at org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:38) at org.gradle.initialization.DefaultGradleLauncher$1.execute(DefaultGradleLauncher.java:161) at org.gradle.initialization.DefaultGradleLauncher$1.execute(DefaultGradleLauncher.java:158) at org.gradle.internal.Transformers$4.transform(Transformers.java:169) at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:106) at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:56) at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:158) at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:119) at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:102) at org.gradle.launcher.exec.GradleBuildController.run(GradleBuildController.java:71) at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28) at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35) at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:41) at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:26) at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:75) at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:49) at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:44) at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:29) at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:67) at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:47) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:26) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:34) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:74) at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:72) at org.gradle.util.Swapper.swap(Swapper.java:38) at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:72) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:60) at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:72) at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:50) at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:297) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54) at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.base/java.lang.Thread.run(Thread.java:844)

== :h2o-assemblies:genmodel: Using optional genmodel component: xgboost, version 3.15.0 == :h2o-assemblies:main: Using optional component: xgboost, version 3.15.0 :compileJava UP-TO-DATE :processResources UP-TO-DATE :classes UP-TO-DATE :jar UP-TO-DATE :assemble UP-TO-DATE :licenseMain UP-TO-DATE :licenseTest UP-TO-DATE :license UP-TO-DATE :check :build :h2o-core:generateBuildVersionJava NOTE: emitBuildVersionJava found a match, nothing to do:h2o-genmodel:compileJava UP-TO-DATE :h2o-genmodel:processResources UP-TO-DATE :h2o-genmodel:classes UP-TO-DATE :h2o-genmodel:jar UP-TO-DATE :h2o-jaas-pam:compileJava UP-TO-DATE :h2o-jaas-pam:processResources UP-TO-DATE :h2o-jaas-pam:classes UP-TO-DATE :h2o-jaas-pam:jar UP-TO-DATE :h2o-core:compileJava UP-TO-DATE :h2o-core:processResources UP-TO-DATE :h2o-core:classes UP-TO-DATE :h2o-core:jar UP-TO-DATE :h2o-algos:compileJava UP-TO-DATE :h2o-algos:processResources UP-TO-DATE :h2o-algos:classes UP-TO-DATE :h2o-algos:jar UP-TO-DATE :h2o-algos:assemble UP-TO-DATE :h2o-algos:check :h2o-algos:build :h2o-automl:compileJava UP-TO-DATE :h2o-automl:processResources UP-TO-DATE :h2o-automl:classes UP-TO-DATE :h2o-automl:jar UP-TO-DATE :h2o-persist-hdfs:compileJava UP-TO-DATE :h2o-persist-hdfs:processResources UP-TO-DATE :h2o-persist-hdfs:classes UP-TO-DATE :h2o-persist-hdfs:jar UP-TO-DATE :h2o-ext-krbstandalone:compileJava UP-TO-DATE :h2o-ext-krbstandalone:processResources UP-TO-DATE :h2o-ext-krbstandalone:classes UP-TO-DATE :h2o-ext-krbstandalone:jar UP-TO-DATE :h2o-app:compileJava UP-TO-DATE :h2o-app:processResources UP-TO-DATE :h2o-app:classes UP-TO-DATE :h2o-app:jar UP-TO-DATE :h2o-app:assemble UP-TO-DATE :h2o-app:check :h2o-app:build :h2o-automl:assemble UP-TO-DATE :h2o-automl:check :h2o-automl:build :h2o-avro-parser:compileJava UP-TO-DATE :h2o-avro-parser:processResources UP-TO-DATE :h2o-avro-parser:classes UP-TO-DATE :h2o-avro-parser:jar UP-TO-DATE :h2o-avro-parser:assemble UP-TO-DATE :h2o-avro-parser:check :h2o-avro-parser:build :h2o-genmodel-ext-xgboost:compileJava UP-TO-DATE :h2o-genmodel-ext-xgboost:processResources UP-TO-DATE :h2o-genmodel-ext-xgboost:classes UP-TO-DATE :h2o-genmodel-ext-xgboost:shadowJar UP-TO-DATE :h2o-genmodel-ext-xgboost:jar SKIPPED :h2o-ext-xgboost:compileJava UP-TO-DATE :h2o-ext-xgboost:processResources UP-TO-DATE :h2o-ext-xgboost:classes UP-TO-DATE :h2o-ext-xgboost:jar UP-TO-DATE :h2o-parquet-parser:compileJava UP-TO-DATE :h2o-parquet-parser:processResources UP-TO-DATE :h2o-parquet-parser:classes UP-TO-DATE :h2o-parquet-parser:jar UP-TO-DATE :h2o-persist-s3:compileJava UP-TO-DATE :h2o-persist-s3:processResources UP-TO-DATE :h2o-persist-s3:classes UP-TO-DATE :h2o-persist-s3:jar UP-TO-DATE :h2o-web:compileJava UP-TO-DATE :h2o-web:checkClientPrerequisites :h2o-web:installNpmPackages :h2o-web:installNpmPackages took 1.087 secs :h2o-web:compileHelpFiles :h2o-web:copyDeepWaterJarToWebRoot UP-TO-DATE :h2o-web:copyExampleFlows UP-TO-DATE :h2o-web:installBowerPackages (node:2435) DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead. :h2o-web:copyFlowToWebRoot UP-TO-DATE :h2o-assemblies:genmodel:compileJava UP-TO-DATE :h2o-assemblies:genmodel:processResources UP-TO-DATE :h2o-assemblies:genmodel:classes UP-TO-DATE :h2o-assemblies:genmodel:downloadLicenses UP-TO-DATE :h2o-assemblies:genmodel:shadowJar UP-TO-DATE :h2o-web:copyGenModelJarToWebRoot UP-TO-DATE :h2o-web:copyHelpImages UP-TO-DATE :h2o-web:processResources UP-TO-DATE :h2o-web:classes UP-TO-DATE :h2o-web:jar UP-TO-DATE :h2o-assemblies:main:compileJava UP-TO-DATE :h2o-assemblies:main:processResources UP-TO-DATE :h2o-assemblies:main:classes UP-TO-DATE :h2o-assemblies:main:jar :h2o-assemblies:main:downloadLicenses UP-TO-DATE :h2o-assemblies:main:shadowJar :h2o-assemblies:main:shadowJar took 17.741 secs :h2o-assemblies:main:copyJar :h2o-bindings:runGenerateRESTAPIBindingsSrc Starting H2O cloud...

ERROR: Too many retries starting cloud 0. Check the output log ../build/logs/java_0_0.out.txt.

:h2o-bindings:runGenerateRESTAPIBindingsSrc FAILED :h2o-bindings:runGenerateRESTAPIBindingsSrc took 30.123 secs

FAILURE: Build failed with an exception.

BUILD FAILED

Total time: 51.911 secs

Task timings: 30,123 secs :h2o-bindings:runGenerateRESTAPIBindingsSrc 17,741 secs :h2o-assemblies:main:shadowJar 1,087 secs :h2o-web:installNpmPackages 0,621 secs :h2o-assemblies:main:copyJar 0,364 secs :h2o-web:installBowerPackages 0,113 secs :h2o-web:compileHelpFiles 0,296 secs All others

{code}

exalate-issue-sync[bot] commented 1 year ago

Pavel Pscheidl commented: Python 3.6.3 works properly ! Removing other versions of python from PATH & installing Python 3.6.3 resolved the issue.

h2o-ops commented 1 year ago

JIRA Issue Migration Info

Jira Issue: PUBDEV-4249 Assignee: Michal Malohlava Reporter: Michal Malohlava State: Reopened Fix Version: N/A Attachments: Available (Count: 2) Development PRs: Available

Linked PRs from JIRA

https://github.com/h2oai/h2o-3/pull/1020

Attachments From Jira

Attachment Name: debug.log Attached By: Pavel Pscheidl File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-4249/debug.log

Attachment Name: simple.log Attached By: Pavel Pscheidl File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-4249/simple.log