Error while using mmlspark Lightgbm : Py4JJavaError('An error occurred while calling o527.fit.\n', JavaObject id=o680)

mybinaryworld commented 2 years ago

I am trying to use hyperopt to tune the hyperparameters for the LightGBM model and sometime I get this error Py4JJavaError('An error occurred while calling o527.fit.\n', JavaObject id=o680). In my previous run, out of 50 hyperopt trials, 34 failed due to above error while 14 trials were successful. Failed trials = 3, 4, 17-50. I am using lightbm from mmlspark and running an airflow DAG to execute it. I can't seem to understand why I am getting this error an dhow can I avoid this. The error block looks like this :

[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_160 !
[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_165 !
[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_163 !
[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_3 !
[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_34 !
[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_99 !
[2021-12-08 02:20:19,951] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:20:19 WARN scheduler.TaskSetManager: Lost task 30.1 in stage 59.0 (TID 2749, 100.66.217.123, executor 5): java.net.ConnectException: Connection refused (Connection refused)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.PlainSocketImpl.socketConnect(Native Method)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:589)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:538)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:434)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:211)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:456)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$16.apply(TrainUtils.scala:549)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$16.apply(TrainUtils.scala:544)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.core.env.StreamUtilities$.using(StreamUtilities.scala:29)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.TrainUtils$.trainLightGBM(TrainUtils.scala:543)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$7.apply(LightGBMBase.scala:225)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$7.apply(LightGBMBase.scala:225)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:188)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:185)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.Task.run(Task.scala:123)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
[2021-12-08 02:20:19,955] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
[2021-12-08 02:20:19,955] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-08 02:20:19,955] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-08 02:20:19,955] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.lang.Thread.run(Thread.java:748)
[2021-12-08 02:20:19,955] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 
[2021-12-08 02:20:31,380] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:20:31 WARN scheduler.TaskSetManager: Lost task 6.1 in stage 59.0 (TID 2748, 100.66.217.123, executor 5): java.net.ConnectException: Connection refused (Connection refused)
[2021-12-08 02:20:31,380] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.PlainSocketImpl.socketConnect(Native Method)
[2021-12-08 02:20:31,380] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[2021-12-08 02:20:31,380] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[2021-12-08 02:20:31,380] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:589)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:538)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:434)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:211)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:456)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$16.apply(TrainUtils.scala:549)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$16.apply(TrainUtils.scala:544)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.core.env.StreamUtilities$.using(StreamUtilities.scala:29)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.TrainUtils$.trainLightGBM(TrainUtils.scala:543)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$7.apply(LightGBMBase.scala:225)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$7.apply(LightGBMBase.scala:225)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:188)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:185)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.Task.run(Task.scala:123)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-08 02:20:31,384] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold     at java.lang.Thread.run(Thread.java:748)
[2021-12-08 02:20:31,384] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 
[2021-12-08 02:20:40,794] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:20:40 ERROR scheduler.TaskSetManager: Task 6 in stage 59.0 failed 8 times; aborting job
[2021-12-08 02:20:40,812] {logging_mixin.py:103} {PID:118 TID:140374674740992} INFO - 

[2021-12-08 02:20:40,812] {logging_mixin.py:103} {PID:118 TID:140374674740992} INFO - 
[2021-12-08 02:20:40,812] {logging_mixin.py:95} {PID:118 TID:140374674740992} INFO - [2021-12-08 02:20:40,812] {agg_modeling.py:562} {PID:118 TID:140374674740992} INFO - Exception while executing train_gbm (agg_modeling.py) : Py4JJavaError('An error occurred while calling o527.fit.\n', JavaObject id=o680)
[2021-12-08 02:20:40,812] {logging_mixin.py:95} {PID:118 TID:140374674740992} INFO - 
[2021-12-08 02:20:40,813] {logging_mixin.py:103} {PID:118 TID:140374674740992} INFO - 
  0%|          | 0/50 [24:31<?, ?trial/s, best loss=?]

I also get frequent warning before this error : WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_65 !

This error doen't seem to be consistent. On few trials it executes seemlessly and on others it fails. For complete log, please see this file .

This is where the error generates :

Info :

mmlspark jar package: [com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc3]
PySpark Version [2.4.4]
Python [3.7]
Hyperopt [0.2.7]

AB#1984507

mhamilton723 commented 2 years ago

Hey @mybinaryworld thanks for raising this. Tagging in @imatiach-msft for more info. On first glance this seems like a bit of an old version and Ilya has made many fixes since then, do you still have this happen on SynapseML 0.9.4?

mybinaryworld commented 2 years ago

@mhamilton723 I am still using spark2 and hence did not try using SynapseML. Migrating to Spark3.0 will take some time for my team so I was looking for ways to make this work. I would also like to add that the above error doesn't occur when I run this on small dataset.

mybinaryworld commented 2 years ago

@mhamilton723 @imatiach-msft I updated my spark to V3.0.0 and package to SynapseML 0.9.4 and the error persists. Is it due to LightGBM? How can I resolve this? Here is the error :

[2021-12-22 20:29:06,702] {logging_mixin.py:103} {PID:117 TID:139828020053760} INFO - 
[2021-12-22 20:29:06,702] {logging_mixin.py:95} {PID:117 TID:139828020053760} INFO - [2021-12-22 20:29:06,702] {agg_modeling.py:551} {PID:117 TID:139828020053760} INFO - {'baggingFraction': 0.763026835184589, 'baggingFreq': 530, 'boostingType': 'gbdt', 'earlyStoppingRound': 24, 'featureFraction': 0.6636630321426704, 'lambdaL1': 4768727.813879636, 'lambdaL2': 7863574.303004997, 'learningRate': 0.20284343350618989, 'maxBin': 203, 'maxDepth': 30, 'minSumHessianInLeaf': 0.06397533908219112, 'numBatches': 23, 'numIterations': 69, 'numLeaves': 120}
[2021-12-22 20:29:06,702] {logging_mixin.py:95} {PID:117 TID:139828020053760} INFO - 
[2021-12-22 20:29:06,702] {logging_mixin.py:103} {PID:117 TID:139828020053760} INFO - 
 10%|#         | 1/10 [01:01<09:09, 61.07s/trial, best loss: 0.3172694882763601]
[2021-12-22 20:30:50,228] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,228 ERROR scheduler.TaskSchedulerImpl: Lost executor 2 on 100.69.181.207: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[2021-12-22 20:30:50,231] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,231 WARN scheduler.TaskSetManager: Lost task 19.0 in stage 59.0 (TID 1057, 100.69.181.207, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[2021-12-22 20:30:50,232] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,232 WARN scheduler.TaskSetManager: Lost task 20.0 in stage 59.0 (TID 1060, 100.69.181.207, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[2021-12-22 20:30:50,233] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,232 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 59.0 (TID 1042, 100.69.181.207, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[2021-12-22 20:30:50,233] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,232 WARN scheduler.TaskSetManager: Lost task 8.0 in stage 59.0 (TID 1051, 100.69.181.207, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[2021-12-22 20:30:50,233] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,232 WARN scheduler.TaskSetManager: Lost task 12.0 in stage 59.0 (TID 1054, 100.69.181.207, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[2021-12-22 20:30:50,233] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,232 WARN scheduler.TaskSetManager: Lost task 23.0 in stage 59.0 (TID 1063, 100.69.181.207, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[2021-12-22 20:30:50,233] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,232 WARN scheduler.TaskSetManager: Lost task 7.0 in stage 59.0 (TID 1048, 100.69.181.207, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
[2021-12-22 20:30:50,235] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,235 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_242_14 !
[2021-12-22 20:30:50,236] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,235 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_242_2 !
[2021-12-22 20:30:50,236] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,235 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_9_22 !
[2021-12-22 20:30:50,236] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,235 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_9_7 !
[2021-12-22 20:30:50,236] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,235 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_242_8 !
[2021-12-22 20:30:50,236] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,236 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_9_13 !
[2021-12-22 20:30:50,236] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,236 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_242_11 !
[2021-12-22 20:30:50,236] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,236 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_9_16 !
[2021-12-22 20:30:50,236] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,236 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_242_20 !
[2021-12-22 20:30:50,236] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,236 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_242_23 !
[2021-12-22 20:30:50,236] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,236 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_9_1 !
[2021-12-22 20:30:50,237] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,236 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_9_4 !
[2021-12-22 20:30:50,237] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,236 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_242_5 !
[2021-12-22 20:30:50,237] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,236 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_242_17 !
[2021-12-22 20:30:50,237] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,236 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_9_19 !
[2021-12-22 20:30:50,237] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:50,236 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_9_10 !
[2021-12-22 20:30:52,279] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:30:52,279 WARN scheduler.TaskSetManager: Lost task 8.1 in stage 59.0 (TID 1068, 100.67.139.131, executor 0): java.net.ConnectException: Connection refused (Connection refused)
[2021-12-22 20:30:52,279] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.PlainSocketImpl.socketConnect(Native Method)
[2021-12-22 20:30:52,280] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[2021-12-22 20:30:52,280] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[2021-12-22 20:30:52,280] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[2021-12-22 20:30:52,280] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[2021-12-22 20:30:52,280] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:589)
[2021-12-22 20:30:52,280] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:538)
[2021-12-22 20:30:52,280] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:434)
[2021-12-22 20:30:52,281] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:211)
[2021-12-22 20:30:52,281] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:240)
[2021-12-22 20:30:52,281] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$2(TrainUtils.scala:341)
[2021-12-22 20:30:52,281] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:24)
[2021-12-22 20:30:52,281] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:30:52,281] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:30:52,281] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:30:52,281] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:30:52,281] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$1(TrainUtils.scala:341)
[2021-12-22 20:30:52,282] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.env.StreamUtilities$.using(StreamUtilities.scala:29)
[2021-12-22 20:30:52,282] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInfo(TrainUtils.scala:337)
[2021-12-22 20:30:52,282] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:354)
[2021-12-22 20:30:52,282] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:481)
[2021-12-22 20:30:52,282] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:195)
[2021-12-22 20:30:52,282] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[2021-12-22 20:30:52,282] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[2021-12-22 20:30:52,282] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-22 20:30:52,283] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-22 20:30:52,283] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-22 20:30:52,283] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-22 20:30:52,283] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-22 20:30:52,283] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-22 20:30:52,283] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-22 20:30:52,283] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-22 20:30:52,283] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-22 20:30:52,283] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-22 20:30:52,284] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.Task.run(Task.scala:127)
[2021-12-22 20:30:52,284] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
[2021-12-22 20:30:52,284] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[2021-12-22 20:30:52,284] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
[2021-12-22 20:30:52,284] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-22 20:30:52,284] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-22 20:30:52,284] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.lang.Thread.run(Thread.java:748)
[2021-12-22 20:30:52,284] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 
[2021-12-22 20:31:00,534] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:31:00,534 ERROR scheduler.TaskSetManager: Task 7 in stage 59.0 failed 8 times; aborting job
[2021-12-22 20:31:00,544] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:31:00,542 ERROR lightgbm.LightGBMRegressor: {"uid":"LightGBMRegressor_c32669dc98e0","className":"class com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor","method":"train","buildVersion":"0.9.4"}
[2021-12-22 20:31:00,544] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 59.0 failed 8 times, most recent failure: Lost task 7.7 in stage 59.0 (TID 1090, 100.67.139.131, executor 0): java.net.ConnectException: Connection refused (Connection refused)
[2021-12-22 20:31:00,544] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.PlainSocketImpl.socketConnect(Native Method)
[2021-12-22 20:31:00,544] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[2021-12-22 20:31:00,544] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:589)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:538)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:434)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:211)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:240)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$2(TrainUtils.scala:341)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:24)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:31:00,545] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$1(TrainUtils.scala:341)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.env.StreamUtilities$.using(StreamUtilities.scala:29)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInfo(TrainUtils.scala:337)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:354)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:481)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:195)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-22 20:31:00,546] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.Task.run(Task.scala:127)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-22 20:31:00,547] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-22 20:31:00,548] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.lang.Thread.run(Thread.java:748)
[2021-12-22 20:31:00,548] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 
[2021-12-22 20:31:00,548] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold Driver stacktrace:
[2021-12-22 20:31:00,548] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2023)
[2021-12-22 20:31:00,548] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1972)
[2021-12-22 20:31:00,548] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1971)
[2021-12-22 20:31:00,548] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[2021-12-22 20:31:00,548] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[2021-12-22 20:31:00,548] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[2021-12-22 20:31:00,548] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1971)
[2021-12-22 20:31:00,549] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
[2021-12-22 20:31:00,549] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
[2021-12-22 20:31:00,549] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at scala.Option.foreach(Option.scala:407)
[2021-12-22 20:31:00,549] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
[2021-12-22 20:31:00,549] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2203)
[2021-12-22 20:31:00,549] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2152)
[2021-12-22 20:31:00,549] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2141)
[2021-12-22 20:31:00,549] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
[2021-12-22 20:31:00,549] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
[2021-12-22 20:31:00,549] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.SparkContext.runJob(SparkContext.scala:2093)
[2021-12-22 20:31:00,550] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.SparkContext.runJob(SparkContext.scala:2188)
[2021-12-22 20:31:00,550] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$reduce$1(RDD.scala:1094)
[2021-12-22 20:31:00,550] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[2021-12-22 20:31:00,550] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[2021-12-22 20:31:00,550] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
[2021-12-22 20:31:00,550] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.reduce(RDD.scala:1076)
[2021-12-22 20:31:00,550] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.Dataset.$anonfun$reduce$1(Dataset.scala:1740)
[2021-12-22 20:31:00,550] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.Dataset.$anonfun$withNewRDDExecutionId$1(Dataset.scala:3605)
[2021-12-22 20:31:00,550] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
[2021-12-22 20:31:00,550] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
[2021-12-22 20:31:00,551] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
[2021-12-22 20:31:00,551] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:763)
[2021-12-22 20:31:00,551] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
[2021-12-22 20:31:00,551] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.Dataset.withNewRDDExecutionId(Dataset.scala:3603)
[2021-12-22 20:31:00,551] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.Dataset.reduce(Dataset.scala:1740)
[2021-12-22 20:31:00,551] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.innerTrain(LightGBMBase.scala:486)
[2021-12-22 20:31:00,551] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.innerTrain$(LightGBMBase.scala:443)
[2021-12-22 20:31:00,551] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.innerTrain(LightGBMRegressor.scala:38)
[2021-12-22 20:31:00,551] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$train$4(LightGBMBase.scala:57)
[2021-12-22 20:31:00,551] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at scala.collection.IndexedSeqOptimized.foldLeft(IndexedSeqOptimized.scala:60)
[2021-12-22 20:31:00,552] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at scala.collection.IndexedSeqOptimized.foldLeft$(IndexedSeqOptimized.scala:68)
[2021-12-22 20:31:00,552] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at scala.collection.mutable.ArrayOps$ofRef.foldLeft(ArrayOps.scala:198)
[2021-12-22 20:31:00,552] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$train$1(LightGBMBase.scala:49)
[2021-12-22 20:31:00,552] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb(BasicLogging.scala:63)
[2021-12-22 20:31:00,552] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb$(BasicLogging.scala:60)
[2021-12-22 20:31:00,552] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.logVerb(LightGBMRegressor.scala:38)
[2021-12-22 20:31:00,552] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain(BasicLogging.scala:49)
[2021-12-22 20:31:00,552] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain$(BasicLogging.scala:48)
[2021-12-22 20:31:00,552] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.logTrain(LightGBMRegressor.scala:38)
[2021-12-22 20:31:00,552] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train(LightGBMBase.scala:44)
[2021-12-22 20:31:00,552] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train$(LightGBMBase.scala:43)
[2021-12-22 20:31:00,553] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.train(LightGBMRegressor.scala:38)
[2021-12-22 20:31:00,553] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.train(LightGBMRegressor.scala:38)
[2021-12-22 20:31:00,553] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.ml.Predictor.fit(Predictor.scala:150)
[2021-12-22 20:31:00,553] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2021-12-22 20:31:00,553] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[2021-12-22 20:31:00,553] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2021-12-22 20:31:00,553] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.lang.reflect.Method.invoke(Method.java:498)
[2021-12-22 20:31:00,553] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
[2021-12-22 20:31:00,553] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
[2021-12-22 20:31:00,554] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at py4j.Gateway.invoke(Gateway.java:282)
[2021-12-22 20:31:00,554] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
[2021-12-22 20:31:00,554] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at py4j.commands.CallCommand.execute(CallCommand.java:79)
[2021-12-22 20:31:00,554] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at py4j.GatewayConnection.run(GatewayConnection.java:238)
[2021-12-22 20:31:00,554] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.lang.Thread.run(Thread.java:748)
[2021-12-22 20:31:00,554] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold Caused by: java.net.ConnectException: Connection refused (Connection refused)
[2021-12-22 20:31:00,554] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.PlainSocketImpl.socketConnect(Native Method)
[2021-12-22 20:31:00,554] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[2021-12-22 20:31:00,554] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[2021-12-22 20:31:00,554] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[2021-12-22 20:31:00,555] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[2021-12-22 20:31:00,555] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:589)
[2021-12-22 20:31:00,555] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:538)
[2021-12-22 20:31:00,555] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:434)
[2021-12-22 20:31:00,555] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:211)
[2021-12-22 20:31:00,555] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:240)
[2021-12-22 20:31:00,555] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$2(TrainUtils.scala:341)
[2021-12-22 20:31:00,555] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:24)
[2021-12-22 20:31:00,555] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:31:00,555] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:31:00,556] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:31:00,556] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-22 20:31:00,556] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$1(TrainUtils.scala:341)
[2021-12-22 20:31:00,556] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.env.StreamUtilities$.using(StreamUtilities.scala:29)
[2021-12-22 20:31:00,556] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInfo(TrainUtils.scala:337)
[2021-12-22 20:31:00,556] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:354)
[2021-12-22 20:31:00,556] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:481)
[2021-12-22 20:31:00,556] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:195)
[2021-12-22 20:31:00,556] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[2021-12-22 20:31:00,556] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[2021-12-22 20:31:00,557] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-22 20:31:00,557] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-22 20:31:00,557] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-22 20:31:00,557] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-22 20:31:00,557] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-22 20:31:00,557] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-22 20:31:00,557] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-22 20:31:00,557] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-22 20:31:00,557] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-22 20:31:00,557] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-22 20:31:00,558] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.Task.run(Task.scala:127)
[2021-12-22 20:31:00,558] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
[2021-12-22 20:31:00,558] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[2021-12-22 20:31:00,558] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
[2021-12-22 20:31:00,558] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-22 20:31:00,558] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-22 20:31:00,558] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold     ... 1 more
[2021-12-22 20:31:00,564] {base_task_runner.py:101} {PID:75 TID:140492833617664} INFO - Job 369740: Subtask run_hyperparam_search_with_fold 2021-12-22 20:31:00,564 WARN scheduler.TaskSetManager: Lost task 12.7 in stage 59.0 (TID 1092, 100.67.139.131, executor 0): TaskKilled (Stage cancelled)
[2021-12-22 20:31:00,599] {logging_mixin.py:103} {PID:117 TID:139828020053760} INFO - 

[2021-12-22 20:31:00,599] {logging_mixin.py:103} {PID:117 TID:139828020053760} INFO - 
[2021-12-22 20:31:00,599] {logging_mixin.py:95} {PID:117 TID:139828020053760} INFO - [2021-12-22 20:31:00,599] {driver_hyperopt_param_search.py:72} {PID:117 TID:139828020053760} INFO - Exception while executing the objective function : Py4JJavaError('An error occurred while calling o1080.fit.\n', JavaObject id=o1251)
[2021-12-22 20:31:00,599] {logging_mixin.py:95} {PID:117 TID:139828020053760} INFO - 
[2021-12-22 20:31:00,600] {logging_mixin.py:103} {PID:117 TID:139828020053760} INFO -

And this is another set of error I am getting:


[2021-12-23 16:55:15,071] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 16:55:15,071 WARN scheduler.TaskSetManager: Lost task 5.0 in stage 1901.0 (TID 92204, 100.69.111.13, executor 96): TaskKilled (Stage cancelled)
[2021-12-23 17:24:39,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 17:24:39,960 WARN scheduler.TaskSetManager: Lost task 26.0 in stage 1902.0 (TID 92273, 100.67.181.19, executor 95): java.lang.NullPointerException
[2021-12-23 17:24:39,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getMainWorkerPort(TrainUtils.scala:304)
[2021-12-23 17:24:39,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getReturnBooster(TrainUtils.scala:361)
[2021-12-23 17:24:39,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:368)
[2021-12-23 17:24:39,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:481)
[2021-12-23 17:24:39,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:195)
[2021-12-23 17:24:39,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[2021-12-23 17:24:39,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.Task.run(Task.scala:127)
[2021-12-23 17:24:39,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
[2021-12-23 17:24:39,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[2021-12-23 17:24:39,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
[2021-12-23 17:24:39,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-23 17:24:39,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-23 17:24:39,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.lang.Thread.run(Thread.java:748)
[2021-12-23 17:24:39,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 
[2021-12-23 17:24:43,394] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 17:24:43,394 WARN scheduler.TaskSetManager: Lost task 34.1 in stage 1902.0 (TID 92285, 100.67.181.19, executor 95): java.net.ConnectException: Connection refused (Connection refused)
[2021-12-23 17:24:43,395] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.PlainSocketImpl.socketConnect(Native Method)
[2021-12-23 17:24:43,395] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[2021-12-23 17:24:43,395] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[2021-12-23 17:24:43,395] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[2021-12-23 17:24:43,395] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[2021-12-23 17:24:43,395] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:589)
[2021-12-23 17:24:43,395] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:538)
[2021-12-23 17:24:43,395] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:434)
[2021-12-23 17:24:43,395] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:211)
[2021-12-23 17:24:43,395] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:240)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$2(TrainUtils.scala:341)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:24)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$1(TrainUtils.scala:341)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.env.StreamUtilities$.using(StreamUtilities.scala:29)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInfo(TrainUtils.scala:337)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:354)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:481)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:195)
[2021-12-23 17:24:43,396] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.Task.run(Task.scala:127)
[2021-12-23 17:24:43,397] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
[2021-12-23 17:24:43,398] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[2021-12-23 17:24:43,398] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
[2021-12-23 17:24:43,398] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-23 17:24:43,398] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-23 17:24:43,398] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.lang.Thread.run(Thread.java:748)
[2021-12-23 17:24:43,398] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 
[2021-12-23 17:24:53,600] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 17:24:53,600 WARN scheduler.TaskSetManager: Lost task 7.3 in stage 1902.0 (TID 92306, 100.67.181.19, executor 95): java.net.ConnectException: Connection refused (Connection refused)
[2021-12-23 17:24:53,600] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.PlainSocketImpl.socketConnect(Native Method)
[2021-12-23 17:24:53,600] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[2021-12-23 17:24:53,600] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[2021-12-23 17:24:53,600] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[2021-12-23 17:24:53,600] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[2021-12-23 17:24:53,600] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:589)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:538)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:434)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:211)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:240)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$2(TrainUtils.scala:341)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:24)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$1(TrainUtils.scala:341)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.env.StreamUtilities$.using(StreamUtilities.scala:29)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInfo(TrainUtils.scala:337)
[2021-12-23 17:24:53,601] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:354)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:481)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:195)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-23 17:24:53,602] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.Task.run(Task.scala:127)
[2021-12-23 17:24:53,603] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
[2021-12-23 17:24:53,603] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[2021-12-23 17:24:53,603] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
[2021-12-23 17:24:53,603] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-23 17:24:53,603] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-23 17:24:53,603] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.lang.Thread.run(Thread.java:748)
[2021-12-23 17:24:53,603] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 
[2021-12-23 17:25:00,953] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 17:25:00,952 ERROR scheduler.TaskSetManager: Task 12 in stage 1902.0 failed 8 times; aborting job
[2021-12-23 17:25:00,954] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 17:25:00,954 ERROR lightgbm.LightGBMRegressor: {"uid":"LightGBMRegressor_12939d4abcf3","className":"class com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor","method":"train","buildVersion":"0.9.4"}
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold org.apache.spark.SparkException: Job aborted due to stage failure: Task 12 in stage 1902.0 failed 8 times, most recent failure: Lost task 12.7 in stage 1902.0 (TID 92324, 100.67.181.19, executor 95): java.net.ConnectException: Connection refused (Connection refused)
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.PlainSocketImpl.socketConnect(Native Method)
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:589)
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:538)
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:434)
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:211)
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:240)
[2021-12-23 17:25:00,955] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$2(TrainUtils.scala:341)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:24)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$1(TrainUtils.scala:341)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.env.StreamUtilities$.using(StreamUtilities.scala:29)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInfo(TrainUtils.scala:337)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:354)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:481)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:195)
[2021-12-23 17:25:00,956] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.Task.run(Task.scala:127)
[2021-12-23 17:25:00,957] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.lang.Thread.run(Thread.java:748)
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold Driver stacktrace:
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2023)
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1972)
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1971)
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[2021-12-23 17:25:00,958] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1971)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at scala.Option.foreach(Option.scala:407)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2203)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2152)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2141)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.SparkContext.runJob(SparkContext.scala:2093)
[2021-12-23 17:25:00,959] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.SparkContext.runJob(SparkContext.scala:2188)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$reduce$1(RDD.scala:1094)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.reduce(RDD.scala:1076)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.Dataset.$anonfun$reduce$1(Dataset.scala:1740)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.Dataset.$anonfun$withNewRDDExecutionId$1(Dataset.scala:3605)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:763)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
[2021-12-23 17:25:00,960] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.Dataset.withNewRDDExecutionId(Dataset.scala:3603)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.Dataset.reduce(Dataset.scala:1740)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.innerTrain(LightGBMBase.scala:486)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.innerTrain$(LightGBMBase.scala:443)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.innerTrain(LightGBMRegressor.scala:38)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$train$4(LightGBMBase.scala:57)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at scala.collection.IndexedSeqOptimized.foldLeft(IndexedSeqOptimized.scala:60)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at scala.collection.IndexedSeqOptimized.foldLeft$(IndexedSeqOptimized.scala:68)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at scala.collection.mutable.ArrayOps$ofRef.foldLeft(ArrayOps.scala:198)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$train$1(LightGBMBase.scala:49)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb(BasicLogging.scala:63)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb$(BasicLogging.scala:60)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.logVerb(LightGBMRegressor.scala:38)
[2021-12-23 17:25:00,961] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain(BasicLogging.scala:49)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain$(BasicLogging.scala:48)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.logTrain(LightGBMRegressor.scala:38)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train(LightGBMBase.scala:44)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train$(LightGBMBase.scala:43)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.train(LightGBMRegressor.scala:38)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.train(LightGBMRegressor.scala:38)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.ml.Predictor.fit(Predictor.scala:150)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.ml.Predictor.fit(Predictor.scala:114)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at sun.reflect.GeneratedMethodAccessor188.invoke(Unknown Source)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.lang.reflect.Method.invoke(Method.java:498)
[2021-12-23 17:25:00,962] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at py4j.Gateway.invoke(Gateway.java:282)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at py4j.commands.CallCommand.execute(CallCommand.java:79)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at py4j.GatewayConnection.run(GatewayConnection.java:238)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.lang.Thread.run(Thread.java:748)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold Caused by: java.net.ConnectException: Connection refused (Connection refused)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.PlainSocketImpl.socketConnect(Native Method)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[2021-12-23 17:25:00,963] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[2021-12-23 17:25:00,964] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:589)
[2021-12-23 17:25:00,964] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.connect(Socket.java:538)
[2021-12-23 17:25:00,964] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:434)
[2021-12-23 17:25:00,964] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.net.Socket.<init>(Socket.java:211)
[2021-12-23 17:25:00,964] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:240)
[2021-12-23 17:25:00,964] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$2(TrainUtils.scala:341)
[2021-12-23 17:25:00,964] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:24)
[2021-12-23 17:25:00,964] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:25:00,964] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:25:00,964] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.$anonfun$getNetworkInfo$1(TrainUtils.scala:341)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.core.env.StreamUtilities$.using(StreamUtilities.scala:29)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.TrainUtils$.getNetworkInfo(TrainUtils.scala:337)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:354)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:481)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:195)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:25:00,965] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.scheduler.Task.run(Task.scala:127)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
[2021-12-23 17:25:00,966] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-23 17:25:00,967] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-23 17:25:00,967] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold     ... 1 more
[2021-12-23 17:25:00,967] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 17:25:00,957 WARN scheduler.TaskSetManager: Lost task 25.0 in stage 1902.0 (TID 92272, 100.67.181.19, executor 95): TaskKilled (Stage cancelled)
[2021-12-23 17:25:00,967] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 17:25:00,964 WARN scheduler.TaskSetManager: Lost task 26.6 in stage 1902.0 (TID 92328, 100.67.181.19, executor 95): TaskKilled (Stage cancelled)
[2021-12-23 17:25:00,975] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 17:25:00,975 WARN scheduler.TaskSetManager: Lost task 2.5 in stage 1902.0 (TID 92329, 100.67.181.19, executor 95): TaskKilled (Stage cancelled)
[2021-12-23 17:25:00,976] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 17:25:00,976 WARN scheduler.TaskSetManager: Lost task 45.5 in stage 1902.0 (TID 92327, 100.67.181.19, executor 95): TaskKilled (Stage cancelled)
[2021-12-23 17:25:00,976] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 17:25:00,976 WARN scheduler.TaskSetManager: Lost task 16.5 in stage 1902.0 (TID 92326, 100.67.181.19, executor 95): TaskKilled (Stage cancelled)
[2021-12-23 17:25:01,003] {base_task_runner.py:101} {PID:75 TID:140191452321536} INFO - Job 370178: Subtask run_hyperparam_search_with_fold 2021-12-23 17:25:01,003 WARN scheduler.TaskSetManager: Lost task 7.5 in stage 1902.0 (TID 92323, 100.67.181.19, executor 95): TaskKilled (Stage cancelled)
[2021-12-23 17:25:01,010] {logging_mixin.py:103} {PID:117 TID:140394256578304} INFO - 

[2021-12-23 17:25:01,010] {logging_mixin.py:103} {PID:117 TID:140394256578304} INFO - 
[2021-12-23 17:25:01,010] {logging_mixin.py:95} {PID:117 TID:140394256578304} INFO - [2021-12-23 17:25:01,009] {driver_hyperopt_param_search.py:72} {PID:117 TID:140394256578304} INFO - Exception while executing the objective function : Py4JJavaError('An error occurred while calling o14705.fit.\n', JavaObject id=o14891)
[2021-12-23 17:25:01,010] {logging_mixin.py:95} {PID:117 TID:140394256578304} INFO - 
[2021-12-23 17:25:01,010] {logging_mixin.py:103} {PID:117 TID:140394256578304} INFO - ```

microsoft / SynapseML

Error while using mmlspark Lightgbm : Py4JJavaError('An error occurred while calling o527.fit.\n', JavaObject id=o680) #1291