search

databricks-demos / dbdemos

Demos to implement your Databricks Lakehouse
Other
277 stars 89 forks source link

lakehouse-iot-platform DLT pipeline is failing #138

Open pushpender-db opened 3 weeks ago

pushpender-db commented 3 weeks ago

lakehouse-iot-platform DLT pipeline is failing in task "turbine_current_status" with error 'Booster' object has no attribute 'handle' Here is complete stacktrace

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 566.0 failed 4 times, most recent failure: Lost task 0.3 in stage 566.0 (TID 1176) (10.224.4.249 executor 0): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/pyfunc/init.py", line 2113, in udf if scoring_server_proc is not None: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/pyfunc/init.py", line 1900, in _predict_row_batch result = predict_fn(pdf, params) ^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/pyfunc/init.py", line 2093, in batch_predict_fn return loaded_model.predict(pdf, params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/pyfunc/init.py", line 738, in predict return self._predict(data, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/pyfunc/init.py", line 771, in _predict return self._predict_fn(data, params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/sklearn/init.py", line 534, in predict return self.sklearn_model.predict(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/sklearn/pipeline.py", line 458, in predict return self.steps[-1][1].predict(Xt, predict_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/lightgbm/sklearn.py", line 984, in predict result = self.predict_proba(X, raw_score, start_iteration, num_iteration, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/lightgbm/sklearn.py", line 997, in predict_proba result = super().predict(X, raw_score, start_iteration, num_iteration, pred_leaf, pred_contrib, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/lightgbm/sklearn.py", line 803, in predict return self._Booster.predict(X, raw_score=raw_score, start_iteration=start_iteration, num_iteration=num_iteration, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/lightgbm/basic.py", line 3532, in predict predictor = self._to_predictor(deepcopy(kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/lightgbm/basic.py", line 3622, in _to_predictor predictor = _InnerPredictor(booster_handle=self.handle, pred_parameter=pred_parameter) ^^^^^^^^^^^ AttributeError: 'Booster' object has no attribute 'handle'

at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:552)
at org.apache.spark.sql.execution.python.PythonArrowOutput$$anon$1.read(PythonArrowOutput.scala:117)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:508)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:50)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:195)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:56)
at org.apache.spark.scheduler.ShuffleMapTask.$anonfun$runTask$3(ShuffleMapTask.scala:92)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.ShuffleMapTask.$anonfun$runTask$1(ShuffleMapTask.scala:87)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:58)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:39)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:211)
at org.apache.spark.scheduler.Task.doRunTask(Task.scala:190)
at org.apache.spark.scheduler.Task.$anonfun$run$5(Task.scala:155)
at com.databricks.unity.UCSEphemeralState$Handle.runWith(UCSEphemeralState.scala:45)
at com.databricks.unity.HandleImpl.runWith(UCSHandle.scala:103)
at com.databricks.unity.HandleImpl.$anonfun$runWithAndClose$1(UCSHandle.scala:108)
at scala.util.Using$.resource(Using.scala:269)
at com.databricks.unity.HandleImpl.runWithAndClose(UCSHandle.scala:107)
at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:149)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.Task.run(Task.scala:101)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$10(Executor.scala:1013)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:106)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:1016)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:903)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:3905) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:3824) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:3811) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:3811) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1682) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1665) at scala.Option.foreach(Option.scala:407) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1665) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:4151) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:4063) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:4051) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:55) org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/pyfunc/init.py", line 2113, in udf if scoring_server_proc is not None: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/pyfunc/init.py", line 1900, in _predict_row_batch result = predict_fn(pdf, params) ^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/pyfunc/init.py", line 2093, in batch_predict_fn return loaded_model.predict(pdf, params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/pyfunc/init.py", line 738, in predict return self._predict(data, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/pyfunc/init.py", line 771, in _predict return self._predict_fn(data, params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/mlflow/sklearn/init.py", line 534, in predict return self.sklearn_model.predict(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/sklearn/pipeline.py", line 458, in predict return self.steps[-1][1].predict(Xt, predict_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/lightgbm/sklearn.py", line 984, in predict result = self.predict_proba(X, raw_score, start_iteration, num_iteration, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/lightgbm/sklearn.py", line 997, in predict_proba result = super().predict(X, raw_score, start_iteration, num_iteration, pred_leaf, pred_contrib, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/lightgbm/sklearn.py", line 803, in predict return self._Booster.predict(X, raw_score=raw_score, start_iteration=start_iteration, num_iteration=num_iteration, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/lightgbm/basic.py", line 3532, in predict predictor = self._to_predictor(deepcopy(kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-0ce49744-a8c5-4b98-af5f-fe1aa4ee5e82/lib/python3.11/site-packages/lightgbm/basic.py", line 3622, in _to_predictor predictor = _InnerPredictor(booster_handle=self.handle, pred_parameter=pred_parameter) ^^^^^^^^^^^ AttributeError: 'Booster' object has no attribute 'handle'

chik-taks commented 1 day ago

reason: it depends on which version of lightgbm AutoML use. it is currently 4.2.0, but default in notebook, it's 3.3.4

quick fix: change library version of lightgbm to 4.2.0 in "./01-Data-ingestion/01.2-DLT-Wind-Turbine-SQL-UDF"

proposal: change to install libraries using mlflow experiments artifact.