Large prediction results unless using repartition(1) in databricks with lgbm model

user673 commented 3 years ago

I'm using mmlspark lgbm model for regression problem and faced strange thing. If using all normal code as in the example, results will be terrible, becouse predictions are huge (around 10^37 , while target is in range from 0 to 200). Testing, I found that using dataset.repartition(1).cache() fixed this problem,but with one detail - modelling began to take longer (around 1h, while 20m earlier). This is logical since all the data (about 4m rows and 150 columns) is collected before learning in one partition.

I tried changing lgbm param useBarrierExecutionModeto True and different parallelism params, but this changes doesn't affect result.

Is there a way not to use such workaround with repartition and still having normal results?

Code, used for training

      repartitioned_data = data_train.repartition(1).cache() #  want to delete this line

      # Define model
      model = LightGBMRegressor(
          objective='regression',
          labelCol='label',
          featuresCol="features"
      )

      # Define grid params
      paramGrid = ParamGridBuilder() \
              .addGrid(model.numIterations, [100, 250])\
            .build()

      # Define cross validation for grid params
      evaluator = RegressionEvaluator(labelCol="label", predictionCol="prediction", metricName="mae")
      crossval = CrossValidator(estimator=model,
                          estimatorParamMaps=paramGrid,
                          evaluator=evaluator,   
                          numFolds=2)

      # Train model
      pipeline = crossval.fit(data_train)

Databricks Runtime Version 6.4 (includes Apache Spark 2.4.5, Scala 2.11)
3 worker nodes Standard_DS4_v2
driver node Standard_DS4_v2
mmlspark version mmlspark_2.11:1.0.0-rc3

AB#1984587

welcome[bot] commented 3 years ago

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

AllardJM commented 3 years ago

I am seeing something similar - enormous predictions that are sometimes positive and sometimes negative (there are no negative values in the target). It seems that if I use mse as the objective the predictions are all extremely negative and extremely positive if using a tweedie objective. The rank order / discrimination is relatively good but the estimates are orders of magnitude uncalibrated

Using Data Bricks 7.3 LTS Spark 3.0.1 Lightgbm: com.microsoft.ml.spark:mmlspark_2.12:1.0.0-rc3-24-495af3e4-SNAPSHOT

mhamilton723 commented 3 years ago

@AllardJM @user673 Could you possibly share an example dataset for us to repro? Adding @imatiach-msft who built LightGBM on Spark