This PR fixes Spark job null pointer error in databricks. FOr current job workflow in Azure DBR, the job status returns failed even if the job succeeded, becaue of this NPE.
This PR is tested on DBR 11.3 (spark 3.3.0, scala 2.12) and BDC (spark 3.1.2, scala 2.12).
This PR fixes Spark job null pointer error in databricks. FOr current job workflow in Azure DBR, the job status returns failed even if the job succeeded, becaue of this NPE. This PR is tested on DBR 11.3 (spark 3.3.0, scala 2.12) and BDC (spark 3.1.2, scala 2.12).