I'm trying to save an Isolation Forest model after training in SynapseML. However, errors occur, and the save method does not work.
Code to reproduce issue
# building a model
from synapse.ml.isolationforest import *
from pyspark.ml.feature import VectorAssembler
# Isolation Forest parameters
contamination = 0.021
num_estimators = 100
max_samples = 100
max_features = 1.0
# Model Setup
isolationForest = (
IsolationForest()
.setNumEstimators(num_estimators)
.setBootstrap(False)
.setMaxSamples(max_samples)
.setMaxFeatures(max_features)
.setFeaturesCol("features")
.setPredictionCol("predictedLabel")
.setScoreCol("outlierScore")
.setContamination(contamination)
.setContaminationError(0.01 * contamination)
.setRandomSeed(1)
)
# Training
va=VectorAssembler(inputCols=inputCols, outputCol="features")
train_data = va.transform(sdf_train)
model_isolationforest_trained = isolationForest.fit(train_data)
# Predictions
test_data = va.transform(sdf_test)
pred = model_isolationforest_trained.transform(test_data)
# Saving
model_isolationforest_trained.write().overwrite().save("path")
Other info / logs
The part of the errors caused from model_isolationforest_trained.write().overwrite().save("path") is like
output Error:
py4JJavaError: An error occured while calling 03919.save: org.apache.spark.SparkException: Job aborted. ~~
Caused by: java.lang.NoSuchMethodError: 'scala.Function1 orgz.apache.spark.sql.execution.datasources.DataSourceUtils$.createDateRebaseFucInWirte(scala.Enumeration$Value, java.lang.String)' ~~
Does the error imply that we cannot save the trained Isolation Forest model in the SynapseML? As a side note, I confirmed that the save method works with the LightGBMClassifier in the SynapseML.
I would appreciate it if someone could provide any solutions.
What component(s) does this bug affect?
[ ] area/cognitive: Cognitive project
[X] area/core: Core project
[ ] area/deep-learning: DeepLearning project
[ ] area/lightgbm: Lightgbm project
[ ] area/opencv: Opencv project
[ ] area/vw: VW project
[ ] area/website: Website
[ ] area/build: Project build system
[ ] area/notebooks: Samples under notebooks folder
[ ] area/docker: Docker usage
[ ] area/models: models related issue
What language(s) does this bug affect?
[X] language/scala: Scala source code
[X] language/python: Pyspark APIs
[ ] language/r: R APIs
[ ] language/csharp: .NET APIs
[ ] language/new: Proposals for new client languages
Hey @shibuya-phys :wave:!
Thank you so much for reporting the issue/feature request :rotating_light:.
Someone from SynapseML Team will be looking to triage this issue soon.
We appreciate your patience.
SynapseML version
0.10.1
System information
Describe the problem
I'm trying to save an Isolation Forest model after training in SynapseML. However, errors occur, and the save method does not work.
Code to reproduce issue
Other info / logs
The part of the errors caused from
model_isolationforest_trained.write().overwrite().save("path")
is likeDoes the error imply that we cannot save the trained Isolation Forest model in the SynapseML? As a side note, I confirmed that the save method works with the LightGBMClassifier in the SynapseML. I would appreciate it if someone could provide any solutions.
What component(s) does this bug affect?
area/cognitive
: Cognitive projectarea/core
: Core projectarea/deep-learning
: DeepLearning projectarea/lightgbm
: Lightgbm projectarea/opencv
: Opencv projectarea/vw
: VW projectarea/website
: Websitearea/build
: Project build systemarea/notebooks
: Samples under notebooks folderarea/docker
: Docker usagearea/models
: models related issueWhat language(s) does this bug affect?
language/scala
: Scala source codelanguage/python
: Pyspark APIslanguage/r
: R APIslanguage/csharp
: .NET APIslanguage/new
: Proposals for new client languagesWhat integration(s) does this bug affect?
integrations/synapse
: Azure Synapse integrationsintegrations/azureml
: Azure ML integrationsintegrations/databricks
: Databricks integrations