nm.save on spark model has multiple sources found for parquet

[Mpicca]

Even adding .format at the end to specify a source do not work. Might need to be updated in the class.

[mbaak]

Hello, Thanks for reporting this issue, I think I see the problem. The SparkEntityMatching class does not inherit from spark's DataFrameWriter but from MLWriter, which does not support the format option (b/c normally it doesn't need it). In other words .format("...") is not picked up here. However we do store the ground-truth dataset, so the format option can be relevant - by default we set this to parquet, but atm there is no overwrite option. I will add a format function, it should be straight-forward to add.

The way it will work: nm.write().format("yourformat").save(path)

Btw, looking around for this error, I'm not fully certain but it seems there's a spark package dependency conflict in your setup. See for example here: https://stackoverflow.com/questions/63579754/error-while-read-or-write-parquet-format-data You may want to look into this and resolve it as well. (The format function is only a workaround.)

[mbaak]

I will pick this up in the next patch release, probably in the next couple of days.

[mbaak]

Resolved with v2.1.0.