logicalclocks / maggy

Distribution transparent Machine Learning experiments on Apache Spark
https://maggy.ai
Apache License 2.0
90 stars 14 forks source link

[Ablation] Develop a dataset generator function for databricks #83

Open RiccardoGrigoletto opened 3 years ago

RiccardoGrigoletto commented 3 years ago

As per https://github.com/logicalclocks/maggy/blob/master/maggy/ablation/ablationstudy.py , we need to write a custom function for the dataset generator to make AblationStudy to work on databricks.

moritzmeister commented 3 years ago

"There is one class that implements the AbstractAbletor class on maggy: LOCO. LOCO implements get_dataset_generator() that uses some hopsworks API to retrieve the datasets and so create the tf_dataset to use during training. We need to create a similar class that do that on databricks."