search

oracle / accelerated-data-science

ADS is the Oracle Data Science Cloud Service's python SDK supporting, model ops (train/eval/deploy), along with running workloads on Jobs and Pipeline resources.
https://accelerated-data-science.readthedocs.io/
Universal Permissive License v1.0
89 stars 44 forks source link

Allowing all parameters from `CreateApplicationDetails` to be passed through `prepare_app` #6

Closed xingyaoc closed 2 years ago

xingyaoc commented 2 years ago

In particular, the Spark configuration properties in OCI dataflow can not be set using ads.dataflow, but it's able to be set with oci.data_flow_models.CreateApplicationDetails. Since prepare_app already calls CreateApplicationDetails it possible to add the rest of the CreateApplicationDetails parameters into prepare_app? https://github.com/oracle/accelerated-data-science/blob/d594ed0c8c1365daf4cf9e860daebc760fa9a24b/ads/dataflow/dataflow.py#L171

mayoor commented 2 years ago

We have new a Data Flow API replacing the old one referred to in this PR. This one lets you pass the configuration as dict.

Doc link - https://accelerated-data-science.readthedocs.io/en/latest/user_guide/apachespark/dataflow.html#ads-python-sdk Code - https://github.com/oracle/accelerated-data-science/blob/main/ads/jobs/builders/infrastructure/dataflow.py#L460