getindata / kedro-azureml

Kedro plugin to support running workflows on Microsoft Azure ML Pipelines
https://kedro-azureml.readthedocs.io
Apache License 2.0
37 stars 17 forks source link

Add option to enable cache via configuration #58

Closed marrrcin closed 1 year ago

marrrcin commented 1 year ago

Right now, Azure ML built-in caching mechanism is disabled by the plugin explicitly. We should add an option to enable it on specific nodes based on tags (in the same way as the resources are configured).

Proposed configuration azureml.yml:

azure:
  # rest of the config
  cache:
    __default__: false
    specific_tag: true

then, all nodes tagged with specific_tag should have is_deterministic flag set to true: https://github.com/getindata/kedro-azureml/blob/2e5836b72256d7455d8525c7769a68d4c844ccf7/kedro_azureml/generator.py#L207

marrrcin commented 1 year ago

Closed by #66