search

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
9.47k stars 875 forks source link

Simplify setting credentials when switching between different environments #3940

Open ElenaKhaustova opened 3 weeks ago

ElenaKhaustova commented 3 weeks ago

Description

Accessing credentials and setting them for different environments is not simple. Users have to define separate catalog entries for different environments to be able to switch between them. They expect to define one catalog entry per credential or/and some guidance on how to make it simpler for such cases.

We propose to:

  1. Explore the feasibility of such a request.
  2. Check if it is possible to simplify switching between environments with the existing credential configuration options and update documentation with the relevant examples.

Relates to https://github.com/kedro-org/kedro/issues/3811

Context

When running on local they had credentials.yml with a password but when running on AWS Glue they had to use this method to access the same credentials.

image (14)

datajoely commented 3 weeks ago

I think a powerful OmegaConf resolver would be really useful...

"${credentials: sql_user}"

The dumbest version of this is something like this, but it could be way more useful:

def get_credentials(path) -> str:
    """Retrieve credentials earlier in the lifecycle"""
    data = OmegaConf.load("conf/local/credentials.yml")
    return OmegaConf.select(data, path)