Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
In our current test setup starters & framework are linked and look to the "latest" version. At release time it leads to a situation where it's looking for the unreleased version and tests fail because of it.
Implement a way to fall back to the latest released version OR specify a specific version.
As a result, when we need to release both kedro-starters and kedro, we cannot specify a custom kedro-starters version on the kedro side. So we have to make a release with failing tests and only after making sure that tests are passing at the CI.
Possible Implementation
We already have a checkout argument that allows pointing to a specific tag or branch, which is used as an optional argument for the kedro new command. We can add a constant to specify the checkout argument as we do for starters repo: https://github.com/kedro-org/kedro/blob/27f5405cefd6701ffac4c6243030486fb7d3c942/kedro/framework/cli/starters.py#L98
Then, in case the checkout argument was provided, it will be used instead of constant. For the rest of the cases, including testing, we will use the constant and we only synchronise kedro-starters branch with kedro version when the checkout argument is not provided and the constant is set to None.
In that case to release both kedro-starters and kedro we would need to:
Point to target kedro branch A in kedro-starters
Set up checkout constant in branch A for kedro, pointing to the corresponding branch in kedro-starters
Description
In our current test setup starters & framework are linked and look to the "latest" version. At release time it leads to a situation where it's looking for the unreleased version and tests fail because of it.
Implement a way to fall back to the latest released version OR specify a specific version.
Context
"There's always cookiecutter" - @lrcouto
Currently
kedro-starterspoints to mainkedrobranch: https://github.com/kedro-org/kedro-starters/blob/db79aec64c4a0f062321bd8c74ee782750bba527/test_requirements.txt#L5 At the same time onkedroside we point to the mainkedro-startersbranch and synchronise it withkedroversion used: https://github.com/kedro-org/kedro/blob/27f5405cefd6701ffac4c6243030486fb7d3c942/kedro/framework/cli/starters.py#L778 https://github.com/kedro-org/kedro/blob/27f5405cefd6701ffac4c6243030486fb7d3c942/tests/framework/cli/test_starters.py#L56As a result, when we need to release both
kedro-startersandkedro, we cannot specify a customkedro-startersversion on thekedroside. So we have to make a release with failing tests and only after making sure that tests are passing at the CI.Possible Implementation
We already have a
checkoutargument that allows pointing to a specific tag or branch, which is used as an optional argument for thekedro newcommand. We can add a constant to specify thecheckoutargument as we do for starters repo: https://github.com/kedro-org/kedro/blob/27f5405cefd6701ffac4c6243030486fb7d3c942/kedro/framework/cli/starters.py#L98 Then, in case thecheckoutargument was provided, it will be used instead of constant. For the rest of the cases, including testing, we will use the constant and we only synchronisekedro-startersbranch withkedroversion when thecheckoutargument is not provided and the constant is set toNone.In that case to release both
kedro-startersandkedrowe would need to:kedrobranchAinkedro-starterscheckoutconstant in branchAforkedro, pointing to the corresponding branch inkedro-startersPossible Alternatives