Should `kedro test` and `kedro lint` be deprecated?

[merelcht]

Investigate what the alternatives would be for users currently using kedro test and kedro lint. Write up a plan to deprecating kedro lint and kedro test (including how the users can "migrate" to these alternatives) or alternatively summarise why those commands shouldn't be deprecated.

Also keep in mind that the alternatives should still make it possible to have a linted/testable Kedro template.

[merelcht]

The original research done on this topic: https://github.com/kedro-org/kedro/issues/1293

[jmholzer]

Proposal

The CLI commands kedro test and kedro lint should be deprecated because this is the consensus in #1293.

A summary of the key arguments for deprecation:

1. Utilisation by users is very low.
2. The interface for Kedro would become simpler.
3. Management of dependencies would become simpler.
4. There would be less code to maintain.
5. It is consistent with other design choices (e.g. the move to kedro-datasets, the deprecation of kedro install).

A summary of the key arguments against deprecation:

1. Some users do use these commands.
2. Any of the discussed alternatives are likely to result in even lower utilisation.
3. These commands are congruent with the Kedro design philosophy.

My view is also that the commands should be deprecated. Both commands are wrappers for tools with many alternatives. I believe that if a user knows the value of testing and linting code, they either:

1. Know how to set up and use the wrapped tools autonomously.
2. Have a preference for alternative tools that are not wrapped by these commands.

In these cases, the commands offer limited to no utility.

Plan

To mitigate the disadvantages discussed in the arguments against deprecation:

1. Add demos for implementing testing and linting in a Kedro project to the documentation.
2. Add code (e.g. a decorator) for both commands to generate a warning to the user that they will be deprecated. A link should be included to the documentation from step one.
3. Consider the creation of a plugin that offers lint, test and other deprecated commands.

In https://github.com/kedro-org/kedro/issues/1293, step three of this plan is somewhat controversial, since it represents an up-front and ongoing investment. One option is to create this plugin as a community-led project. In any case, further discussion is warranted for this step.

[merelcht]

This summary makes sense to me, but I'm missing what the concrete the alternatives would be for users currently using kedro test and kedro lint and how the users can "migrate" to these alternatives.

[jmholzer]

There are two concrete alternatives:

1. Users Install Testing and Linting Tools Independently

Users install and configure the tools that kedro lint (black, flake8, isort) and kedro test (pytest) wrap. They then use these tools through their normal interfaces. We will add documentation to Kedro to describe installation and configuration.

For isort and black, users can create and enter settings into a pyproject.toml file. For flake8, they need to do the same for a setup.cfg file. We will guide users to recreating the current default settings shipped with Kedro in our documentation (step 2 of the plan). These linting tools each have a CLI, though they can also be integrated with most common IDEs through easily-installed plugins. Again, this will be added to our documentation.

In the case of pytest, users can switch to using the pytest CLI very easily, as kedro test is a very lightweight wrapper. This is as simple as running pytest instead of kedro test. There are some pytest-specific configuration settings in Kedro's pyproject.toml file. We can guide users in what these options do and how to add them in the documentation that we will write.

The advantage of this alternative is that it captures the major advantages of deprecation outlined in the arguments above. I estimate that implementing these changes would take an experienced Kedro developer one week.

In summary, users can migrate by replicating the configuration for each tool and using their interfaces independently, which we will guide them in doing in newly written documentation.

Benefit:

• Achieves the key advantages of deprecation outlined in my original response.
• Addresses the third argument against deprecation in my original response (is in line with the Kedro philosophy of improving coding standards in data science).

Cost:

• I estimate that implementing these changes will take one developer one week of full-time work.
• Documentation will have to be maintained to reflect changes to tools.

2. Users Install our 'Dev Plugin'

If we decide to implement step 3 of the plan, users will be able to install the plugin, which bundles both kedro test and kedro lint, potentially among other deprecated commands.

In this case, the user will not need to undergo any migration beyond installing the plugin; `kedro lint' and 'kedro test' will be available as before.

Benefit:

• Achieves the key advantages of deprecation (outlined above).
• Addresses each argument against deprecation, though is still likely to result in lower utilisation overall.

Cost:

• I estimate that implementing these changes will take one experienced Kedro developer two weeks of full-time work.
• The plugin will need to be maintained on an ongoing basis.

[deepyaman]

My view is also that the commands should be deprecated. Both commands are wrappers for tools with many alternatives. I believe that if a user knows the value of testing and linting code, they either:

1. Know how to set up and use the wrapped tools autonomously.
2. Have a preference for alternative tools that are not wrapped by these commands.

In these cases, the commands offer limited to no utility.

This is a fair view. However, why should we even "add demos for implementing testing and linting in a Kedro project to the documentation" in that case? I'd rather a user who "knows the value of testing and linting code" choose their own tools and read the respective docs, so there's nothing for us to maintain here. Otherwise, Kedro maintainers will spend time resolving issues setting up pytest and flake8 when we in fact have no strong view there. IIRC @AntonyMilneQB mentioned something in this vein (I could be wrong). I think it could be handled with a docs page stating the linter config Kedro uses (i.e. project template is linted with isort, black, flake8).

On that note, I think the plan is missing the bit on how to make sure the project template is still linted (somehow).

[yetudada]

This write-up and evaluation is stellar 🎉 An additional benefit is a simpler starter for new users.

If I quickly scan the documentation and starters it would mean that we would be making the following changes (if I haven't forgotten anything):

• Starters:
  • Remove setup.cfg
  • Edit pyproject.toml
  • Edit src/requirements.txt
• Documentation:
  • Edits to anywhere where setup.cfg is mentioned like in the Project directory structure, the section on Astronomer, and the one on Starters
  • Add new sections on testing and linting, probably under Development or Tools integration

I like the documentation addition because in future, the section on testing could be expanded to cover #1271. I think @deepyaman's comment is the only thing left to cover, which is how do we make sure that we deliver linted and tested starters to our users without those configuration files being present in the starters.

[yetudada]

While I was walking, I realised there are more changes 😂 What should happen to src/tests and what should happen to the tests folder created by kedro pipeline create? And we'd also need to make changes to the Kedro website.

[antonymilne]

Both commands are wrappers for tools with many alternatives. I believe that if a user knows the value of testing and linting code, they either... In these cases, the commands offer limited to no utility.

Slightly playing devil's advocate, but something I think we're missing here is what if a user doesn't know the value of testing and linting? Having these commands shipped with kedro does increase discoverability of these notions. If I'm completely new to writing data science pipelines I might not even know about testing and linting, let alone be equipped to choose which tools I should choose or how to set them up. Are we setting users on the right path if we provide them with a project template that doesn't even have a tests folder to suggest that they should write tests?

IMO the value of these tools being built into kedro is most significant to inexperienced users, and so any documentation that effectively replaces them should also cater to that user group.

Just to clarify on alternative 2 (Users Install our 'Dev Plugin'), which I personally still like: this can be done with essentially zero development effort and no ongoing maintenance if we outsource it to the community and make it a community-maintained plugin. And then our docs can just say "linting code is good. If you want a standard set of tools that help with this then we recommend this plugin". See https://github.com/kedro-org/kedro/issues/1622. If there's no demand for the plugin then it won't be created, which is fine. I don't think we need to do anything about it ourselves right away.

That would still leave open the question of what standards we should be linting our own starter templates to and how to do it though.

[merelcht]

Can't we just keep the linting and testing setup in the starters as is now and use that as an example for users to show how they can configure their project to have linting and testing? I realise that with this approach we don't clean up as much, but I feel this would solve the problem of how we keep the starters in good shape and have concrete examples for users how to set up linting and testing. The only change is that we don't lint and test by doing kedro lint and kedro test but by running the underlying commands.

[jmholzer]

In technical design (14/09/22) it was concluded that both commands should be deprecated. The following action items were agreed:

1 (#1619 and #1620). Short guides for linting and testing should be written and added to the docs. Depending on the traffic metrics for this part of the docs, a next step of creating either a lint / test starter* (see below) or a plugin will be considered. 2 (#1849). Explore whether the /tests folder and / or the linting configuration (requirements, config files) should be kept or dropped in the starters and / or template.

* @yetudada proposed moving the elements needed to add both commands to a starter project. This may provide significant advantages in ease of maintenance relative to the plugin option.