search

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
10.02k stars 906 forks source link

Users cannot install specific components of Kedro separately #3659

Closed astrojuanlu closed 5 months ago

astrojuanlu commented 9 months ago

Description

Hello, I'm currently building a python package that derives some of the source code of Kedro, primarily the catalog and config classes. I might be open sourcing the package, how do I properly attribute Kedro? Any advice? ... My team is reluctant to add kedro pipelines but liked catalogs and the config management. Also I'm working on integrating lakefs and dvc for version control and dbt.

https://linen-slack.kedro.org/t/16626961/hello-i-m-currently-building-a-python-package-that-derives-s#3e2cc457-1184-4b2d-b7b3-7cab59fa1fa0

Over the years, some Kedro power users and also folks evaluating the project have made it clear that they really like specific parts of Kedro, such as the catalog and the configuration loading, and they don't care much about the rest. See also #2741 (2021), https://github.com/kedro-org/kedro/issues/2898#issuecomment-1736000094, https://linen-slack.kedro.org/t/16593946/is-there-a-way-of-installing-only-the-data-catalog-part-of-k#b6d532c4-2d7f-4add-b0ee-b0bfcffbdd5e, https://github.com/kedro-org/kedro/issues/2409#issuecomment-1466992742 ("kedro is scary") and many more.

What has been done so far

We have already done a lot to make Kedro leaner and simpler. For example, Kedro 0.19 dropped datasets, which now have to be installed separately as kedro-datasets #2126 This has been unanimously celebrated.

In addition, we also dropped some CLI commands that were rarely used #1616. There were some users that raised concerns, but so far we haven't heard more concerns.

We also improved our packaging infrastructure to avoid having explicit dependencies on pip and setuptools #2350 again, this has been received with silent approval.

Why is this still a problem?

And yet, this is not enough.

In principle nothing prevents these users from doing pip install kedro and using only the parts they need. This was the argument @idanov and myself defended in https://github.com/kedro-org/kedro/issues/2409#issuecomment-1479496194

However, this is not how most folks vet open source dependencies. In this absolutely fantastic survey from @/simonw, from Django fame, on Mastodon, people generally favour lean packages over bloated ones, and minimal dependencies over a large number of them. Some quotes:

The only other factor beyond this list I care about is size. Is it a large or "bloated" dependency?

Check how many/what dependencies are required.

one really important step I haven't seen many people mention: look at dependencies. Basically determine at what level the thing is solving its use case.

For users that would like to use only our Data Catalog and/or only our Configuration Loader, they would be forced to:

cookiecutter
├── Jinja2<4.0.0,>=2.7
│   └── MarkupSafe>=2.0
├── arrow
│   ├── python-dateutil>=2.7.0
│   │   └── six>=1.5
│   └── types-python-dateutil>=2.8.10
├── binaryornot>=0.4.4
│   └── chardet>=3.0.2
├── click<9.0.0,>=7.0
├── python-slugify>=4.0.0
│   └── text-unidecode>=1.3
├── pyyaml>=5.3.1
├── requests>=2.23.0
│   ├── certifi>=2017.4.17
│   ├── charset-normalizer<4,>=2
│   ├── idna<4,>=2.5
│   └── urllib3<3,>=1.21.1
└── rich
    ├── markdown-it-py>=2.2.0
    │   └── mdurl~=0.1
    └── pygments<3.0.0,>=2.13.0
gitpython
└── gitdb<5,>=4.0.1
    └── smmap<6,>=3.0.1
pre-commit-hooks
├── ruamel.yaml>=0.15
│   └── ruamel.yaml.clib>=0.2.7
└── tomli>=1.1.0
build
├── importlib-metadata>=4.6
│   └── zipp>=0.5
├── packaging>=19.0
├── pyproject_hooks
│   └── tomli>=1.1.0
└── tomli>=1.1.0
rope
└── pytoolconfig[global]>=1.2.2
    ├── packaging>=23.2
    ├── platformdirs>=3.11.0
    └── tomli>=2.0.1

Proposed solution

The solution I propose is making kedro a meta-package, that would depend on smaller packages.

Specifically, we could start with 2: kedro-catalog and kedro-new. Why?

The case for splitting kedro-catalog

From the links above:

Would it make sense to make mini-kedro installable? My use case for projects like that are users doing EDA and just want easy access to the data with no fuss.

2741

Some months back I experimented with using the kedro datacatalog as a dataloader for streamlit dashboards.

https://github.com/kedro-org/kedro/issues/2898#issuecomment-1736000094

For our use case, only the DataCatalog part is relevant - nothing about the nodes, pipelines or orchestration. So I'm thinking this might help us avoid additional dependencies.

For some reason, some clients also don't like and/or accept Kedro as well 🤔

https://linen-slack.kedro.org/t/16593946/is-there-a-way-of-installing-only-the-data-catalog-part-of-k#b6d532c4-2d7f-4add-b0ee-b0bfcffbdd5e

The case for splitting kedro-new

kedro new is a weird command for at least 2 reasons:

From the kedro-boot docs:

from typing import List
from fastapi import FastAPI, Query
from fastapi.responses import RedirectResponse
from pydantic import BaseModel

from kedro_boot.app.fastapi.session import KedroFastApi

app = FastAPI(title="Spaceflights shuttle price prediction")

...

@app.post("/predict", tags=["Inference"], operation_id="inference")
def predictions(
    features_store: ShuttleFeature, kedro_run: KedroFastApi
) -> ShuttlePrediction:
    return {"shuttles_prices": kedro_run}

Why should an app like this carry all the weight of unneeded commands?

So I have done venv first, then install kedro, then kedro new, then pip install requirements. Which is a bit confusing for python newbies

Future developments

If we are happy with the initial iterations we could take this idea further and make more smaller packages. Something like:

kedro
├── kedro-pipelines  # Pipelines and nodes
├── kedro-cli  # All things CLI
├── kedro-new
│   └── kedro-cli  # Obviously `kedro new` would depend on `kedro-cli`, plus other things
├── kedro-io  # The catalog
├── kedro-config  # The config loader
├── kedro-framework  # Session, context, hooks, Kedro "classic way of working"
│   ├── kedro-config
│   ├── kedro-pipelines
│   └── kedro-io
└── kedro-micropackaging
   └── kedro-cli  # Everything that extends the CLI depends on `kedro-cli`

Advantages

Considerations

Implementation idea We could even retain imports by leveraging PEP 420 implicit namespace packages https://packaging.python.org/en/latest/guides/packaging-namespace-packages/

Alternative solutions

One alternative solution is to reject the metapackage approach, tell users to keep installing the full kedro, and continue gradually reducing the number of dependencies. The disadvantages are that

  1. There's so much one can do to trim dependencies. Ultimately Kedro engineers want to, well, leverage open source to be more productive.
  2. Ultimately pip install kedro will keep containing parts some users will not want, hence not addressing the core issue.

[^1]: Although quite honestly, #1807 and https://github.com/pallets/click/issues/2249 made me question some Click design choices, and https://github.com/kedro-org/kedro-plugins/pull/552 made me think that definitely I'd love to find a modern alternative.

sbrugman commented 8 months ago

Thanks for the writeup! Addressing this was also on my list. Not being able to prevent Kedro from installing unnecessary dependencies can be quite painful (e.g. when a feature is not used locally, or for deployment).

Would be interesting to be pragmatic and compare this proposal (elegant) to feature-gating functionality as is being done with jupyter. The biggest con in my opinion is the lack of backwards compatibility in pip install kedro. But is that really such a big problem if the fix is to change to pip install kedro[quickstart]? (or any other extra name) In the technical dimension I agree that the above approach is superior, but what about other dimensions (time and effort required to deliver this feature, cost of maintaining these packages etc.)

astrojuanlu commented 8 months ago

Thanks for the validation @sbrugman !

The biggest con in my opinion is the lack of backwards compatibility in pip install kedro.

On the contrary, my idea is that this is completely backwards compatible:

Regular Kedro users: pip install kedro. Absolutely nothing changes. We could even retain imports by leveraging PEP 420 implicit namespace packages

Does that address your concerns?

In the technical dimension I agree that the above approach is superior, but what about other dimensions (time and effort required to deliver this feature, cost of maintaining these packages etc.)

Yep that's something to be discussed and evaluated.

sbrugman commented 8 months ago

@astrojuanlu Sorry for the misunderstanding. "the biggest con ..." was referring to the alternative solution of using pyproject.toml extra's for feature gating.

astrojuanlu commented 8 months ago

Oh sorry I didn't read your sentence:

Would be interesting to be pragmatic and compare this proposal (elegant) to feature-gating functionality as is being done with jupyter.

datajoely commented 7 months ago

The other point here is that as much as we'd like everyone to upgrade to 0.19.x ASAP, that doesn't work in practice and in many cases people like to keep production deployments static.

The situation we find ourselves in today is that lots of people still pull 0.18.x and that now has outdated dependencies which (1) crop up on enterprise scanning tools like Sonarqube (2) We don't retroactively patch.

The great unbundling of Kedro will allow us to be much more dynamic and potentially support individual components better and for longer.

idanov commented 7 months ago

Kedro is a rather small framework by all standards, e.g. running cloc . yields that there's only 23k lines of Python code (including tests). Breaking it further down into smaller micro-packages (because if we go with the proposed split into 7 subpackages, each of them will be in the order of 3k lines with tests) will result into too much overhead for too little functionality for each package separately. I would certainly be in favour in trying to reduce further the dependencies or completely removing features like micropackaging (or factoring them out as a separate plugin), but breaking it down into a constellation of subpackages will only make things way more difficult to coordinate in terms of versions.

Most issues with dependencies as always have been coming from the datasets and now that is largely solved by splitting them out of Kedro as a separate package. And just to illustrate that, before the move, our Snyk score was in the low 80s, check our Snyk score now: https://snyk.io/advisor/python/kedro (on par with Prefect and MLFlow and better than ZenML).

merelcht commented 7 months ago

I wanted to link this conversation https://github.com/kedro-org/kedro/issues/1758 where we considered turning Kedro into a meta-package when separating out kedro-datasets. Long story short: we decided against it because of the engineering overhead of managing the separate packages and dependencies/release flows etc.

I personally still feel that the meta package structure introduces more burden than it offers benefits to our users. I think the points brought up are very valid, but they all seem to be coming from a very specific user groups: those that have developed Kedro plugins and/or are very advanced users that use Kedro as a library. From my understanding, this is a vocal but also a minority group of our users and I would like to hear other perspectives before betting on splitting Kedro into sub components.

I also hesitate to commit to an overhaul like this, because it would be a significant effort that doesn't introduce any new functionality to Kedro. It would just be a restructure of what we have and I question whether that's enough to gain new users and remain relevant as a tool.

astrojuanlu commented 7 months ago

I wanted to link this conversation https://github.com/kedro-org/kedro/issues/1758 where we considered turning Kedro into a meta-package when separating out kedro-datasets.

Thanks a lot for this, will have a look.

From my understanding, this is a vocal but also a minority group of our users and I would like to hear other perspectives before betting on splitting Kedro into sub components.

I'm not so sure about that. There might be some survivorship bias at play here - IOW, people who see Kedro and are discouraged by its approach to dependencies but never complain to us. By its own nature, this would be very difficult to detect. We need a leap of imagination.

Note that this is not an all-or-nothing approach. We can start by spinning off kedro-catalog, the one that gets requested the most.

Splitting this into 7 packages is a proposed solution, but it's not necessarily the only one. I feel the discussion is too centered on that proposed solution rather than the user problem, which I argue has plenty of supporting evidence.

astrojuanlu commented 7 months ago

@datajoely gave some feedback and I reworked the top comment a bit. Changes:

astrojuanlu commented 7 months ago

We introduced this topic in yesterday's Tech Design session: I gave some views on the broader context, described the user problem, and collected some feedback.

Summary

Some ideas were proposed, broadly falling under 3 categories:

Some concerns were raised:

Key concerns

Kedro is not heavy

I presented abundant qualitative evidence in https://github.com/kedro-org/kedro/issues/3659#issue-2158346418 that our users perceive Kedro as heavyweight. Some of those comments are old, but have been unaddressed. Some other comments are recent.

📣 We don't get to decide how our users perceive our product 📣. We can only influence their perception.[^1]

[^1]: This applies to "Kedro is not an orchestrator" too.

I also presented quantitative evidence that makes it crystal clear that Kedro is too heavyweight in terms of dependencies. I purposedly did not discuss about SLOC (Single Lines Of Code) or bytesize because those are uninteresting metrics.

Re-stating the quantitative evidence here:

❯ pipdeptree -p django
Django==5.0.4
├── asgiref [required: >=3.7.0,<4, installed: 3.8.1]
└── sqlparse [required: >=0.3.1, installed: 0.4.4]
❯ pipdeptree -p fastapi
fastapi==0.110.1
├── pydantic [required: >=1.7.4,<3.0.0,!=2.1.0,!=2.0.1,!=2.0.0,!=1.8.1,!=1.8, installed: 2.6.4]
│   ├── annotated-types [required: >=0.4.0, installed: 0.6.0]
│   ├── pydantic_core [required: ==2.16.3, installed: 2.16.3]
│   │   └── typing_extensions [required: >=4.6.0,!=4.7.0, installed: 4.11.0]
│   └── typing_extensions [required: >=4.6.1, installed: 4.11.0]
├── starlette [required: >=0.37.2,<0.38.0, installed: 0.37.2]
│   └── anyio [required: >=3.4.0,<5, installed: 4.3.0]
│       ├── idna [required: >=2.8, installed: 3.6]
│       └── sniffio [required: >=1.1, installed: 1.3.1]
└── typing_extensions [required: >=4.8.0, installed: 4.11.0]
`pipdeptree -p kedro` ``` kedro==0.19.3 ├── attrs [required: >=21.3, installed: 23.2.0] ├── build [required: >=0.7.0, installed: 1.2.1] │ ├── packaging [required: >=19.1, installed: 24.0] │ └── pyproject_hooks [required: Any, installed: 1.0.0] ├── cachetools [required: >=4.1, installed: 5.3.3] ├── click [required: >=4.0, installed: 8.1.7] ├── cookiecutter [required: >=2.1.1,<3.0, installed: 2.6.0] │ ├── arrow [required: Any, installed: 1.3.0] │ │ ├── python-dateutil [required: >=2.7.0, installed: 2.9.0.post0] │ │ │ └── six [required: >=1.5, installed: 1.16.0] │ │ └── types-python-dateutil [required: >=2.8.10, installed: 2.9.0.20240316] │ ├── binaryornot [required: >=0.4.4, installed: 0.4.4] │ │ └── chardet [required: >=3.0.2, installed: 5.2.0] │ ├── click [required: >=7.0,<9.0.0, installed: 8.1.7] │ ├── Jinja2 [required: >=2.7,<4.0.0, installed: 3.1.3] │ │ └── MarkupSafe [required: >=2.0, installed: 2.1.5] │ ├── python-slugify [required: >=4.0.0, installed: 8.0.4] │ │ └── text-unidecode [required: >=1.3, installed: 1.3] │ ├── PyYAML [required: >=5.3.1, installed: 6.0.1] │ ├── requests [required: >=2.23.0, installed: 2.31.0] │ │ ├── certifi [required: >=2017.4.17, installed: 2024.2.2] │ │ ├── charset-normalizer [required: >=2,<4, installed: 3.3.2] │ │ ├── idna [required: >=2.5,<4, installed: 3.6] │ │ └── urllib3 [required: >=1.21.1,<3, installed: 2.2.1] │ └── rich [required: Any, installed: 13.7.1] │ ├── markdown-it-py [required: >=2.2.0, installed: 3.0.0] │ │ └── mdurl [required: ~=0.1, installed: 0.1.2] │ └── Pygments [required: >=2.13.0,<3.0.0, installed: 2.17.2] ├── dynaconf [required: >=3.1.2,<4.0, installed: 3.2.5] ├── fsspec [required: >=2021.4, installed: 2024.3.1] ├── GitPython [required: >=3.0, installed: 3.1.43] │ └── gitdb [required: >=4.0.1,<5, installed: 4.0.11] │ └── smmap [required: >=3.0.1,<6, installed: 5.0.1] ├── importlib_metadata [required: >=3.6,<8.0, installed: 7.1.0] │ └── zipp [required: >=0.5, installed: 3.18.1] ├── importlib_resources [required: >=1.3,<7.0, installed: 6.4.0] ├── jmespath [required: >=0.9.5, installed: 1.0.1] ├── more-itertools [required: >=8.14.0, installed: 10.2.0] ├── omegaconf [required: >=2.1.1, installed: 2.3.0] │ ├── antlr4-python3-runtime [required: ==4.9.*, installed: 4.9.3] │ └── PyYAML [required: >=5.1.0, installed: 6.0.1] ├── parse [required: >=1.19.0, installed: 1.20.1] ├── pluggy [required: >=1.0,<1.4.0, installed: 1.3.0] ├── pre-commit-hooks [required: Any, installed: 4.6.0] │ └── ruamel.yaml [required: >=0.15, installed: 0.18.6] │ └── ruamel.yaml.clib [required: >=0.2.7, installed: 0.2.8] ├── PyYAML [required: >=4.2,<7.0, installed: 6.0.1] ├── rich [required: >=12.0,<14.0, installed: 13.7.1] │ ├── markdown-it-py [required: >=2.2.0, installed: 3.0.0] │ │ └── mdurl [required: ~=0.1, installed: 0.1.2] │ └── Pygments [required: >=2.13.0,<3.0.0, installed: 2.17.2] ├── rope [required: >=0.21,<2.0, installed: 1.13.0] │ └── pytoolconfig [required: >=1.2.2, installed: 1.3.1] │ └── packaging [required: >=23.2, installed: 24.0] ├── toml [required: >=0.10.0, installed: 0.10.2] └── toposort [required: >=1.5, installed: 1.10] ```

Users can ignore what they don't want

I have presented abundant qualitative evidence in https://github.com/kedro-org/kedro/issues/3659#issue-2158346418 that this is not how people vet their open source dependencies.

There is also quantitative evidence that our monolithic approach to dependency management causes extensive user pains. For example: #1807, #1752, #1733, #681, and large parts of #3094, just to name a few.

Kedro is not a monopoly, there are plenty of adjacent open source frameworks whose functionality intersects or directly competes with Kedro. If we force users to make decisions against their will, they might as well use something else.

We have already done enough

We have already done very important things. Spinning off kedro-datasets was crucial, a massive achievement, one that our users celebrate, and something we should be proud of.

But, ironically, kedro-datasets is its own monolith now, and we are struggling to cope with the maintenance cost, see discussions and linked issues in https://github.com/kedro-org/kedro-plugins/issues/535.

This is difficult to maintain

It's easy to grasp the idea that more repositories or more subprojects in monorepos somewhat increases the complexity.

There's two sources of complexity:

My proposal to move forward with this is:

datajoely commented 7 months ago

For me there are key really critical parts of this, I've ranked them in 'unbundling' effort/impact

  1. `Cookiecutter feels like the best candidate for unbundling, it is used in two ways:

    • kedro new / kedro starter workflows, this happens 1 per project and by that measure <1 time per user!
    • kedro pipeline create this can happen many times per project and can even point to a custom template

    The question I have for engineering is as follows - could we use something like pipx to include a dependency like this when it has no impact on Kedro's wider API

ChatGPT generated the following and I (read possible dum dum) think it looks neat:

import subprocess

def create_project_from_template(template_url, project_name):
    subprocess.run(["pipx", "install", "cookiecutter"])  # Install Cookiecutter if not already installed
    subprocess.run(["cookiecutter", template_url, "--output-dir", ".", "--no-input", "-f", "-o", project_name])

# Example usage
create_project_from_template("https://github.com/audreyr/cookiecutter-pypackage.git", "my_project")

  1. The data catalog would be useful on it's own, this is user validated and I'm excited to see where @iamelijahko's research lands in terms of how it could improve as a standalone component.

  2. It would be great to remove click in favour of argparse, it's used everywhere and we have "click>=4.0" set. My hunch is that the long term user value versus effort calculation doesn't make this a compelling option.

idanov commented 7 months ago

@astrojuanlu Thanks for the writeup and the tech design session from last week. Let me comment on some of the arguments you brought forward in favour of splitting Kedro into smaller packages.

Kedro is (not) heavy

I presented abundant qualitative evidence in https://github.com/kedro-org/kedro/issues/3659#issue-2158346418 that our users perceive Kedro as heavyweight. Some of those comments are old, but have been unaddressed. Some other comments are recent.

I followed all links and I am still unconvinced by the evidence, a lot of it has subjective opinions on what we should do. What I'd rather see is what actual problem we are causing by not doing the suggested actions.

I would really love to have a compilation of "whys" in order to have a grounded conversations in facts and not opinions.

📣 We don't get to decide how our users perceive our product 📣. We can only influence their perception.

Yes, and that's a communication challenge mostly, which we still haven't addressed. It will not be solved by splitting or not splitting Kedro, but with proper comms. Including the orchestrator confusion. If we self-assign a label to Kedro heavyweight, imo fully undeserved, then we only add to the problem, not solving it.

I also presented quantitative evidence that makes it crystal clear that Kedro is too heavyweight in terms of dependencies. I purposedly did not discuss about SLOC (Single Lines Of Code) or bytesize because those are uninteresting metrics.

It is not crystal clear, to me at least. Why are those uninteresting? This is quite an important metric for determining if something is heavyweight. When saying that "kedro is a huge monolith" and "heavyweight", we need to define many words here, like huge, monolith and heavyweight. Because, I, for one, certainly measure the size of software projects in terms of how many lines of code they are, having worked on projects from a couple of hundred lines to a couple of hundred thousand lines and knowing the difference in complexity of either extremes. By that measure, Kedro is tiny, not huge. Here's a table to illustrate that:

Package Python LOC Python LOC w/o tests Direct Dependencies Size on Disk (+ deps)
kedro 24K 7K 23 36MB
django 351K 107K 3 37MB
fastapi 87K 10K 3 11MB
mlflow 190K 76K 27 522MB
prefect 212K 92K 54 170MB
zenml 144K 100K 22 285MB
metaflow 54K 48K 2 34MB
dbt-core 83K 32K 23 90MB

As you can see, Kedro is definitely not an outlier from comparable frameworks, and I'd argue its the most lightweight one (bar FastAPI, which is quite impressively small btw).

The only valid discussion arising from this table (and your research) is if it has too many direct dependencies, and the answer probably is maybe and certainly not a definitive yes. However this issue can very easily be solved without breaking Kedro up into tiny packages, but carefully examining what we really need as mandatory dependency and what can be optional. Here's just my list of dependencies that can either be removed or made optional easily:

That's already 11 removed, which will make us second only to metaflow from the ML ecosystem packages in this list. Nothing suggesting Kedro is huge or monolith, but rather the opposite in my opinion - in fact, Kedro seems to be more similar to FastAPI than Django by almost all measures of huge-ness.

Users can(not) ignore what they don't want

I have presented abundant qualitative evidence that this is not how people vet their open source dependencies.

As I've pointed, the evidence you've shown only beats around the bush of the actual problem, not revealing the problem itself. We need a double-click on all of the evidence you have provided and this is what we are lacking, although we can all suppose that the problem people face is indeed the list of direct dependencies, but that'd be an educated guess, not a fact.

There is also quantitative evidence that our monolithic approach to dependency management causes extensive user pains. For example: #1807, #1752, #1733, #681, and large parts of #3094, just to name a few.

I don't see the connection between those issues to dependency management. There's a lot about CLI (rich concretely) and logging, some research on deployment, but nothing suggesting that the dependency management is the culprit for all of those. I would try to tackle one problem at a time and not conflate all issues at once with the silver bullet of splitting.

If we choose to force our users to ignore the parts that they don't want, we are going against well established software engineering principles and also decades of collectively learned behaviors.

I am unsure what you mean by that. How can you force someone to ignore something? A package or module in Python is unused if not imported. I am pulling a number out of thin air here, but I won't be surprised that most packages have no more than 20% of their API being utilised by any given application. Does that mean that each package forces the users to ignore the other 80% of their API footprint? People have agency over importing modules, we can't automagically force them to import the Pipeline class if they don't want to use it.

Kedro is not a monopoly, there are plenty of adjacent open source frameworks whose functionality intersects or directly competes with Kedro, and as such if we force users to make decisions against their will, they might as well use something else.

Sure it isn't, but this comment is not very constructive. By looking at the table above - comparable tools don't seem to be too bothered by being huge and monolith either...

📣 We don't get to force our users to do what they don't want to do 📣.

True, but we are not really forcing them anything. Our users can make a good judgement of the trade-offs of different packages, including Kedro. We can only try to convince them that Kedro is suitable for their use case, but that happens through a good community with lots of examples and education materials. And, obviously, try to improve their experience where we can, but first we need to fully understand the failure modes.

We have (not) already done enough

But, ironically, kedro-datasets is its own monolith now! And discussions around that are becoming more dense and difficult as well, see https://github.com/kedro-org/kedro-plugins/pull/560, https://github.com/kedro-org/kedro-plugins/issues/535.

As far as I am aware, you can install whatever you want fairly independently: https://github.com/kedro-org/kedro-plugins/blob/main/kedro-datasets/pyproject.toml Sure, you can scatter all of those datasets into one package per class, but what's the point in that and is the multi-package setup worth it? What is the difference between pip install kedro-datasets[pandas] vs pip install kedro-pandas? How does that improve discoverability? As for the issue where someone installed all kedro datasets at once, I think this is rather a development workflow issues than anything else. Maybe we should have better contribution guide on how not to install everything, when you are modifying only one group of kedro datasets.

Before, kedro was a huge monolith. Now we have two less huge monoliths: kedro and kedro-datasets.

Again, less huge monolith of 7K lines of Python code doesn't sit right to me. I suspect we have different definitions of huge and monolith. A monolith in software engineering is not defined only by its dependencies, so the usage of those words from where I stand is incorrect and causes confusion and unnecessary disagreement, apart from leading us towards potentially unsuitable solutions.

Finally, I don't recall anyone claiming we've done enough. Absolutely, we can do more! We should always look out for some small improvements that amount to a bigger change at the end. Such small improvement is removing unneeded dependencies.

This is difficult to maintain

It's easy to grasp the idea that more repositories or more subprojects in monorepos somewhat increases the complexity.

There's two sources of complexity:

  • Design. How do we spin off components while avoiding circular dependencies?
  • Maintenance. How do we ensure juggling more repos or more subprojects doesn't consume an excessive amount of engineering resources?

Even with the best design and great automation, over the years entropy is increasing, team members rotate and forget, make automated things less automated and so on. Coordinating the releases of 2 things is easier than coordinating the releases of 10 things. Currently the team experiences only a taster of that by releasing kedro, kedro-starters, kedro-datasets, kedro-telemetry, kedro-airflow, kedro-viz and yet, we quite often have some issues with some components not working well together because we forgot to bump a version or something else breaking one of the components (most recent one is release 0.19.1, but there was one with kedro-viz depending on toposort which is to be dropped, occasionally some kedro starter not working, etc, etc).

And that's not surprising, the probability of something going wrong is $1 - P(one release failing)^{number of packages}$. It takes only 15 packages with 5% error rate to get to a point that we need to release hotfixes on every other release. 5% error rate is 1 in 20 releases - you be the judge what our release error rate is currently and if 5% is representative. Bringing this error rate down is not trivial and it's a lot of work and many integration tests for n inter-dependent components ($tests = \frac{n * (n - 1)}{2}$), which is high burden no matter what. We can decrease this number a lot through the topology of the dependencies, but it will still be a big number of tests.

Let's agree to include this in the equation when deciding if splitting is the right solution for the problem, when we find an agreement on what the actual problem is.

As a conclusion, let's start with the problem and then define a solution and not the other way around. So in that spirit, what is preventing users from installing Kedro as is and use only the DataCatalog for example?

astrojuanlu commented 6 months ago

TIL: MLFlow publishes a separate package mlflow-skinny with fewer dependencies https://github.com/mlflow/mlflow/blob/master/README_SKINNY.rst, https://github.com/mlflow/mlflow/blob/master/pyproject.skinny.toml#L24-L36

astrojuanlu commented 5 months ago

From a related discussion at FastAPI https://github.com/tiangolo/fastapi/discussions/11525#discussioncomment-9326151

in my humble opinion cli is not the thing that is needed for web server.

Top upvoted comment in the thread.

astrojuanlu commented 5 months ago

Sequence of events in FastAPI:

And yet, Kedro is even bigger than the new big FastAPI:

Django fastapi-slim FastAPI (full) Kedro
image image image image

About

in fact, Kedro seems to be more similar to FastAPI than Django by almost all measures of huge-ness.

@idanov I don't know how to say this in a way that it doesn't sound bad or confrontational, but this was wrong in April 14th, and is still wrong today. Everyone is entitled to their opinion but I'd rather not mix opinions with facts in this way.

I feel that the push to conduct a full fledged research stream on why developers don't want fat dependencies is just delaying the inevitable.

In any case, #3884 is already exploring options to make cookiecutter optional, next is rich #2928, and probably click will be next before we can meaningfully tackle #143. Let's continue this conversation in the places in which we're taking action.

datajoely commented 5 months ago

With a purely 80/20 hat on:

Big wins:

Stuff we can easily kill:

Already on chopping block: