kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
9.53k stars 879 forks source link

`kedro pipeline create` create `tests` folder inside `src` #3711

Closed bpmeek closed 4 months ago

bpmeek commented 4 months ago

Description

The test directory created by a new starter and the test directory created by a new pipeline don't match.

Context

Having multiple test locations can be confusing and difficult to manage.

Steps to Reproduce

  1. Create a new project with cli using kedro new
  2. Select tool 2 (Test) when asked for Project Tools
  3. Create a new pipeline with cli using `kedro pipeline create

Expected Result

Creating a new project creates <repo_name>/tests directory. Creating a new pipeline creates the following:

Actual Result

Creating a new project creates <repo_name>/tests directory. Creating a new pipeline creates the following:

-- If you received an error, place it here.
-- Separate them if you have more than one.

Your Environment

noklam commented 4 months ago

@bpmeek Can you provide some command /screenshot for us to reproduce? I did this just now and work as expected. I think you were talking test should be created in the same directory of src instead of inside it?

image

bpmeek commented 4 months ago

@noklam this is what I get, but yes, I would expect the tests directory to be in the same directory as src not inside of it. You can see below kedro pipeline create dummy_pipeline creates /Users/brandonmeek/dummy-project/src/tests/pipelines/dummy_pipeline/test_pipeline.py

(dummy-project) brandonmeek@MacBook-Pro ~ % kedro new --name=dummy-project --tools=test --example=n

Congratulations!
Your project 'dummy-project' has been created in the directory 
/Users/brandonmeek/dummy-project

You have selected the following project tools: ['Testing']
(dummy-project) brandonmeek@MacBook-Pro ~ % tree dummy-project 
dummy-project
├── README.md
├── conf
│   ├── README.md
│   ├── base
│   │   ├── catalog.yml
│   │   └── parameters.yml
│   └── local
│       └── credentials.yml
├── notebooks
├── pyproject.toml
├── requirements.txt
├── src
│   └── dummy_project
│       ├── __init__.py
│       ├── __main__.py
│       ├── pipeline_registry.py
│       ├── pipelines
│       │   └── __init__.py
│       └── settings.py
└── tests
    ├── __init__.py
    ├── pipelines
    │   └── __init__.py
    └── test_run.py

10 directories, 15 files
(dummy-project) brandonmeek@MacBook-Pro ~ % cd dummy-project 
(dummy-project) brandonmeek@MacBook-Pro dummy-project % kedro pipeline create dummy_pipeline
Using pipeline template at: '/Users/brandonmeek/opt/anaconda3/envs/dummy-project/lib/python3.10/site-packages/kedro/templates/pipeline'
Creating the pipeline 'dummy_pipeline': OK
  Location: '/Users/brandonmeek/dummy-project/src/dummy_project/pipelines/dummy_pipeline'
Creating '/Users/brandonmeek/dummy-project/src/tests/pipelines/dummy_pipeline/__init__.py': OK
Creating '/Users/brandonmeek/dummy-project/src/tests/pipelines/dummy_pipeline/test_pipeline.py': OK
Creating '/Users/brandonmeek/dummy-project/conf/base/parameters_dummy_pipeline.yml': OK

Pipeline 'dummy_pipeline' was successfully created.

(dummy-project) brandonmeek@MacBook-Pro dummy-project % tree
.
├── README.md
├── conf
│   ├── README.md
│   ├── base
│   │   ├── catalog.yml
│   │   ├── parameters.yml
│   │   └── parameters_dummy_pipeline.yml
│   └── local
│       └── credentials.yml
├── notebooks
├── pyproject.toml
├── requirements.txt
├── src
│   ├── dummy_project
│   │   ├── __init__.py
│   │   ├── __main__.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-310.pyc
│   │   │   └── settings.cpython-310.pyc
│   │   ├── pipeline_registry.py
│   │   ├── pipelines
│   │   │   ├── __init__.py
│   │   │   └── dummy_pipeline
│   │   │       ├── __init__.py
│   │   │       ├── nodes.py
│   │   │       └── pipeline.py
│   │   └── settings.py
│   └── tests
│       └── pipelines
│           └── dummy_pipeline
│               ├── __init__.py
│               └── test_pipeline.py
└── tests
    ├── __init__.py
    ├── pipelines
    │   └── __init__.py
    └── test_run.py

15 directories, 23 files
noklam commented 4 months ago

@bpmeek oh, I didn't get it correctly in the beginning, so it's the kedro pipeline create producing the unexpected test folder

noklam commented 4 months ago

@bpmeek I can confirm this behavior and I have updated the title. I will make sure this is discussed in the team soon.

merelcht commented 4 months ago

I think this is a duplicate of https://github.com/kedro-org/kedro/issues/3614. If so, we'll close this issue and prioritise #3614 for our next sprint.

bpmeek commented 4 months ago

@merelcht it is a duplicate, I must've missed that ticket when I was looking.

Thank you!