Closed Galileo-Galilei closed 1 year ago
Patch coverage: 79.62%
and project coverage change: -11.46%
:warning:
Comparison is base (
a35e978
) 100.00% compared to head (280f088
) 88.54%.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
@datajoely as you can see in the diff (or maybe not, this is becoming far too big) this is what was originally done, but I face several issues:
pytest
commandOmegaConfigLoader
the default, remove tests/
folder to avoid conflicts with pytest...) and it was easier to do it manually rather than programatically (overwriting file, mocking imports...)Tests should likely be refactored in the future to find a better way to do this, but I try to be pragmatic in this PR and keep changes relatively "small" (seems to be a failure 😅 )
Good reasons! I'm happy either-way - only real value in a dynamic/configurable integration test is catching future regressions easier, but that's deffo a premature optimization!
@Galileo-Galilei Is there anything particular you would like me to review/test? I see this still in draft, anything I can help to push this to the finishing line?
Hi @noklam, I think this is almost good to go. I want to write tests and rebase to make commit more readable but it you want I can merge it today without all tests so you can work in parallel without too many merge conflicts to resolve.
I think the main next step is to support DataframeModel
written in python and not only yaml configuration which is very restrictive.
Do you have another feature in mind?
FYI - Our teams that have adopted Pandera internally have leveraged the Python approach on the functions called by the nodes. In this situation Kedro isn't really aware of this, but our plugin could introspect the function bounded to the node
object and bring these into scope.
Additionally - shall we make a little feature backlog, because we can split up the work?
@Galileo-Galilei That's fine. This is all I need, I can start working on that tomorrow, I will branch off from this PR. I don't expect major changes so I can relies on existing structure.
@datajoely is that the @check_types annotation?
@noklam yes exactly, the pattern in use looks something like this:
project/
├─ data_processing/
│ ├─ nodes.py <--- Functions called call schemas via type hints
│ ├─ pipelines.py
├─ data_science/
│ ├─ ...
├─ schemas/
│ ├─ raw_preprocessing.py
│ |─ prm_features.py
Outside of the node definition, where the bound function is defined the signature looks something like this:
import pandera as pa
from pandera.typing import DataFrame
@pa.check_types
def preprocess_data(raw_data: DataFrame,...) -> DataFrame[RawDataSchema]:
...
Now this is independent of the Kedro catalog, but we could tie it back to Kedro world through introspection
Thanks @datajoely for the information about the check_types
decorator. I think this is the "right way" to integrate pandera to standard function (mostly because when you don't have the dataaset/pipeline abstractions, you want to be protected against running the function against invalid data), but I think this is not necessarily what we should encourage to do. Conceptually, schema belongs to datasets and we should keep the decoupling between I/O and compute that kedro offers to have more flexibility (e.g. to turn validation off, to validate data manually and not at runtime, visualize data schema, document the pipeline...)
Additionally - shall we make a little feature backlog, because we can split up the work?
Absolutely agree on this. I suggest to keep what you do in the kedro repo:
At this stage (no users, not compatible with any public kedro version, no public declaration that the plugin exist, 8 stars), I am really comfortable with having a fast development pace, breaking things and iterating so feel free to go and merge without my validation. Havings an issue describing roughly what is at stake is enough for synchronization for now IMHO.
@noklam I've just rebased so it may have messed up your hsitory. It's likely better to start from the main branch, i'll proceed on merging it
I just go on to create some issues first - I will fill that issues up eventually as I also have habit to read/write my issues during my commute without internet.
Agree at this point it's small, let's keep the administrative work in a minimal base.
Description
The goal of this PR is to create a MVP to enable runtime data validation easily in an existing kedro project.
Development notes
The goal is to enable the following user workflow in a kedro project:
Generate a configuration file with the schema of a dataset you want to check at runtime. I did the following changes:
infer_schema
toinfer_dataset_schema
because of import conflicts with pandera functionbase
instead oflocal
because of a bug withOmegaConfigLoader
which cannot resolve the key -> should open an issue in kedro's repo.catalog_{dataset_name}.yaml
with an extra_{dataset_name}
key to enable discovery in the catalogLet catalog datasets accept an extra
metadata.pandera.schema
key which will be resolved as a panderaDataframeSchema
. I did the following changes:pa.dict
andpa.yaml
) which can parse another catalog key and convert it to aDataframeSchema
after_context_created
hook to register these resolvers automaticallyAdd runtime validation each time a session is run. I did the following changes:
before_node_run
which looks for aDataframeSchema
in the dataset (undermetadata.pandera.schema
key ) and validate it against data passed to the node.I also did the following extra actions:
pandas-iris
starterChecklist
CHANGELOG.md
file. Please respect Keep a Changelog guidelines.Notice
[x] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":
I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.
I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorised to submit this contribution on behalf of the original creator(s) or their licensees.
I certify that the use of this contribution as authorised by the Apache 2.0 license does not violate the intellectual property rights of anyone else.