Closed noklam closed 1 year ago
Hi @noklam, I think there are several sub tasks to this ticket but not all with the same priority.
DataFrameModel
it will be enough for the hook to work "as is" with the exact same syntax. Something like : my_data:
type: ...
filepath: ...
metadata:
pandera:
schema: ${pa.python: my_kedro_package.schemas.my_data.MyDataSchema} # we should "just" create the resovler which will import and instantiate the class
Does the design look ok for you? Do you have time to work on this one?
my_kedro_package.schemas.my_data.py
file (pseudo code below): # my_kedro_package.schemas.my_data.py
from pandera import DataframeModel
from pandera.io import Field
class MyDataSchema(DataframeModel):
var1: str = Field()
var2: <var_typ>e= Field()
...
How does 2. different from the current infer CLI? I'll work on 1.
The current CLI has a flag --python
for this but it is not implemented. The little difference is that the infer
CLI for yaml use a built-in pandera function which creates basic tests and the file, but there is no such function for python so we should create it on our own. that is why I want to keep it really simple, I want to avoid creating boilerplate code, not really infering advanced tests.
The current CLI has a flag --python for this but it is not implemented I am not sure what do you mean, I thought this function exist already? isn't it using the
schema.to_script()
method?
Pandera natively support convert DataFrameModel -> DataFrameSchema, but not the other way round.
Description
In addition to the YAML API, we should support the class-base API
DataFrameModel
(pydantic)Context
TBD
Possible Implementation
TBD
Possible Alternatives
TBD