Data dictionary verification

[lewisjared]

The problem

We now have some metadata in about the expected shape of the data in bookshelf, but we do not verify that a dataset conforms with this.

We can start off pretty simple with the following checks:

• [ ] Are all the columns in the data dictionary
• [ ] Is the type of a column correct
• [ ] If a CV is present, are there any values not in the CV
• [ ] If a column is required are all values are not empty
• [ ] The data dictionary model is valid and complete (https://docs.pydantic.dev/latest/api/base_model/#pydantic.main.BaseModel.model_validate)

Definition of "done"

• [ ] Add function verify_data_dictionary
• [ ] Tests for each of the checks
• [ ] Update test_notebook test to call verify_data_dictionary after running a notebook (We might need to skip and warn on books that don't have a data dictionary until we have full coverage)

Additional context

Related to #21

[lewisjared]

In GitLab by @mikapfl on Jan 10, 2024, 19:52

Maybe it is worth building some of this functionality on https://github.com/great-expectations/great_expectations . It already has a large collection of possible checks. But maybe also not exactly what we need.

[lewisjared]

In GitLab by @yinilaii on Jan 25, 2024, 12:58

created branch 34-data-dictionary-verification to address this issue

[lewisjared]

unassigned @yinilaii