Closed netsettler closed 10 months ago
Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | ||
---|---|---|---|---|---|
dcicutils/misc_utils.py | 37 | 40 | 92.5% | ||
dcicutils/validation_utils.py | 110 | 122 | 90.16% | ||
dcicutils/sheet_utils.py | 350 | 391 | 89.51% | ||
dcicutils/bundle_utils.py | 351 | 396 | 88.64% | ||
<!-- | Total: | 866 | 967 | 89.56% | --> |
Files with Coverage Reduction | New Missed Lines | % | ||
---|---|---|---|---|
dcicutils/lang_utils.py | 2 | 97.83% | ||
<!-- | Total: | 2 | --> |
Totals | |
---|---|
Change from base Build 6422359319: | 0.9% |
Covered Lines: | 9332 |
Relevant Lines: | 11875 |
New module
bundle_utils.py
that is intended for schema-respecting worksheets ("metadata bundle"). There are various modular bits of functionality here, but the main entry point here is:load_items
to load data from a given table set, doing certain notational canonicalizations, and checking that things are in the appropriate format.In
common.py
, new hint types:CsvReader
JsonSchema
Regexp
In
lang_utils.py
:New arguments
just_are=
tothere_are
get verb conjugation without the details.Add "while" to "which" and "that" as clause handlers in the string pluralizer (e.g., so that "error while parsing x" pluralizes as "errors while parsing x")
In
misc_utils.py
, miscellaneous new functionality:New class
AbstractVirtualApp
that is either an actual VirtualApp or can be used to make mocks if the thing being called expects anAbstractVirtualApp
instead of aVirtualApp
.New function
to_snake_case
that assumes its argument is either a CamelCase string or snake_case string, and returns the snake_case form.New function
is_uuid
(migrated from Fourfront)New function
pad_to
New class
JsonLinesReader
In
qa_checkers.py
:VERSION_IS_BETA_PATTERN
to recognize alpha or beta patterns. Probably a rename would be better, but also incompatible. As far as I know, this is used only to not fuss if you haven't made a changelog entry for a beta (or now also alpha).New module
sheet_utils.py
for loading workbooks in a variety of formats, but without schema interpretation.A lot of this is implementation classes for each of the kinds of files, but the main entry point is intended to be
load_table_set
if you are not working with schemas. For schema-related support, seebundle_utils.py
.New module
validation_utils.py
with these facilities:New class
SchemaManager
for managing a set of schemas so that programs asking for a schema by name only download one time and then use a cache. There are also facilities here for populating a dictionary with all schemas in a table set (the kind of thing returned byload_table_set
insheet_utils.py
) in order to pre-process it as a metadata bundle for checking purposes.New functions:
validate_data_against_schemas
to validate that table sets (workbooks, or the equivalent) have rows in each tab conforming to the schema for that tab.summary_of_data_validation_errors
to summarize the errors obtained fromvalidate_data_against_schemas
.