Better sheet utils

[netsettler]

• New module bundle_utils.py that is intended for schema-respecting worksheets ("metadata bundle"). There are various modular bits of functionality here, but the main entry point here is:

  • load_items to load data from a given table set, doing certain notational canonicalizations, and checking that things are in the appropriate format.

• In common.py, new hint types:

  • CsvReader
  • JsonSchema
  • Regexp

• In lang_utils.py:

  • New arguments just_are= to there_are get verb conjugation without the details.

  • Add "while" to "which" and "that" as clause handlers in the string pluralizer (e.g., so that "error while parsing x" pluralizes as "errors while parsing x")

• In misc_utils.py, miscellaneous new functionality:

  • New class AbstractVirtualApp that is either an actual VirtualApp or can be used to make mocks if the thing being called expects an AbstractVirtualApp instead of a VirtualApp.

  • New function to_snake_case that assumes its argument is either a CamelCase string or snake_case string, and returns the snake_case form.

  • New function is_uuid (migrated from Fourfront)

  • New function pad_to

  • New class JsonLinesReader

• In qa_checkers.py:

  • Change the VERSION_IS_BETA_PATTERN to recognize alpha or beta patterns. Probably a rename would be better, but also incompatible. As far as I know, this is used only to not fuss if you haven't made a changelog entry for a beta (or now also alpha).

• New module sheet_utils.py for loading workbooks in a variety of formats, but without schema interpretation.

  A lot of this is implementation classes for each of the kinds of files, but the main entry point is intended to be load_table_set if you are not working with schemas. For schema-related support, see bundle_utils.py.

• New module validation_utils.py with these facilities:

  • New class SchemaManager for managing a set of schemas so that programs asking for a schema by name only download one time and then use a cache. There are also facilities here for populating a dictionary with all schemas in a table set (the kind of thing returned by load_table_set in sheet_utils.py) in order to pre-process it as a metadata bundle for checking purposes.

  • New functions:

  • validate_data_against_schemas to validate that table sets (workbooks, or the equivalent) have rows in each tab conforming to the schema for that tab.

  • summary_of_data_validation_errors to summarize the errors obtained from validate_data_against_schemas.

[coveralls]

Pull Request Test Coverage Report for Build 6710266183

• 866 of 967 (89.56%) changed or added relevant lines in 7 files are covered.
• 2 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.9%) to 78.585%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
dcicutils/misc_utils.py	37	40	92.5%
dcicutils/validation_utils.py	110	122	90.16%
dcicutils/sheet_utils.py	350	391	89.51%
dcicutils/bundle_utils.py	351	396	88.64%
<!--	Total:	866	967	89.56%	-->

Files with Coverage Reduction	New Missed Lines	%
dcicutils/lang_utils.py	2	97.83%
<!--	Total:	2		-->

Totals
Change from base Build 6422359319:	0.9%
Covered Lines:	9332
Relevant Lines:	11875

---

💛 - Coveralls