Enforce Data Frame Schema

[lasuk]

Enforce DataFrame Schema

Scope

• Add function enforce_dtypes() that returns a DataFrame with a guaranteed set of columns with known type.
• Modifies list_files() and list_categories() to return a type consistent DataFrame.

Background

Converting the results of a GET request from '../list.json' to pd.DataFrame returns a reliable DataFrame only when the list is non-empty. If the list is empty, it results in a DataFrame with no columns, which can cause downstream errors.

cc_client = CashCtrlClient()
df = pd.DataFrame(cc_client.get('journal/list.json')['data'])

Refer to the Type Consistency section in the CONTRIBUTING.md for more details.

[lasuk]

Here's a review by ChatGPT:

Click for instructions given to ChatGPT

``` > Review this Python unit and the corresponding unit test. [Uploaded enforce_dtypes.py and test_enforce_dtypes.py] > Provide as downloadable markdown file ```

enforce_dtypes.py Review

Functionality

• Inputs:

  • data: a pandas DataFrame or None
  • required: dictionary mapping column names to their required data types
  • optional: dictionary mapping column names to optional data types
  • keep_extra_columns: boolean to keep or drop extra columns

• Outputs:

  • Returns a DataFrame with enforced data types for specified columns and handles the presence of required columns.

Error Handling

• Throws ValueError if required columns are missing.
• Throws TypeError if data type conversion fails.

Type Conversion Specifics

• Manages timezone conversions for datetime types.
• Uses pandas' astype() for other data types.

Code Structure

• Well-structured with clear separation of functionality.

Documentation and Examples

• Includes comprehensive docstrings with detailed descriptions and examples.

test_enforce_dtypes.py Review

Test Cases

• General Functionality: Tests for empty input, required and optional columns addition, and behavior of keep_extra_columns flag.
• Error Handling: Verifies the handling of missing required columns and invalid data type conversions.
• Datetime Handling: Tests adding datetime columns with and without timezone information, and handling of invalid datetime formats.

Coverage and Robustness

• Comprehensive coverage of scenarios including null inputs, data type enforcement, error handling, and datetime conversions.

Improvement Suggestions

• Additional Edge Cases: Test for mixed data types within columns.
• Performance Testing: Consider testing with large data sets.
• Documentation: Enhance readability with inline comments or descriptive test function names.

[lasuk]

Implements first step in macxred/cashctrl_ledger#2.