All Python functions are annotated

marcenacp commented 1 year ago

Why?

This feature can have several benefits:

This will speed up pytype.
This will make typing less loose.

How?

Install pip install monkeytype, but do not add it to pyproject.toml. We'll use monkeytype to just generate types.
Install pip install flake8-annotations and add it to pyproject.toml in the [dev] section.
In the Makefile, change pyflakes . --disallow-untyped-defs.

Run MonkeyType to generate the database:

monkeytype run scripts/load.py --file ../../datasets/huggingface-mnist/metadata.json --record_set=default

List all ml_croissant modules:

sqlite3 monkeytype.sqlite3 "SELECT DISTINCT(module) FROM monkeytype_call_traces WHERE module LIKE '%ml_croissant%';"

For all modules, apply: monkeytype apply ml_croissant._src.core.json_ld.
Fix other typing problems manually.

sheyaln commented 1 year ago

Will this need to be assigned to an individual, or can I take a stab at it and make a PR?

marcenacp commented 1 year ago

@Milksheyke Please, feel free to send a PR :) Thanks!

sheyaln commented 1 year ago

Installed repo and dev tools, but I'm unable to generate the database with monkeytype. At least not fully.

croissant/python/mlcroissant/mlcroissant/_src/operation_graph/execute.py", line 98, in build_record_set
    len(result) == 1
AssertionError: "GroupRecordSet(default)" should have one and only one predecessor. Got: 0.

execute_operations_in_streaming in mlcroissant/_src/operation_graph/execute.py is passing in a DiGraph while build_record_set is expecting a MultiDiGraph. I'm guessing that's where the assertion error is coming from.

Am I missing something here, or should we introduce some sort of default behavior to handle if a node lacks a predecessor?

mlcommons / croissant

All Python functions are annotated #208