search

rogerbinns / apsw

Another Python SQLite wrapper
https://rogerbinns.github.io/apsw/
Other
733 stars 97 forks source link

Fix syntax error in types stub file #436

Closed cjrh closed 1 year ago

cjrh commented 1 year ago

This fixes a syntax error in the types stub file. This error turns out to be quite hard to deal with because currently mypy has no good way of excluding a specific third party package (like apsw) from the parsing stage. The only way that works is to use the --no-site-packages flag, which excludes all external packages. The --exclude option does not work for syntax errors during parsing, like this one. In a large project, this error is preventing us from running mypy.

This is reproducible in the APSW repo (python 3.11):

$ pip install mypy
$ mypy apsw
apsw/__init__.pyi:350: error: invalid syntax  [syntax]
Found 1 error in 1 file (errors prevented further checking)

The fix seems simple, but I confess it's been a while since I've worked with the C api. The code where the no_change is declared looks like this:

static PyTypeObject apsw_no_change_object = {
    PyVarObject_HEAD_INIT(NULL, 0)
        .tp_name = "apsw.no_change",
    .tp_doc = Apsw_no_change_DOC,
};

It's a PyTypeObject so that would mean that from Python the superclass would be type, right? That's what I have in this PR. This allows mypy to continue with the full run, output shown below. Note that I am not focusing on the specific issues raised by mypy in the APSW project: my goal is to fix the unavoidable syntax error during parsing.

# including this PR
$ mypy apsw
apsw/__init__.pyi:2763: error: Unexpected "..."  [misc]
apsw/__init__.pyi:2763: error: "Sequence" expects 1 type argument, but 2 given  [type-arg]
apsw/__init__.pyi:2975: error: "Sequence" expects 1 type argument, but 2 given  [type-arg]
apsw/shell.py:72: error: Incompatible default for argument "stdin" (default has type "None", argument has type "TextIO")  [assignment]
apsw/shell.py:72: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
apsw/shell.py:72: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
apsw/shell.py:94: error: Need type annotation for "widths" (hint: "widths: List[<type>] = ...")  [var-annotated]
apsw/shell.py:98: error: Need type annotation for "_output_stack" (hint: "_output_stack: List[<type>] = ...")  [var-annotated]
apsw/shell.py:115: error: Need type annotation for "_input_stack" (hint: "_input_stack: List[<type>] = ...")  [var-annotated]
apsw/shell.py:119: error: Need type annotation for "_input_descriptions" (hint: "_input_descriptions: List[<type>] = ...")  [var-annotated]
apsw/shell.py:2972: error: Name "n" is not defined  [name-defined]
apsw/shell.py:2973: error: Name "x" is not defined  [name-defined]
apsw/shell.py:2974: error: Name "v" is not defined  [name-defined]
apsw/ext.py:193: error: Incompatible return value type (got "Union[int, bytes, str, NoneType, float]", expected "Union[None, int, float, bytes, str]")  [return-value]
apsw/ext.py:213: error: Argument 2 to "DictAdapter" has incompatible type "Mapping[str, Union[Union[None, int, float, bytes, str], zeroblob]]"; expected "Mapping[str, Union[None, int, float, bytes, str]]"  [arg-type]
apsw/ext.py:243: error: Too many arguments for "__init__" of "object"  [call-arg]
apsw/ext.py:429: error: "object" has no attribute "__iter__"; maybe "__dir__" or "__str__"? (not iterable)  [attr-defined]
apsw/ext.py:437: error: "object" has no attribute "__iter__"; maybe "__dir__" or "__str__"? (not iterable)  [attr-defined]
apsw/ext.py:441: error: "object" has no attribute "__iter__"; maybe "__dir__" or "__str__"? (not iterable)  [attr-defined]
apsw/ext.py:447: error: "object" has no attribute "update"  [attr-defined]
apsw/ext.py:458: error: Name "Literal" is not defined  [name-defined]
apsw/ext.py:546: error: All conditional function variants must have identical signatures  [misc]
apsw/ext.py:546: note: Original:
apsw/ext.py:546: note:     def colour_wrap(text: str, kind: type, header: Any = ...) -> str
apsw/ext.py:546: note: Redefinition:
apsw/ext.py:546: note:     def colour_wrap(text: Any, *args: Any, **kwargs: Any) -> Any
apsw/ext.py:817: error: Incompatible return value type (got "Tuple[Tuple[str, ...], VTColumnAccess]", expected "Tuple[List[str], VTColumnAccess]")  [return-value]
apsw/ext.py:823: error: Incompatible return value type (got "Tuple[Tuple[Any, ...], VTColumnAccess]", expected "Tuple[List[str], VTColumnAccess]")  [return-value]
apsw/ext.py:825: error: Incompatible return value type (got "Tuple[Tuple[str, ...], VTColumnAccess]", expected "Tuple[List[str], VTColumnAccess]")  [return-value]
apsw/ext.py:935: error: Incompatible types in assignment (expression has type "Tuple[str, ...]", variable has type "Tuple[str]")  [assignment]
apsw/ext.py:961: error: Incompatible return value type (got "Tuple[str, Table]", expected "Tuple[str, VTTable]")  [return-value]
apsw/ext.py:1010: error: Name "Iterator" is not defined  [name-defined]
apsw/ext.py:1010: note: Did you forget to import it from "typing"? (Suggestion: "from typing import Iterator")
apsw/ext.py:1024: error: Name "SQLiteValue" is not defined  [name-defined]
apsw/ext.py:1055: error: Argument 1 to "next" has incompatible type "Optional[Any]"; expected "SupportsNext[Any]"  [arg-type]
apsw/ext.py:1058: error: Item "None" of "Optional[Any]" has no attribute "close"  [union-attr]
apsw/ext.py:1066: error: "Callable[..., Any]" has no attribute "columns"  [attr-defined]
apsw/ext.py:1066: error: "Callable[..., Any]" has no attribute "column_access"  [attr-defined]
apsw/ext.py:1072: error: Argument 2 to "createmodule" of "Connection" has incompatible type Module; expected "Optional[VTModule]"  [arg-type]
apsw/ext.py:1072: note: "Module" is missing following "VTModule" protocol member:
apsw/ext.py:1072: note:     ShadowName
apsw/ext.py:1072: note: Following member(s) of Module have conflicts:
apsw/ext.py:1072: note:     Expected:
apsw/ext.py:1072: note:         def Connect(self, connection: Connection, modulename: str, databasename: str, tablename: str, *args: Tuple[Union[None, int, float, bytes, str], ...]) -> Tuple[str, VTTable]
apsw/ext.py:1072: note:     Got:
apsw/ext.py:1072: note:         def Create(self, db: Any, modulename: Any, dbname: Any, tablename: Any, *args: Union[None, int, float, bytes, str]) -> Tuple[str, VTTable]
apsw/ext.py:1072: note:     Expected:
apsw/ext.py:1072: note:         def Create(self, connection: Connection, modulename: str, databasename: str, tablename: str, *args: Tuple[Union[None, int, float, bytes, str], ...]) -> Tuple[str, VTTable]
apsw/ext.py:1072: note:     Got:
apsw/ext.py:1072: note:         def Create(self, db: Any, modulename: Any, dbname: Any, tablename: Any, *args: Union[None, int, float, bytes, str]) -> Tuple[str, VTTable]
apsw/ext.py:1121: error: "Callable[[Any, Any, Any], Any]" has no attribute "columns"  [attr-defined]
apsw/ext.py:1122: error: "Callable[[Any, Any, Any], Any]" has no attribute "column_access"  [attr-defined]
apsw/ext.py:1123: error: "Callable[[Any, Any, Any], Any]" has no attribute "primary_key"  [attr-defined]
apsw/ext.py:1169: error: "Callable[[Any, Any, Any], Any]" has no attribute "columns"  [attr-defined]
apsw/ext.py:1170: error: "Callable[[Any, Any, Any], Any]" has no attribute "column_access"  [attr-defined]
apsw/ext.py:1171: error: "Callable[[Any, Any, Any], Any]" has no attribute "primary_key"  [attr-defined]
apsw/ext.py:1298: error: Incompatible types in assignment (expression has type "List[Dict[str, Any]]", target has type "str")  [assignment]
apsw/ext.py:1300: error: "str" has no attribute "append"  [attr-defined]
apsw/tests.py:25: error: Module has no attribute "apsw_should_fault"  [attr-defined]
apsw/tests.py:96: error: Incompatible types in assignment (expression has type "None", variable has type Module)  [assignment]
apsw/tests.py:97: error: Incompatible types in assignment (expression has type "None", variable has type Module)  [assignment]
apsw/tests.py:205: error: Need type annotation for "bgdelq"  [var-annotated]
apsw/tests.py:301: error: Need type annotation for "saved_connection_hooks" (hint: "saved_connection_hooks: List[<type>] = ...")  [var-annotated]
apsw/tests.py:4619: error: Cannot infer type of lambda  [misc]
apsw/tests.py:4620: error: Incompatible return value type (got "bytes", expected "str")  [return-value]
apsw/tests.py:5506: error: Need type annotation for "errs" (hint: "errs: List[<type>] = ...")  [var-annotated]
apsw/tests.py:7961: error: Library stubs not installed for "simplejson"  [import]
apsw/tests.py:7961: note: Hint: "python3 -m pip install types-simplejson"
apsw/tests.py:7961: note: (or run "mypy --install-types" to install all missing stub packages)
apsw/tests.py:7961: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
apsw/tests.py:9114: error: Cannot find implementation or library stub for module named "coverage"  [import]
apsw/tests.py:9426: error: Item "None" of "Union[None, int, float, bytes, str]" has no attribute "split"  [union-attr]
apsw/tests.py:9426: error: Item "int" of "Union[None, int, float, bytes, str]" has no attribute "split"  [union-attr]
apsw/tests.py:9426: error: Item "float" of "Union[None, int, float, bytes, str]" has no attribute "split"  [union-attr]
apsw/tests.py:9426: error: Argument 1 to "split" of "bytes" has incompatible type "str"; expected "Optional[Union[bytes, Union[bytearray, memoryview, array[Any], mmap, _CData, PickleBuffer]]]"  [arg-type]
apsw/tests.py:9524: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
apsw/tests.py:9525: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
apsw/tests.py:9526: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
apsw/tests.py:9527: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
apsw/tests.py:9528: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
apsw/tests.py:9529: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
apsw/tests.py:9708: error: Item "None" of "Optional[List[QueryAction]]" has no attribute "__iter__" (not iterable)  [union-attr]
apsw/tests.py:9709: error: Item "None" of "Optional[List[QueryAction]]" has no attribute "__iter__" (not iterable)  [union-attr]
apsw/tests.py:9738: error: Item "None" of "Optional[List[VDBEInstruction]]" has no attribute "__iter__" (not iterable)  [union-attr]
apsw/tests.py:9740: error: Argument 1 to "len" has incompatible type "Optional[List[VDBEInstruction]]"; expected "Sized"  [arg-type]
apsw/tests.py:9747: error: Argument 1 to "check_instance" has incompatible type "Optional[QueryPlan]"; expected "QueryPlan"  [arg-type]
apsw/tests.py:9753: error: Argument 1 to "count" has incompatible type "Optional[QueryPlan]"; expected "QueryPlan"  [arg-type]
apsw/tests.py:10079: error: Item "BaseException" of 
"Optional[BaseException]" has no attribute "code"  [union-attr]
apsw/tests.py:10079: error: Item "None" of "Optional[BaseException]" has no attribute "code"  [union-attr]
apsw/tests.py:10085: error: Incompatible types in assignment (expression has type "int", variable has type "Optional[str]")  [assignment]
apsw/tests.py:10086: error: No overload variant of "range" matches argument type "str"  [call-overload]
apsw/tests.py:10086: note: Possible overload variants:
apsw/tests.py:10086: note:     def __init__(self, SupportsIndex, /) -> range
apsw/tests.py:10086: note:     def __init__(self, SupportsIndex, SupportsIndex, SupportsIndex = ..., /) -> range
apsw/tests.py:10091: error: Item "BaseException" of "Optional[BaseException]" has no attribute "code"  [union-attr]
apsw/tests.py:10091: error: Item "None" of "Optional[BaseException]" has no attribute "code"  [union-attr]
Found 66 errors in 4 files (checked 7 source files)
rogerbinns commented 1 year ago

Thanks for finding this. The file is generated, so that is what had to be fixed. I have also updated the release procedure in the wiki to make sure the stubtest Makefile rule is run.

no_change is a type at the C API level because that is required for it to have a docstring. It is however a singleton so is given as object in the type stub.

There still remain 3 errors in the type stubs because mypy doesn't allow a Sequence where the members are different types. I'll have to figure out how to deal with that.

cjrh commented 1 year ago

Great, thank you 👍🏼

cjrh commented 1 year ago

doesn't allow a Sequence where the members are different types. I'll have to figure out how to deal with that.

FWIW I've run into this. If you know the range of types, you can just make a new type which is a union of the known types, and then declare the var as a sequence of the union type. At least that used to work when I last had this issue.

rogerbinns commented 1 year ago

The C code has comments which are extracted by some tools. That extracted text is then turned into the restructured text built with sphinx for the main doc. It is also processed to make docstrings as C macros so that has docs. And it is processed to do argument parsing in the C code (analagous to argument clinic). Oh and that has two different syntaxes! And the same text is also used to generate the type stubs.

So the difficulty is the same text is serving duty communicating with humans, and several different tools (sphinx, PyArg_ParseTupleAndKeywords, mypy). And this all has to work on Python 3.6 onwards! In general I have prioritised humans.

IIRC Tuple typing does allow each member being a different type, so the fix will probably be along the lines of saying Tuple instead of Sequence,. when technically Sequence is what is correct. Worst case I'll have to make what goes into the type stubs be different than the other places.

I find the stuff at the top of the module doc to be pretty terrible.