stdlib/3/json/JSONDecodeError is totally wrong

[jirutka]

I wonder how this happened and how it can be avoided, because stub for JSONDecodeError looks like a stub for totally different class.

stdlib/3/json.pyi:

class JSONDecodeError(ValueError):
    def dumps(self, obj: Any) -> str: ...
    def dump(self, obj: Any, fp: IO[str], *args: Any, **kwds: Any) -> None: ...
    def loads(self, s: str) -> Any: ...
    def load(self, fp: IO[str]) -> Any: ...

Lib/json/decoder.py:

class JSONDecodeError(ValueError):
    """Subclass of ValueError with the following additional properties:

    msg: The unformatted error message
    doc: The JSON document being parsed
    pos: The start index of doc where parsing failed
    lineno: The line corresponding to pos
    colno: The column corresponding to pos

    """
    # Note that this exception is used from _json
    def __init__(self, msg, doc, pos):
        lineno = doc.count('\n', 0, pos) + 1
        colno = pos - doc.rfind('\n', 0, pos)
        errmsg = '%s: line %d column %d (char %d)' % (msg, lineno, colno, pos)
        ValueError.__init__(self, errmsg)
        self.msg = msg
        self.doc = doc
        self.pos = pos
        self.lineno = lineno
        self.colno = colno

    def __reduce__(self):
        return self.__class__, (self.msg, self.doc, self.pos)

[gvanrossum]

Ooh, that's bad. Did you look through the git log and blame? Can you send a PR to fix it?

The general problem is that we don't have tests that verify stubs against the implementation; that's an unsolved problem, and so far we have done this by letting users send bug reports.

A more systematic approach would be great but is not easy to build.

--Guido (mobile)

[jirutka]

Yes, it seems that it’s here from the beginning (337abed05a35f0c5b67c02d42bca0af1e2adc76e).

Okay, I’ll send PR later.

[jirutka]

BTW, I’m afraid that without any automatization (with verification) and versioning of stubs, it’s not maintainable in long-term. :(

[gvanrossum]

That looks like an import from mypy, maybe there's still some history there. But I don't know if it's worth researching; things were different then...

--Guido (mobile)

[gvanrossum]

We'll worry about the long term later. For now this is better than nothing, don't you think?

--Guido (mobile)

[jirutka]

We'll worry about the long term later. For now this is better than nothing, don't you think?

That’s hard to say. I think that for projects like this is the process of keeping it up-to-date with the source the most important part at all. Python stdlib changes in time, not even talking about third-party modules, and it’s IMHO practically impossible to keep it in sync just by hand and without any verification. This inevitably leads to huge amount of out-of-sync mistakes. And if user can (reasonably) rely on type checker, because stubs are out-of-sync, then I’m afraid that it fails in its very basic purpose.

I’ve just spent about 15 minutes trying to figure out why io.FileIO seems to not be a subtype of typing.IO, if there’s some mistake in my code, stubs are wrong or what the heck is going on. Using explicit # type: for variables or # type: ignore leaded to even more confusion. And that’s just in 25 LoC in currently very small (< 100 LoC) script. Static types checker is most useful for big projects, but imagine hunting such false negatives in large code base…

I know that it’s definitely not easy and that it can’t work perfectly from the beginning, but I’m afraid that questions like how to maintain it in long-term and how to ensure correctness are very essential and not something that can be solved later.

To be more positive, I’m very happy that such effort (optional typing for Python) even exists and I really wish to be successful and accepted by community, maybe even become a killer feature of Python 3.x.

[gvanrossum]

What's your background? How can you help?