search

python / cpython

The Python programming language
https://www.python.org
Other
62.17k stars 29.88k forks source link

Add source location to tomllib #115516

Open jfu334 opened 6 months ago

jfu334 commented 6 months ago

Feature or enhancement

Proposal:

Hello CPython devs!

Currently tomllib does not allow to link the returned data to the line number it originated from. This information would be immensely helpful to create proper error messages in a semantic verification step, like: 

Key 'foo' on line 64 has wrong type ...

The acceptance of text based configuration files for novice users usually grows with the degree and usefulness of errors reported when using them. All major XML parsers support this but I don't want to bloat users with XML as it is just too much for simple cases. TOML would fit perfectly.

The information is all around in the parser but not forwarded to the result. An easy approach would be to provide this information at least for all dicts which would enable messages like: 

Key 'foo' in table at line 64 has wrong type ...

A quick and dirty solution would be to add a .position property to NestedDict.dict. I see that this leads to some difficulties regarding type annotations, but there may be better solutions.

I was able to get the position by monkey patching create_list_rule_original but a more sustainable solution would be really nice.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

hugovk commented 6 months ago

Perhaps this should be proposed to https://github.com/hukkin/tomli first? cc @hauntsaninja.

jfu334 commented 6 months ago

Sorry if this is the wrong project. Since 3.11 tomllib is integrated into cpython, is this just a mirror of https://github.com/hukkin/tomli?

There have also already been feature requests regarding tomllib on this tracker (https://github.com/python/cpython/issues/103188) so I'm a bit confused.

hugovk commented 6 months ago

Yep, the read-only parts of tomli was copied into CPython as tomllib for 3.11.

Since then, there haven't really been any notable additions to tomli that warrant adding to tomllib, and no there's been no changes to tomllib to send to tomli.

So I guess it's not really come up, seeing as feature requests such as #103188 were closed.

In general, it's much easier to add things to PyPI packages, which can release as often as they like, compared to CPython with one feature release per year. And after that consider moving to CPython.

Anyway, I'll leave this to @hauntsaninja and @hukkin.

hauntsaninja commented 6 months ago

Hm, you can't patch a position attribute onto a normal dict. In general, my inclination is that attaching metadata to parse results is out of scope for tomllib — the design questions feel similar to attaching style metadata in that you need custom types. For whatever it's worth, I've found error messages that mention the key and value to be sufficient for error messages even for newer users (acknowledging that all the projects I'm involved with that use TOML for configuration are not heavy array of tables users).

re tomli vs tomllib: don't have a strong opinion on what issues are reported where, but if we were to make non-trivial extensions to the API, it's best if they bake in a place that isn't the standard library for a while.