python-babel / babel

The official repository for Babel, the Python Internationalization Library
http://babel.pocoo.org/
BSD 3-Clause "New" or "Revised" License
1.34k stars 448 forks source link

Initial support for reading mapping configuration as TOML #1108

Closed akx closed 3 months ago

akx commented 4 months ago

This PR offers to implement part of https://github.com/python-babel/babel/issues/777.

Namely, following this PR you would be able to run pybabel extract -F babel.toml – or even pybabel extract -F pyproject.toml, as we already will support tool.babel as well as babel for the namespace. (Auto-detecting a pyproject.toml instead of setup.cfg or babel.cfg is not implemented in this PR.)

I'm requesting comments on the proposed TOML format, which borrows the idea of mappings-as-lists from Mypy's overrides configuration.

[babel.extractors]
custom = "mypackage.module:myfunc"

# Python source files
[[babel.mappings]]
method = "python"
pattern = "**.py"

# Genshi templates
[[babel.mappings]]
method = "genshi"
pattern = "**/templates/**.html"
include_attrs = ""

[[babel.mappings]]
method = "genshi"
pattern = "**/templates/**.txt"
template_class = "genshi.template:TextTemplate"
encoding = "latin-1"

# Some custom extractor
[[babel.mappings]]
method = "custom"
pattern = "**/custom/*.*"
akx commented 4 months ago

cc @zharktas, @tomasr8, @eemeli – please weigh in if you have ideas :)

codecov[bot] commented 4 months ago

Codecov Report

Attention: Patch coverage is 91.30435% with 6 lines in your changes missing coverage. Please review.

Project coverage is 91.22%. Comparing base (d3346ee) to head (a820cf5). Report is 2 commits behind head on master.

Files Patch % Lines
babel/messages/frontend.py 91.30% 6 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #1108 +/- ## ========================================== + Coverage 91.21% 91.22% +0.01% ========================================== Files 27 27 Lines 4496 4561 +65 ========================================== + Hits 4101 4161 +60 - Misses 395 400 +5 ``` | [Flag](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | Coverage Δ | | |---|---|---| | [macos-12-3.10](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `90.02% <91.30%> (+0.03%)` | :arrow_up: | | [macos-12-3.11](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `89.95% <86.95%> (-0.04%)` | :arrow_down: | | [macos-12-3.12](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `90.13% <86.95%> (-0.04%)` | :arrow_down: | | [macos-12-3.13-dev](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `89.65% <86.95%> (-0.03%)` | :arrow_down: | | [macos-12-3.8](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `89.95% <91.30%> (+0.03%)` | :arrow_up: | | [macos-12-3.9](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `89.95% <91.30%> (+0.03%)` | :arrow_up: | | [macos-12-pypy3.10](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `?` | | | [ubuntu-22.04-3.10](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `90.04% <91.30%> (+0.03%)` | :arrow_up: | | [ubuntu-22.04-3.11](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `89.98% <86.95%> (-0.04%)` | :arrow_down: | | [ubuntu-22.04-3.12](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `90.15% <86.95%> (-0.04%)` | :arrow_down: | | [ubuntu-22.04-3.13-dev](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `89.67% <86.95%> (-0.03%)` | :arrow_down: | | [ubuntu-22.04-3.8](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `89.97% <91.30%> (+0.03%)` | :arrow_up: | | [ubuntu-22.04-3.9](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `89.97% <91.30%> (+0.03%)` | :arrow_up: | | [ubuntu-22.04-pypy3.10](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `90.04% <91.30%> (+0.03%)` | :arrow_up: | | [windows-2022-3.10](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `90.16% <91.30%> (+0.03%)` | :arrow_up: | | [windows-2022-3.11](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `90.10% <86.95%> (-0.04%)` | :arrow_down: | | [windows-2022-3.12](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `90.27% <86.95%> (-0.04%)` | :arrow_down: | | [windows-2022-3.13-dev](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `89.79% <86.95%> (-0.04%)` | :arrow_down: | | [windows-2022-3.8](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `90.09% <91.30%> (+0.03%)` | :arrow_up: | | [windows-2022-3.9](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `90.09% <91.30%> (+0.03%)` | :arrow_up: | | [windows-2022-pypy3.10](https://app.codecov.io/gh/python-babel/babel/pull/1108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | `90.16% <91.30%> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

akx commented 4 months ago

Oh, and cc @miigotu too of course as the opener of #777 😄

tomasr8 commented 4 months ago

Looks great! I was just wondering how you can specify ignore paths. Would this work?

[[babel.mappings]]
method = "ignore"
pattern = "test/*"

Can the pattern be a list?

akx commented 4 months ago

Looks great! I was just wondering how you can specify ignore paths.

By not specifying them in a mapping at all... 😅 There's also the command-line --ignore-dirs option for wholesale directory ignoring.

But if your imagined use-case is "extract from all HTML files with jinja2, but never extract anything from test*.html", that would be a separate issue. Perfectly doable with a no-op extractor (that we don't yet have), since the list of mappings is in priority order, but that'd also work for .cfg style mapping files.

Can the pattern be a list?

Might as well support that too!

akx commented 4 months ago

I'm a bit tired so maybe I missed something, but there already exists a noop extractor - extract_nothing so for example this cfg currently works as expected:

My bad, it's not you being tired. I had forgotten there's a special if method == 'ignore': case in extract_from_file that handles that (as well as the 'ignore': extract_nothing case in _BUILTIN_EXTRACTORS)...

akx commented 4 months ago

Entirely unsolicited,

Not at all, thank you for the input! Much appreciated.

but an option might be: ...

I'm not completely sold on that – feels like it becomes too easy to accidentally do [mappings.python] if you expect to only have a single mapping for python, and you'd then get a (possibly inscrutable) parsing type error.

I would also drop the babel. prefix for the babel.toml file -- it is needlessly explicit (see e.g. .ruff.toml which drops the full tool.ruff prefix).

That's probably a good idea.

AA-Turner commented 4 months ago

I'm not completely sold on that – feels like it becomes too easy to accidentally do [mappings.python] if you expect to only have a single mapping for python, and you'd then get a (possibly inscrutable) parsing type error.

I think this is a problem regardless. For example this PR adds a config file with one entry, [jinja2: **.html]. Someone without knowledge of TOML's quirks might write

[mapping]
method = "jinja2"
pattern = "**.html"

which looks right, but would instantly fail. Worse would be:

[babel.mappings]
method = "genshi"
pattern = "**/templates/**.html"
include_attrs = ""

[babel.mappings]
method = "genshi"
pattern = "**/templates/**.txt"
template_class = "genshi.template:TextTemplate"
encoding = "latin-1"

This is luckily an error in TOML, but still confusing. There should be a clear and early error message if isinstance(config['mapping'], dict) is True (or in my proposal, any(isinstance(k, dict) for k in config['mapping'].keys())).

I think with the clear error message (that I posit is needed regardless of the option taken), my proposal still has merit. However, it does have drawbacks as you note.

akx commented 3 months ago

@AA-Turner @tomasr8 Do you have the time to take another look at this? Thanks!

akx commented 3 months ago

Thank you for the wonderful comments and review! ❤️