Mako should declare what extra dependencies are required for the custom gettext message extractors

sinoroc commented 5 years ago

Mako provides gettext message extractors for lingua and Babel. They are declared as setuptools entry points. Such entry points can and should specify a list of 'extras': third-party dependencies that should be installed in order to use these custom entry points.

See for example issue wichert/lingua#94: lingua has the capability to use custom message extractors for both lingua and Babel. Now, if Mako and lingua are installed but Babel is not, then lingua blindly loads Mako's custom message extractors for Babel and fails while trying to use them since Babel is not installed. To help lingua (and potentially other tools as well) to recognize that Babel should be installed in order to use the custom Mako-for-Babel message extractor, Mako should declare Babel as an extra dependency of the entry point for the message extractor.

See the setuptools documentation on "Dynamic Discovery of Services and Plugins" for details.

sinoroc commented 5 years ago

I am working on fixes for both Mako and lingua.

zzzeek commented 5 years ago

OK silly question. Why isn't Babel a dependency of Lingua then?

sinoroc commented 5 years ago

It's a bit confusing, but straightforward: lingua does not import any code from Babel, Mako does.

zzzeek commented 5 years ago

OK that answer is not enough information - Mako only imports Babel in the Babel plugin which is not referenced anywhere else. in your stack trace at https://github.com/wichert/lingua/issues/94, we see that lingua is not necessarily importing babel, but it is in fact calling upon it:

File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/lingua/extract.py", line 308, in main
    register_babel_plugins()

above is where lingua has a depedendency on babel. But it's not an "import" dependency, it's a set of dependencies at the entrypoint level, not something I've seen before. if lingua is hardcoded to install a babel entrypoint, I guess this could go either way but it still seems like lingua has babel as a pretty strong dependency.

zzzeek commented 5 years ago

here's the better doc:

https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies

now I do have some "extras_require" in some other projects but I've not combined these with entrypoints. When exactly does "Babel" get installed?

sinoroc commented 5 years ago

I have published a fix for lingua: wichert/lingua#95 Hopefully it clears up some points.

sinoroc commented 5 years ago

My use case looks like this:

Pyramid application with Chameleon templates.
I use lingua to extract messages from my Chameleon templates and I don't want to use Babel at all.
For debug purposes I also have installed pyramid_debugtoolbar which has a non optional dependency on Mako. I assume that its Mako templates have already been localized, so I don't need to extract messages from these.

This causes the issue presented in wichert/lingua#94.

When exactly does "Babel" get installed?

When the user wants to use it to extract messages in their project. For this to work well whether or not Babel is installed, following changes are required:

Mako has to clearly publicize the fact that the entry point works only if Babel is installed.
lingua has to ignore the entry point if a dependency (Babel) is not installed.

This way lingua loads the Mako-for-Babel extractor only if Babel is explicitly installed, for example the user could add Mako[babel] to their project dependencies.

zzzeek commented 5 years ago

OK by "when does babel get installed" I was more trying to determine the imperative effects of these directives being added. that is, without the directive, right now you're getting I assume the error, "can't find any module called Babel", or something like that (feel free to spell this out for me). and with the directive, what will happen in your case? the paths to this plugin will be silently ignored and your program just proceeds? (since this is all in a debug toolbar you aren't developing anyway)?

sinoroc commented 5 years ago

Not entirely sure I understand the question, I will try to answer with this, I might still be off-topic though...

In short: with the extra directive lingua gains clearer knowledge of what is going wrong: a dependency is missing as well as its exact name ; and yes, it can then choose to ignore this plugin entirely.

Adding the extras to the entry points (when Babel is not installed), brings us from:

ModuleNotFoundError: No module named 'babel'

to:

pkg_resources.DistributionNotFound: The 'Babel' distribution was not found and is required by the application

(complete stack traces below ...)

The former is a relatively vague exception, the module name does not actually give viable information about which dependency is missing.

The latter shows clearly that a dependency is missing and what its name is.

In our case it makes sense to let lingua simply ignore the entry points that raise pkg_resources.DistributionNotFound, which is what has been done in wichert/lingua#95.

Alternatively lingua could show the user a warning listing the message extractors that have been ignored and which dependencies should be installed to activate them.

Complete stack traces

Without extras directive:

Traceback (most recent call last):
  File "/home/user/workspace/project/.tox/develop/bin/pot-create", line 11, in <module>
    sys.exit(main())
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/lingua/extract.py", line 308, in main
    register_babel_plugins()
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/lingua/extractors/babel.py", line 51, in register_babel_plugins
    extractor = entry_point.load(require=True)
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/mako/ext/babelplugin.py", line 8, in <module>
    from babel.messages.extract import extract_python
ModuleNotFoundError: No module named 'babel'

With extras directive:

Traceback (most recent call last):
  File "/home/user/workspace/project/.tox/develop/bin/pot-create", line 11, in <module>
    sys.exit(main())
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/lingua/extract.py", line 308, in main
    register_babel_plugins()
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/lingua/extractors/babel.py", line 51, in register_babel_plugins
    extractor = entry_point.load(require=True)
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2323, in load
    self.require(*args, **kwargs)
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2346, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/home/user/workspace/project/.tox/develop/lib/python3.6/site-packages/pkg_resources/__init__.py", line 778, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'Babel' distribution was not found and is required by the application

zzzeek commented 5 years ago

OK so the change gives us a more informative error message, that's what I was looking for, thanks.

sqla-tester commented 5 years ago

sinoroc has proposed a fix for this issue in the master branch:

Specify extras for gettext message extractors https://gerrit.sqlalchemy.org/1436

sqlalchemy / mako

Mako should declare what extra dependencies are required for the custom gettext message extractors #304