Support loading feature/protocol modules from outside the project

[ZeroKnight]

Leverage namespace packages to enable the loading of protocol and feature modules from outside of ZeroBot's distribution and the regular Python path. This is a rather critical feature, as ZeroBot is intended to have drop-in user modules.

The default place to load extra features and protocols from will be $XDG_DATA_HOME/ZeroBot/{features, protocols}. More locations may be specified in zerobot.toml, i.e. Core.ModulePath.

Implementation

First off, sys.path should probably not be modified permanently. Having the feature/protocol loading directories in sys.path permanently would potentially allow unrelated modules/packages to be loaded from here elsewhere in ZeroBot. The addition of these paths must be done on-demand or done some other way.

The sys.path route

We simply modify sys.path at some point, then call importlib.import_module as usual. For example (from here):

>>> import sys

>>> sys.path += ['Lib/test/namespace_pkgs/project1', 'Lib/test/namespace_pkgs/project2']

>>> import parent.child.one
>>> parent.__path__

_NamespacePath(['Lib/test/namespace_pkgs/project1/parent', 'Lib/test/namespace_pkgs/project2/parent'])

>>> parent.child.__path__

_NamespacePath(['Lib/test/namespace_pkgs/project1/parent/child', 'Lib/test/namespace_pkgs/project2/parent/child'])

>>> import parent.child.two
>>>

The importlib shenanigans route

Alternatively, we might be able to avoid changing sys.path by setting up our own importer... unsure if this is overkill, though.

Brainstorming

Meta hooks might be what we're after:

Meta hooks are called at the start of import processing, before any other import processing has occurred, other than sys.modules cache look up. This allows meta hooks to override sys.path processing, frozen modules, or even built-in modules.

Actually, I think we really need an import path hook after all—which in itself is powered by a default meta hook. This hook makes use of sys.path and package __path__ attributes (see comment).

Nope, I misunderstood how import path hooks worked (and still don't grasp them 100% honestly). I specifically want to avoid adding ZeroBot's external feature/protocol module paths to sys.path. Also, the path hooks seem to only make use of __path__ attributes, and not set them.

[ZeroKnight]

For the importlib route, ModuleSpec.submodule_search_locations might be helpful. Though keep in mind this note in the ModuleSpec description:

In the descriptions below, the names in parentheses give the corresponding attribute available directly on the module object. E.g. module.__spec__.origin == module.__file__. Note however that while the values are usually equivalent, they can differ since there is no synchronization between the two objects. Thus it is possible to update the module’s __path__ at runtime, and this will not be automatically reflected in __spec__.submodule_search_locations.

Or...

Since this attribute is tied to a package module's __path__ attribute... could we be cheeky and simply do something like:

# somewhere, probably in ZeroBot.module
import ZeroBot.feature
import ZeroBot.protocol

# somewhere in Core
ZeroBot.feature.__path__.append(self._config_dir / 'feature')
ZeroBot.protocol.__path__.append(self._config_dir / 'protocol')

As it turns out, this is actually more or less how namespace packages were implemented before PEP 420.

NOTE: PEP 420 states that the import machinery will dynamically compute __path__ when needed; so would this overwrite the __path__ modification we make here at any point?

[ZeroKnight]

Note to self regarding the sys.path route:

While trying out this approach, I was noticing that even though I had appended self._config_dir / 'feature' to sys.path, ZeroBot.feature.__path__ did not include this directory upon import, despite it being a namespace package. This was due to the fact that ZeroBot at this time has an __init__.py, and this not only prevents ZeroBot from being an implicit namespace package, but all subpackages as well. Removing ZeroBot/__init__.py allowed ZeroBot.feature to be correctly interpreted as an implicit namespace package, with both directories appearing in ZeroBot.feature.__path__.

This is a particularly important detail of how implicit namespace packages are detected; one that I didn't realize until now.