posts/plugins-as-python-native-namespace-packages/

[utterances-bot]

Python packages and plugins as namespace packages - pawamoy's website

Findings, thoughts, tutorials, work. Pieces of my mind!

https://pawamoy.github.io/posts/plugins-as-python-native-namespace-packages/

[bennyrowland]

Thanks for this confirmation of what my own researches were leading me towards. The scenario you lay out at the end is exactly what I wanted, but I really don't want to give up on my init.py in the top level namespace of the original package, as that imports various functions, classes etc. into the main namespace, including things as basic as a version string. My current thought for a fix is to create a separate top level (namespace) package for the plugins, which can do without the init.py, but I suspect there will be some issues I haven't yet thought of.

[pawamoy]

Glad to see I'm not the only one with this idea!

So, what you are thinking about is to have your regular main package, lets call it package, and then a second package meant for plugins, called package-plugins?

package/
    __init__.py
    module.py
    plugins_loader.py

package-plugins/
    builtin_plugin.py

package-plugins/
    my_custom_plugin.py

The main package would then be able to import plugins from the namespaced package-plugins package (using some import machinery from a plugin_loader module/function)?

That's a very nice idea! I'll try to think a bit more about it and see if I could apply it to my own project :slightly_smiling_face:

Thanks a lot for sharing :smile:

[pawamoy]

I can see a limitation indeed. Lets say the main package lists the plugins one as a dependency to provide some "builtin" plugins, like a plugin whose module is named thing.py. Now imagine someone else wants to override this thing.py with their own. They write the namespaced plugin, list it in their dependencies and... which one gets installed? The original one or the custom one? I guess that is non-deterministic :confused:

Possible answers to this are:

• don't support overriding plugins (just like Python doesn't support installing and using two packages of the same name)
• don't originally depend on plugins (make them extras)
• put the native plugins in the main package? but that would make the loading machinery more complex

[bennyrowland]

Your summary is exactly what I was thinking of, I was thinking of using straight.plugin to handle the loading part. My use case is in image processing - I want the user to basically have a list of plugins that will each transform an image and that can come from a variety of packages installed on the same namespace.

Supporting overriding is not something I was worried about but I did have the concern of possible namespace clashes. I don't really know what to do about that and I haven't been able to find any references as to what happens under those circumstances. In my case I primarily want the plugin architecture so I can have individual versioning on each plugin, I will probably be writing most of the plugins myself so can avoid that problem, but in general I don't have a good solution. Even if we separate out the native plugins, there is nothing stopping people trying to install two plugin packages that compete for the same module name.

[pawamoy]

Yeah you're right. Actually I'm not even sure overriding is possible with current mkdocstrings architecture. I guess I'll update the docs to say that overriding builtin plugins is not possible, and two plugins using the same module names is not supported either.

[MatthiasLohr]

Do you have any new ideas on this? Also running in the same problem, but no real elegant solution yet...

[pawamoy]

Well, I've been playing more and more with namespace packages (actually trying to support them in a project of mine), and I can tell you that namespace packages are complicated. Two packages of the same namespace can be installed in the same folder, or in different folders (different entries in sys.path). Even a single namespace package can be split into several locations when installing it in editable mode. So in the end I think it's usually better to just rely on entry points.