Remove `keystoneauth1` error when using filesystem storage

[cooperlees]

2020-08-10 00:12:50,875 INFO: Selected storage backend: filesystem
ERROR:root:Unable to load entry point swift_plugin = bandersnatch_storage_plugins.swift:SwiftStorage: No module named 'keystoneauth1'

• Example CI Job I saw this in: https://github.com/pypa/bandersnatch/pull/648/checks?check_run_id=964744678

I think this is due to not installing the deps, but why do we need to load this if we're not using it? Is this a limitation of the way plugins are? How could we refactor if so?

[RWoodring79]

Confirming that I see this same error in my local setup. Using a clean Python 3.6.9 environment set up like this

python3 -m venv venv
source venv/bin/activate
pip install -U pip setuptools
pip install bandersnatch

When I tried to run bandersnatch I got the same keystoneauth1 error

2020-09-30 08:08:18,226 INFO: Selected storage backend: filesystem
ERROR:root:Unable to load entry point swift_plugin = bandersnatch_storage_plugins.swift:SwiftStorage: No module named 'keystoneauth1'

I then did a pip install keystoneauth1 in hopes that would resolve it, but it now complains about swiftclient

2020-09-30 08:01:58,643 INFO: Selected storage backend: filesystem
ERROR:root:Unable to load entry point swift_plugin = bandersnatch_storage_plugins.swift:SwiftStorage: No module named 'swiftclient'

[zmylk]

pip install swiftclient

[bduff-walleye]

Possibly: pip install python-swiftclient Search for "swiftclient" and see.

[electricworry]

I'd like to help with this. The purist in me wants to see if there's a good way to suppress the error without having the dependency. Seems to me that if the plugin isn't in use then it shouldn't create noise. I'll maybe get a chance to look early Jan.

[electricworry]

I'm not sure how to elegantly fix this. The problem is in the src/bandersnatch/storage.py function load_storage_plugins():

    for entry_point in pkg_resources.iter_entry_points(group=entrypoint_group):
        try:
            plugin_class = entry_point.load()
            plugin_instance = plugin_class(config=config)
            if plugin_instance.name == enabled_plugin:
                plugins.add(plugin_instance)
        except ModuleNotFoundError as me:
            logger.error(f"Unable to load entry point {entry_point}: {me}")

enabled_plugin is set to "filesystem", as per configuration. When iterating the entry points and hitting the swift plugin, entry_point object has the name property "swift_plugin", but we need to check name property of the SwiftStorage class (which is "swift") which can only be checked after entry_point.load() (which fails due to missing package keystoneauth1).

Rather than loading each storage class and then checking its name attribute, we could instead check the entry_point name. Like this:

    for entry_point in pkg_resources.iter_entry_points(group=entrypoint_group):
        if entry_point.name == enabled_plugin + "_plugin":
            try:
                plugin_class = entry_point.load()
                plugin_instance = plugin_class(config=config)
                plugins.add(plugin_instance)
            except ModuleNotFoundError as me:
                logger.error(f"Unable to load entry point {entry_point}: {me}")

Which resolves the problem. However, it only works if entry_point names are consistent with the storage class names.

class SwiftStorage(StoragePlugin):
    name = "swift"

class FilesystemStorage(StoragePlugin):
    name = "filesystem"

So, entry points must be the class name + "_plugin".

    swift_plugin = bandersnatch_storage_plugins.swift:SwiftStorage
    filesystem_plugin = bandersnatch_storage_plugins.filesystem:FilesystemStorage

Thoughts?