search

metoppv / improver

IMPROVER is a library of algorithms for meteorological post-processing.
http://improver.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
101 stars 84 forks source link

Plugin discovery #2009

Open cpelley opened 2 weeks ago

cpelley commented 2 weeks ago

Closes https://github.com/metoppv/improver/issues/2005

Description

https://github.com/metoppv/improver/blob/1eb1b195c7525c73d1530d1d7dc4030404675f9a/improver/api/__init__.py#L6-L8

Issues

cpelley commented 2 weeks ago

lazy import: downside: no discoverability (tab completion etc.)

from importlib import import_module

PROCESSING_MODULES = {
    'OccurrenceBetweenThresholds': 'improver.between_thresholds',
    'TriangularWeightedBlendAcrossAdjacentPoints': 'improver.blending.blend_across_adjacent_points',
...
}

def __getattr__(name):
    mod = import_module(PROCESSING_MODULES[name])
    return getattr(mod, name)

VS

straight import:

from improver.between_thresholds import OccurrenceBetweenThresholds
from improver.blending.blend_across_adjacent_points import TriangularWeightedBlendAcrossAdjacentPoints
...

I think most of the time spent importing is due to numpy and most modules use numpy. Once you have imported one, you have little to do to import the rest. All in all, sub 2.5s for importing everything (not much less for importing a single module).

EDIT: TBH it doesn't matter which approach we go for in the short term. We can always go from the 1 approach to the other without an impact to users of IMPROVER.

cpelley commented 2 weeks ago

TBH, I think I'll just polish this up. I don't think it matters which approach we take really. The interface is the same to users of IMPROVER. We can always change our implementation details down the road. The only thing we need to agree upon right now is making the plugins available via:

improver.api.<plugin-name>