metoppv / improver

IMPROVER is a library of algorithms for meteorological post-processing.
http://improver.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
100 stars 84 forks source link

Plugin discoverability #2005

Open cpelley opened 1 month ago

cpelley commented 1 month ago

As part of the https://github.com/metoppv/improver/issues/1998 work on moving CLI logic to the plugin layer to ultimately access plugins directly rather than via the CLI, this open up a necessary plication right now that plugins when called from where they currently live, would make users subject to changes in the layout of the IMPROVER repository.

CLIs are all available in a flat space namespace. That is, all under improver.cli Similarly, I think it might be useful to make plugins available under a flat namespace (thereby protecting science configurations from changes in project layout/structure). For example, perhaps improver.plugin.Standardise.

This doesn't mean the plugins themselves need live under a root subpackage folder, instead they could simply be made available through a single namespace.

To illustrate (not proposed solution):

# improver/plugin.py
improver/standardise.py
from improver.standardise import StandardiseMetadata
...
from improver.psychrometric_calculations.cloud_condensation_level import CloudCondensationLevel
...

Issues

TomekTrzeciak commented 4 days ago

CLIs are all available in a flat space namespace. That is, all under improver.cli Similarly, I think it might be useful to make plugins available under a flat namespace (thereby protecting science configurations from changes in project layout/structure). For example, perhaps improver.plugin.Standardise.

This doesn't mean the plugins themselves need live under a root subpackage folder, instead they could simply be made available through a single namespace.

At the time I was playing with https://github.com/metoppv/improver/pull/1654, I had an idea of adding improver.api as a way of marking what is intended as public interface. At the time that was for CLI-generating functions, but could be instead for other interfaces.

# improver/api/__init__.py
"""IMPROVER public API."""

from importlib import import_module

def __getattr__(name):
    mod = import_module(f"improver.cli.{name}")
    return mod.process
cpelley commented 3 days ago

Interesting idea šŸ‘ One downside is discoverability. You wont be able to query what there is, only try asking for it and seeing if it fails Also, at least currently, plugins are defined all over the shop in IMPROVER. The main benefit I see of this it not having to import all the plugins into the 1 namespace (which I'm not sure has much of a performance concern anymore if everything runs within a single python invocation). There could be other benefits that I haven't thought of mind :)

cpelley commented 3 days ago

...plugins are defined all over the shop in IMPROVER...

click to expand ``` from improver.between_thresholds import OccurrenceBetweenThresholds from improver.blending.blend_across_adjacent_points import TriangularWeightedBlendAcrossAdjacentPoints from improver.blending.calculate_weights_and_blend import WeightAndBlend from improver.blending.spatial_weights import SpatiallyVaryingWeightsFromMask ... ``` ``` from improver.blending.weighted_blend import MergeCubesForWeightedBlending from improver.blending.weighted_blend import WeightedBlendAcrossWholeDimension from improver.blending.weights import ChooseWeightsLinear from improver.blending.weights import ChooseDefaultWeightsLinear from improver.blending.weights import ChooseDefaultWeightsNonLinear from improver.blending.weights import ChooseDefaultWeightsTriangular from improver.calibration.ensemble_calibration import ContinuousRankedProbabilityScoreMinimisers from improver.calibration.ensemble_calibration import EstimateCoefficientsForEnsembleCalibration from improver.calibration.ensemble_calibration import CalibratedForecastDistributionParameters from improver.calibration.ensemble_calibration import ApplyEMOS from improver.calibration.reliability_calibration import ConstructReliabilityCalibrationTables from improver.calibration.reliability_calibration import AggregateReliabilityCalibrationTables from improver.calibration.reliability_calibration import ManipulateReliabilityTable from improver.calibration.reliability_calibration import ApplyReliabilityCalibration from improver.calibration.rainforest_calibration import ApplyRainForestsCalibration from improver.calibration.dz_rescaling import EstimateDzRescaling from improver.calibration.dz_rescaling import ApplyDzRescaling from improver.calibration.simple_bias_correction import CalculateForecastBias from improver.calibration.simple_bias_correction import ApplyBiasCorrection from improver.cube_combiner import Combine from improver.cube_combiner import CubeCombiner from improver.cube_combiner import MaxInTimeWindow from improver.ensemble_copula_coupling.ensemble_copula_coupling import RebadgeRealizationsAsPercentiles from improver.ensemble_copula_coupling.ensemble_copula_coupling import RebadgePercentilesAsRealizations from improver.ensemble_copula_coupling.ensemble_copula_coupling import ResamplePercentiles from improver.ensemble_copula_coupling.ensemble_copula_coupling import ConvertProbabilitiesToPercentiles from improver.ensemble_copula_coupling.ensemble_copula_coupling import EnsembleReordering from improver.generate_ancillaries.generate_ancillary import CorrectLandSeaMask from improver.generate_ancillaries.generate_ancillary import GenerateOrographyBandAncils from improver.generate_ancillaries.generate_orographic_smoothing_coefficients import OrographicSmoothingCoefficients from improver.generate_ancillaries.generate_svp_table import SaturatedVapourPressureTable from improver.generate_ancillaries.generate_derived_solar_fields import GenerateSolarTime from improver.generate_ancillaries.generate_derived_solar_fields import GenerateClearskySolarRadiation from improver.generate_ancillaries.generate_topographic_zone_weights import GenerateTopographicZoneWeights from improver.lapse_rate import ApplyGriddedLapseRate from improver.lapse_rate import LapseRate from improver.threshold import Threshold from improver.nbhood.nbhood import BaseNeighbourhoodProcessing from improver.nbhood.nbhood import MetaNeighbourhood from improver.nbhood.use_nbhood import ApplyNeighbourhoodProcessingWithAMask from improver.nbhood.recursive_filter import RecursiveFilter from improver.nowcasting.accumulation import Accumulation from improver.nowcasting.forecasting import AdvectField from improver.nowcasting.forecasting import CreateExtrapolationForecast from improver.nowcasting.lightning import NowcastLightning from improver.nowcasting.optical_flow import OpticalFlow from improver.nowcasting.pysteps_advection import PystepsExtrapolate from improver.nowcasting.utilities import ExtendRadarMask from improver.nowcasting.utilities import FillRadarHoles from improver.nowcasting.utilities import ApplyOrographicEnhancement from improver.orographic_enhancement import OrographicEnhancement from improver.percentile import PercentileConverter from improver.precipitation_type.convection import ConvectionRatioFromComponents from improver.precipitation_type.shower_condition_probability import ShowerConditionProbability from improver.precipitation_type.snow_fraction import SnowFraction from improver.precipitation_type.snow_splitter import SnowSplitter from improver.precipitation_type.freezing_rain import FreezingRain from improver.precipitation_type.hail_fraction import HailFraction from improver.psychrometric_calculations.precip_phase_probability import PrecipPhaseProbability from improver.psychrometric_calculations.psychrometric_calculations import HumidityMixingRatio from improver.psychrometric_calculations.psychrometric_calculations import PhaseChangeLevel from improver.psychrometric_calculations.significant_phase_mask import SignificantPhaseMask from improver.psychrometric_calculations.hail_size import HailSize from improver.psychrometric_calculations.cloud_condensation_level import MetaCloudCondensationLevel from improver.psychrometric_calculations.cloud_condensation_level import CloudCondensationLevel from improver.psychrometric_calculations.cloud_top_temperature import CloudTopTemperature from improver.psychrometric_calculations.wet_bulb_temperature import WetBulbTemperature from improver.psychrometric_calculations.wet_bulb_temperature import WetBulbTemperatureIntegral from improver.psychrometric_calculations.wet_bulb_temperature import MetaWetBulbFreezingLevel from improver.regrid.landsea import RegridLandSea from improver.regrid.landsea import AdjustLandSeaPoints from improver.regrid.landsea2 import RegridWithLandSeaMask from improver.spotdata.apply_lapse_rate import SpotLapseRateAdjust from improver.spotdata.neighbour_finding import NeighbourSelection from improver.spotdata.spot_extraction import SpotExtraction from improver.spotdata.height_adjustment import SpotHeightAdjustment from improver.standardise import StandardiseMetadata from improver.utilities.cube_extraction import ExtractSubCube from improver.utilities.cube_extraction import ExtractLevel from improver.utilities.cube_manipulation import MergeCubes from improver.utilities.interpolation import InterpolateUsingDifference from improver.utilities.mathematical_operations import Integration from improver.utilities.solar import DayNightMask from improver.utilities.spatial import DifferenceBetweenAdjacentGridSquares from improver.utilities.spatial import GradientBetweenAdjacentGridSquares from improver.utilities.spatial import OccurrenceWithinVicinity from improver.utilities.temporal_interpolation import TemporalInterpolation from improver.utilities.textural import FieldTexture from improver.utilities.time_lagging import GenerateTimeLaggedEnsemble from improver.utilities.forecast_reference_enforcement import EnforceConsistentForecasts from improver.utilities.copy_attributes import CopyAttributes from improver.lightning import LightningFromCapePrecip from improver.lightning import LightningMultivariateProbability_USAF2024 from improver.wind_calculations.wind_components import ResolveWindComponents from improver.wind_calculations.wind_direction import WindDirection from improver.wind_calculations.wind_downscaling import FrictionVelocity from improver.wind_calculations.wind_downscaling import RoughnessCorrection from improver.wind_calculations.wind_gust_diagnostic import WindGustDiagnostic from improver.wind_calculations.vertical_updraught import VerticalUpdraught from improver.categorical.decision_tree import ApplyDecisionTree from improver.categorical.modal_code import ModalCategory from improver.expected_value import ExpectedValue from improver.visibility.visibility_combine_cloud_base import VisibilityCombineCloudBase ```

šŸ˜±

cpelley commented 3 days ago

TBH, it's numpy that take most of the import hit so importing 1 processing module is the for the most part the same as importing them all.