Feature Proposal: built-in way to inject Preset into sys.modules?

[beasteers]

Nice package! I've been using something similar with Sacred and was wondering if I could use something similar to change a package's defaults. And then I stumbled across this!

So I'm thinking about changing the defaults of librosa program-wide, but seeing as I need to wrap it, I need to import everything through wherever I define the Preset.

This is only a minor inconvenience so it's not a big deal, but I was wondering if there would be a way to tap into python's import system to override it everywhere.

import librosa
from presets import patch_preset

def patch_preset(module):
    import sys
    sys.modules[module.__name__] = Preset(module)

patch_preset(librosa)

# testing
del librosa
import librosa
print(librosa) # it's a Patch
import librosa.display # ModuleNotFoundError: No module named 'librosa.display'; 'librosa' is not a package

It doesn't quite work because it breaks subsequent imports of submodules, which makes sense, so I think we'd have to extend Python's Module class (I believe I've seen it under ast? but the docs are lousy) so that it can still read a Patch as a module.

Anyways, I'm not saying that we have to do anything about this, but I thought it could be a cool/convenient feature.

Bonus: I wonder if we could use Python's Import Hooks to do something like this:

# __init__.py
from presets import librosa # overrides sys.modules
librosa.update(...)

# some_scripy.py
import librosa # Preset

not even sure if that's a good idea. but it's an idea 🤷‍♀️

I think awkward thing is that, if the Preset replacement is called after anywhere else imports the package, then they'd be using the original module reference. So maybe explicit is better.

[beasteers]

I've been playing around and you still have to be concerned about import order.

from presets import Preset
import librosa
import librosa.display

librosa = Preset(librosa)
librosa.display  
# <presets.Preset at 0x7f94d0dc0da0>
librosa 
# <presets.Preset at 0x7f950c332630>

from presets import Preset
import librosa

librosa = Preset(librosa)
import librosa.display
librosa.display  
# <module 'librosa.display' from '/home/cusp/bs3639/.conda/envs/a/lib/python3.6/site-packages/librosa/display.py'>
librosa 
# <module 'librosa' from '/home/cusp/bs3639/.conda/envs/a/lib/python3.6/site-packages/librosa/__init__.py'>

So you need to make sure that you import all of the submodules before creating a Preset

[bmcfee]

but I was wondering if there would be a way to tap into python's import system to override it everywhere.

Hmmmm, dangerous territory! I see where you're coming from, but hacking in sys.modules makes me nervous.

Bonus: I wonder if we could use Python's Import Hooks to do something like this:

That would be super cool! I never thought of it, but it does seem a bit cleaner syntactically than the current setup. I was also thinking about patching into importlib directly so you could do

librosa = presets.import_lib('librosa')

but that's not quite as elegant as your proposal. I don't think it needs to hack sys.modules to work as you describe, but that might also be the only way to have global effects.

So you need to make sure that you import all of the submodules before creating a Preset

I didn't realize that was a problem, but it's definitely worth documenting. Care to open a PR?

[beasteers]

This works for me!!

import sys
import importlib
from importlib.machinery import ModuleSpec
from presets import Preset

from ast import Module

class Preset(Preset, Module):
    def __repr__(self):
        return f'<Preset \n\tof={self._module} \n\tdefaults={self._defaults}>'

class PresetsFinder(importlib.abc.MetaPathFinder):

    def __init__(self, *a, **kw):
        super().__init__(*a, **kw)
        self.preset_modules = {} # tracks modules that we've wrapped
        self._already_matched_preset = None # prevents circular calls

    def is_preset(self, parts):
        '''Checks if a module path (split on '.') has been implicitly loaded already.'''
        parts = tuple(parts)
        # if 'librosa' in parts:
        #     print(parts, self.preset_modules)
        # find any matches (match starting at beginning, obvs)
        return next((name for name in self.preset_modules
                     if name == parts[:len(name)]), None)

    def find_spec(self, fullname, path, target=None):
        # if 'librosa' in fullname:
        #     print('find_spec', fullname, path, target)
        parts = tuple(fullname.split('.'))

        if parts:
            # i.e. from presets import librosa  <<<<
            explicit_preset_import = parts[0] == 'presets'
            # i.e. from presets import librosa
            #      import librosa.display       <<<<
            preset_parent = self.is_preset(parts)
            implicit_preset_import = (
                not explicit_preset_import
                and self.is_preset(parts))

            if explicit_preset_import:
                parts = parts[1:] # cut off 'presets'

        if parts and not self._already_matched_preset == parts and (
                explicit_preset_import or preset_parent):
            # prevent inf recursion, get module the normal way
            self._already_matched_preset = parts
            spec = importlib.util.find_spec('.'.join(parts))
            self._already_matched_preset = None

            if spec:

                # share dispatch and defaults between the same package
                if explicit_preset_import and not preset_parent:
                    preset_parent = parts
                    self.preset_modules[preset_parent] = {}, {} # dispatch, defaults
                dispatch, defaults = self.preset_modules[preset_parent]

                # monkey patch in the Preset call
                # I didn't want to create my own loader, because I
                # wanted it to reuse whatever the logic of the original loader
                # used, and just tack this on at the end.
                #
                # There has to be a better way to do this, but it seems like
                # this gets the job done.
                _create_module = spec.loader.create_module
                def create_module(spec):
                    spec.loader.create_module = _create_module # replace method to avoid infinite recursion
                    module = importlib.util.module_from_spec(spec)
                    module = Preset(module, dispatch, defaults)
                    return module
                spec.loader.create_module = create_module

            return spec

sys.meta_path.insert(0, PresetsFinder())

if __name__ == '__main__':
    # import librosa
    # print(111, librosa)
    # from librosa import display
    # print(222, librosa)
    # import librosa
    # print(333, librosa)
    # import numpy
    # print(555, librosa)
    from presets import librosa
    # print(444, librosa)
    # import librosa
    # print(555, librosa)
    # import librosa.display
    # print(666, librosa.display)
    # print(66666, librosa)
    print(librosa)
    librosa['n_fft'] = 2**14
    print(librosa)
    import librosa.display
    print(librosa.display)