Open beasteers opened 5 years ago
I've been playing around and you still have to be concerned about import order.
from presets import Preset
import librosa
import librosa.display
librosa = Preset(librosa)
librosa.display
# <presets.Preset at 0x7f94d0dc0da0>
librosa
# <presets.Preset at 0x7f950c332630>
from presets import Preset
import librosa
librosa = Preset(librosa)
import librosa.display
librosa.display
# <module 'librosa.display' from '/home/cusp/bs3639/.conda/envs/a/lib/python3.6/site-packages/librosa/display.py'>
librosa
# <module 'librosa' from '/home/cusp/bs3639/.conda/envs/a/lib/python3.6/site-packages/librosa/__init__.py'>
So you need to make sure that you import all of the submodules before creating a Preset
but I was wondering if there would be a way to tap into python's import system to override it everywhere.
Hmmmm, dangerous territory! I see where you're coming from, but hacking in sys.modules makes me nervous.
Bonus: I wonder if we could use Python's Import Hooks to do something like this:
That would be super cool! I never thought of it, but it does seem a bit cleaner syntactically than the current setup. I was also thinking about patching into importlib directly so you could do
librosa = presets.import_lib('librosa')
but that's not quite as elegant as your proposal. I don't think it needs to hack sys.modules to work as you describe, but that might also be the only way to have global effects.
So you need to make sure that you import all of the submodules before creating a
Preset
I didn't realize that was a problem, but it's definitely worth documenting. Care to open a PR?
This works for me!!
import sys
import importlib
from importlib.machinery import ModuleSpec
from presets import Preset
from ast import Module
class Preset(Preset, Module):
def __repr__(self):
return f'<Preset \n\tof={self._module} \n\tdefaults={self._defaults}>'
class PresetsFinder(importlib.abc.MetaPathFinder):
def __init__(self, *a, **kw):
super().__init__(*a, **kw)
self.preset_modules = {} # tracks modules that we've wrapped
self._already_matched_preset = None # prevents circular calls
def is_preset(self, parts):
'''Checks if a module path (split on '.') has been implicitly loaded already.'''
parts = tuple(parts)
# if 'librosa' in parts:
# print(parts, self.preset_modules)
# find any matches (match starting at beginning, obvs)
return next((name for name in self.preset_modules
if name == parts[:len(name)]), None)
def find_spec(self, fullname, path, target=None):
# if 'librosa' in fullname:
# print('find_spec', fullname, path, target)
parts = tuple(fullname.split('.'))
if parts:
# i.e. from presets import librosa <<<<
explicit_preset_import = parts[0] == 'presets'
# i.e. from presets import librosa
# import librosa.display <<<<
preset_parent = self.is_preset(parts)
implicit_preset_import = (
not explicit_preset_import
and self.is_preset(parts))
if explicit_preset_import:
parts = parts[1:] # cut off 'presets'
if parts and not self._already_matched_preset == parts and (
explicit_preset_import or preset_parent):
# prevent inf recursion, get module the normal way
self._already_matched_preset = parts
spec = importlib.util.find_spec('.'.join(parts))
self._already_matched_preset = None
if spec:
# share dispatch and defaults between the same package
if explicit_preset_import and not preset_parent:
preset_parent = parts
self.preset_modules[preset_parent] = {}, {} # dispatch, defaults
dispatch, defaults = self.preset_modules[preset_parent]
# monkey patch in the Preset call
# I didn't want to create my own loader, because I
# wanted it to reuse whatever the logic of the original loader
# used, and just tack this on at the end.
#
# There has to be a better way to do this, but it seems like
# this gets the job done.
_create_module = spec.loader.create_module
def create_module(spec):
spec.loader.create_module = _create_module # replace method to avoid infinite recursion
module = importlib.util.module_from_spec(spec)
module = Preset(module, dispatch, defaults)
return module
spec.loader.create_module = create_module
return spec
sys.meta_path.insert(0, PresetsFinder())
if __name__ == '__main__':
# import librosa
# print(111, librosa)
# from librosa import display
# print(222, librosa)
# import librosa
# print(333, librosa)
# import numpy
# print(555, librosa)
from presets import librosa
# print(444, librosa)
# import librosa
# print(555, librosa)
# import librosa.display
# print(666, librosa.display)
# print(66666, librosa)
print(librosa)
librosa['n_fft'] = 2**14
print(librosa)
import librosa.display
print(librosa.display)
Nice package! I've been using something similar with
Sacred
and was wondering if I could use something similar to change a package's defaults. And then I stumbled across this!So I'm thinking about changing the defaults of librosa program-wide, but seeing as I need to wrap it, I need to import everything through wherever I define the Preset.
This is only a minor inconvenience so it's not a big deal, but I was wondering if there would be a way to tap into python's import system to override it everywhere.
It doesn't quite work because it breaks subsequent imports of submodules, which makes sense, so I think we'd have to extend Python's
Module
class (I believe I've seen it underast
? but the docs are lousy) so that it can still read a Patch as a module.Anyways, I'm not saying that we have to do anything about this, but I thought it could be a cool/convenient feature.
Bonus: I wonder if we could use Python's Import Hooks to do something like this:
not even sure if that's a good idea. but it's an idea 🤷♀️
I think awkward thing is that, if the Preset replacement is called after anywhere else imports the package, then they'd be using the original module reference. So maybe explicit is better.