Dev: hot-reloading newm's own source?

cben commented 2 years ago

First, thanks! newm is both innovative and (together with pywm) probably the easiest starting point that now exists for hackable wayland WM :clap:

Do you have a recipe for reloading newm source itself?

Things I found out today:

I can reload() modules and log stuff from config file. Poor man's REPL — edit config, press hot-key, see output in tail -f $HOME/.cache/newm_log | grep -v 'GL_INVALID_VALUE|TIMER|DEBUG|wm_output.c' (I tweak the command to whatever I consider noise at the moment) :rofl:
~~It's convenient to do things in on_reconfigure function so the output appears closer to the end.~~ (OTOH if I were hacking on logic such as _setup_widgets I'd maybe want to reload it before it runs? EDIT: see below, this proved better)

As usual in Python, reload()ing modules is not enough when other modules:

already have instances of them.
worse, have code running in threads!
have copies e.g. via from mod import var, Class.

At least for widgets, (1) (2) are luckily covered by layout.py stopping threads, destroying and re-creating instances.

(3) is a problem, specifically from .widget import TopBar, BottomBar, Background, Corner, FocusBorders means layout modules would still use references to previous class object to re-create instances. I now came up with a quick hack in config file that partially monkey-patches such copies, enough to let me modify widgets appearance and seeing the results without newm restart:

def on_reconfigure():
logger.warning('~' * 120)

import sys, importlib

mod_names = ['newm.widget.bar', 'newm.widget.focus_border', 'newm.layout']
for mod_name in mod_names:
    if mod_name in sys.modules:
        importlib.reload(sys.modules[mod_name])

for mod_name in mod_names:
    for name, obj in vars(sys.modules[mod_name]).items():
        source_mod_name = getattr(obj, '__module__', None)
        if source_mod_name and source_mod_name != mod_name and source_mod_name in sys.modules and 'newm' in source_mod_name:
            current_obj = getattr(sys.modules[source_mod_name], name, None)
            if type(obj) == type(current_obj) and obj is not current_obj:
                logger.warning('RELOAD PATCHING %s.%s \t<- %s = %r', mod_name, name, source_mod_name, current_obj)
                setattr(sys.modules[mod_name], name, current_obj)

logger.warning('_' * 120)

I'm sure a better stuff is possible, there are auto-reload modules that do much more intelligent patching...
Do you have a ready setup that you're using? Or do you always start-newm?

CRAG666 commented 2 years ago

This is my configuration if it is useful https://github.com/CRAG666/dotfiles/blob/master/config/newm/configV3.py

jbuchermn commented 2 years ago

That's a very interesting thought!

To be honest, I just restart newm over and over when debugging stuff, but specifically for sth like the widgets, or "higher-level behaviour", doing what you do in your example could be very worthwhile.

What kind of auto-reload modules do you have in mind?

Also, another path might be to make newm itself as minimal as possible and move a lot more behaviour into config files, which is sth I've done e.g. for the XF86 keys (volume control, dimming the screen, ... - that was all once coded into KeyProcessor). A lot of the functions that Layout offers, could probably be moved to helper modules and imported directly from configuration

cben commented 2 years ago

Tried IPython.lib.deepreload.reload(), it messed up the import machinery and I had to restart. Anyway that doesn't address (3).
The real magical one is https://ipython.org/ipython-doc/3/config/extensions/autoreload.html which does impressive stuff like patching function objects in-place. But that's entangled with IPython, don't know if it can be used stand-alone.
Correction: layout.py does not re-create ALL widgets, e.g. self.focus_borders instance is not destroyed. EDIT: though it does get .update() method call to re-create FocusBorder instances.
The top-level Layout instance is long-lived so its methods do not get updated. Possible fix: https://github.com/jbuchermn/newm/commit/d11cb985e24f7f1dce7f6f56daf13ae772e99cbe but that only handles that file. => Extracted smaller #122.
Correction: on_reconfigure is too late, best to reload early in config file.

I extended my recipe to patch instances in all files, and so far it seems to hold :crossed_fingers: :

def reload_modules():
    logger.warning('~' * 120)

    import sys, importlib, gc
    logger.warning('newm.layout._INSTANCE = %r', getattr(newm.layout, '_INSTANCE', None))

    mod_names = ['newm.widget.bar', 'newm.widget.focus_border', 'newm.layout', 'newm.run']
    # This also seems to work but reloads A LOT of stuff, so maybe it's safer listing what I touch?
    #mod_names = [n for n in sys.modules if 'newm.' in n]

    orig_vars = {}
    for mod_name in mod_names:
        orig_vars[mod_name] = vars(sys.modules[mod_name]).copy()
        importlib.reload(sys.modules[mod_name])

    for mod_name in mod_names:
        logger.debug('?? %s', mod_name)
        for name, orig_obj in orig_vars[mod_name].items():
            source_mod_name = getattr(orig_obj, '__module__', None)
            if source_mod_name and 'newm.' in source_mod_name:
                current_obj = getattr(sys.modules[source_mod_name], name, None)
                if current_obj is None:
                    continue
                if current_obj is orig_obj:
                    logger.debug('  same obj %s = %r at %r', name, orig_obj, id(orig_obj))
                    continue

                logger.debug('  NEW OBJ %s = %r at %s -> %r at %s', name, orig_obj, id(orig_obj), current_obj, id(current_obj))
                # Update `from foo import Class` copies
                logger.warning('    PATCHING GLOBAL %s.%s \t= %r', mod_name, name, current_obj)
                setattr(sys.modules[mod_name], name, current_obj)

                # Update instances to point to new class object.
                if source_mod_name == mod_name and isinstance(orig_obj, type):
                    for ref in gc.get_referrers(orig_obj):
                        rep = repr(ref)
                        if len(rep) > 60:
                            rep = rep[:60] + '...'
                        logger.debug('    type %r ref %s', ref.__class__, rep)
                        if ref.__class__ is orig_obj:
                            logger.warning('    PATCHING INSTANCE %r.__class__ \t= %r', ref, current_obj)
                            ref.__class__ = current_obj

    logger.warning('_' * 120)

which (for those 4 files) does something like:

?? newm.widget.bar
    PATCHING GLOBAL newm.widget.bar.Bar     = <class 'newm.widget.bar.Bar'>
    PATCHING GLOBAL newm.widget.bar.TopBar  = <class 'newm.widget.bar.TopBar'>
    PATCHING INSTANCE <TopBar(Thread-1083, started 140462168409664)>.__class__  = <class 'newm.widget.bar.TopBar'>
    PATCHING INSTANCE <TopBar(Thread-1084, started 140462160016960)>.__class__  = <class 'newm.widget.bar.TopBar'>
    PATCHING GLOBAL newm.widget.bar.BottomBar   = <class 'newm.widget.bar.BottomBar'>
    PATCHING INSTANCE <BottomBar(Thread-1081, started 140463441081920)>.__class__   = <class 'newm.widget.bar.BottomBar'>
    PATCHING INSTANCE <BottomBar(Thread-1082, started 140464370665024)>.__class__   = <class 'newm.widget.bar.BottomBar'>
?? newm.widget.focus_border
    PATCHING GLOBAL newm.widget.focus_border.FocusBorder    = <class 'newm.widget.focus_border.FocusBorder'>
    PATCHING INSTANCE <newm.widget.focus_border.FocusBorder object at 0x7fc077157460>.__class__     = <class 'newm.widget.focus_border.FocusBorder'>
    PATCHING INSTANCE <newm.widget.focus_border.FocusBorder object at 0x7fc0771546a0>.__class__     = <class 'newm.widget.focus_border.FocusBorder'>
    PATCHING GLOBAL newm.widget.focus_border.FocusBorders   = <class 'newm.widget.focus_border.FocusBorders'>
?? newm.layout
    PATCHING GLOBAL newm.layout.TopBar  = <class 'newm.widget.bar.TopBar'>
    PATCHING GLOBAL newm.layout.BottomBar   = <class 'newm.widget.bar.BottomBar'>
    PATCHING GLOBAL newm.layout.FocusBorders    = <class 'newm.widget.focus_border.FocusBorders'>
    PATCHING GLOBAL newm.layout.TKeyBindings    = ~TKeyBindings
    PATCHING GLOBAL newm.layout._score  = <function _score at 0x7fc0c115cee0>
    PATCHING GLOBAL newm.layout.Animation   = <class 'newm.layout.Animation'>
    PATCHING GLOBAL newm.layout.LayoutThread    = <class 'newm.layout.LayoutThread'>
    PATCHING GLOBAL newm.layout.Layout  = <class 'newm.layout.Layout'>
    PATCHING INSTANCE <newm.layout.Layout object at 0x7fc0e4606b30>.__class__   = <class 'newm.layout.Layout'>
?? newm.run
    PATCHING GLOBAL newm.run.Layout     = <class 'newm.layout.Layout'>
    PATCHING GLOBAL newm.run.run    = <function run at 0x7fc0c115cf70>

cben commented 2 years ago

caveat: Order matters with inheritance

[ ] If the list of reloaded files contains both some base.py and subclass.py, order matters. If you first execute subclass.py:
```
from base import BaseClass

class SubClass(BaseClass):
  ...
```
then a new instance of SubClass is created, but SubClass.bases is set at that moment to point to the old BaseClass. Instances of old SubClass will be patched to point to new SubClass, but IIUC their method lookup chain will still call methods from old SubClass :confused:

For now, safest to reload things in bottom-up order.

Future directions to explore:
- patch __bases__?
- patch old class object's dict so new class object is interchangable with to old class (not sure if sane or increasingly confusing)?

CRAG666 commented 2 years ago

@cben How does the reload config work now? Why do I try to update and change in on_reconfigure or view and it doesn't seem to change?

cben commented 2 years ago

I'm not sure I understand your question right, can you elaborate where/what exactly are you editing?

tips if you tried my recipe above and newm/**py edits have no visible effects:

add new logging to the code you edited to be sure.
on_reconfigure is too late if you touch logic that runs after config reload; if that's the case than reloading twice helps — and moving to top level of config script instead of on_reconfigure makes it more predictable.
Try looking at "PATCHING INSTANCE" / "PATCHING GLOBAL" log lines to see whether what it did makes sense / is sufficient to replace the code you're editing.

I haven't tried editing view.py yet if that's what you mean.

There are certainly gaps in this approach. Python is inherently not optimal for this: 1, 2, though I know internals well enough to deal...

cben commented 2 years ago

if you are asking how it works without my kludge:

newm's current config reload flow

The shortcut or newm-cmd update-config trigger this:
https://github.com/jbuchermn/newm/blob/d5831bbca8815fa77c14616c778d3398116b0808/newm/layout.py#L1212-L1217
- Your config file is reloaded. Top-level code is executed immediately; can also define callbacks for later.
- it calls _setup()
- which calls _setup_widgets() which destroys and recreates most Widget instances.
- it also re-creates some other stuff.
- This part: https://github.com/jbuchermn/newm/blob/d5831bbca8815fa77c14616c778d3398116b0808/newm/layout.py#L526-L528 calls into pywm setting _pending_config attr that will be sent to C next time it calls _update callback.
- self.damage() will cause process() to run again and recompute down_state. @jbuchermn would it be useful to self.damage(propagate=True) instead to trigger this pretty much everywhere?
- After all this, your on_reconfigure() function, if defined, is executed.

Additionally, all config params are used like this:

conf_enable_dbus_gestures = configured_value("gestures.dbus.enabled", True)

which returns a callable; wherever code wants to actually use the value it calls it e.g. if conf_enable_dbus_gestures(): so it always gets a fresh value.
And config params that specify a callback get used like conf_foo()(args).

jbuchermn / newm

Dev: hot-reloading newm's own source? #120

caveat: Order matters with inheritance

newm's current config reload flow