Docs

[Shinyzenith]

Hi! Is it possible to make our own compositor with pywm? I find wlroots incredibly hard to understand and if this can help then I'm more than interested in trying it out.

Is there any possibility that documentation/examples will be added ?

[jbuchermn]

Hi,

it's certainly possible to build your own compositor with it. pywm is meant to be quite generic to allow for different compositors (although not as generic as wlroots itself).

At the moment I don't have time to provide pywm with a comprehensive documentation... but there are example, you can check out https://github.com/jbuchermn/pywm-fullscreen for a basic setup and trivial compositor and take a look at https://github.com/jbuchermn/newm for more details. Also check out the python sources of pywm (it's not much, compared to the c part).

Feel free to ask specific questions here, I'm happy to help.

[Shinyzenith]

Hi! Thank you for the reply!! However things have changed since I wrote this issue, I now somewhat understand wlroots, the initial push is what I needed. I'm writing my own compositor in zig along with nim bindings for wlroots.

However pywm and pywlroots still intrigue me. I'll test all three out and find what fits my workflow the best.

[cben]

After a few days flailing around newm source there were basic things I still didn't understand; reading the Python part of pywm helped a lot :+1: It is indeed pretty readable without reading the C.

Suggested minimal reading order:

1. damage_tracked.py
2. pywm.py
3. pywm_widget.py
4. pywm_view.py

Meanwhile I'll start dumping some notes on what I learnt here (mistakes possible)...

[cben]

C <-> Python control flow

The C interface is remarkably narrow:

from ._pywm import (
    run,
    register,
    damage
)

These are ignorant of Python classes; they mostly communicate by python primitive types (strings, numbers, tuples...). Views & widgets are referred to by "handle" integers.

They manipulate global C state, so being wrapped by a PyWM object instance is mostly for stylistic reasons — there ought to be only one PyWM instance.

The main event loop run() is in C and calls register()ed PyWM methods e.g. _update_view. The PyWM instance keeps track of self.widgets and self.views objects, and dispatches the relevant callbacks to them, according to numeric handle it gets from C.

Except for damage() func, the initiative for all communication is on C side. "Don't call us, we'll call you". [I guess this has to do with Wayland's "every frame is perfect" mantra?]
This is getting amusing with things like:

    def _query_new_widget(self, new_handle: int) -> int:
        if len(self._pending_widgets) > 0:
            ...

    def _query_destroy_widget(self) -> Optional[int]:
        if len(self._pending_destroy_widgets) > 0:
            return self._pending_destroy_widgets.pop(0)._handle

where Python can't tell C create/destroy widgets, it can only queue the requests and wait for C to ask...

This kind of buffering on Python side is pervasive, and allows Python code deriving from pywm base classes to run in multiple threads and to pretend it can make changes at any time, mostly forgetting about this inversion of control.

Python->C data flow, "DownstreamState"

C requests fresh data by calling update/update_view/update_view callbacks, which get routed by to _update() methods on correct instance PyWM/PyWMWidget/PyWMView.

Many things are buffered in _pending_foo attributes on the instances. Some holding single value, some arrays serving as a queue.
These tend to be cleared/pop(0)ed as soon as they are passed down to C.

Some data are wrapped in ...DownstreamState objects. These are computed on 2 levels:

1. process() method can do heavier computations and completely replaces the down_state instance, but it's called only when damaged:

       if self.is_damaged():
           self._down_state = self.process()   
       # not obvious from its name: note that is_damaged()
       # clears the damaged flag.

   (that's one reason it's called "state", because it may persist between _update calls. Another is that it remains available as self._down_state attribute, useful also for other threads.)

2. Last thing _update does, each time, is call _down_state.get(...) with some current params (including self._pending_foo!). .get() methods mostly assemble things to a tuple of the form C likes, but they may also do some lighter computations where freshness is preferred to caching.

The role of .damage() tracking

DamageTracked mixin implements a per-instance self._damaged flag, plus optional ability to recursively mark self._damage_children. (Views are children of PyWM instance; widgets too, by default, but can override_parent.)

IIUC, the purpose effect of marking something _damaged is forcing process() to compute a new DownstreamState.

So wm.damage(propogate=true) is a quick way to force pretty much everything to be refreshed.

There are also PyWM methods enter_constant_damage() / exit_constant_damage() / wm.damage_once(), which call C damage().

TODO: does any of this relate to Wayland's concept of surface damage?

TODO: animations and reducer()

These are extra concepts added by newm, I haven't grokked them yet.

[cben]

C->Python data flow, events, "UpstreamState"

In general, pywm's private callbacks for events e.g. PyWM._modifiers first save data in attributes, then call the public abstract method e.g. self.on_modifiers(self.modifiers, last_modifiers). This way you can peek at last received data at any time (e.g. on_key will probably want to look self.modifiers), including from other threads.

PyWMOutput

These are entirely passive objects representing output metadata received from C. Saved in self.layout array.

They have no methods to modify layout; instead it may be indirectly influenced by config, open_virtual_output, close_virtual_output fields on PyWMDownstreamState.

Views: PyWMViewUpstreamState -> python -> PyWMDownstreamState

PyWMView._update() is bi-directional :arrows_counterclockwise:

It gets lots of data from C. Some it stuffs into attributes e.g. self.title, some wraps in PyWMViewUpstreamState instance. 2 instances of it are kept — self.last_up_state and current self.up_state — allowing comparisons and triggering some extra abstract methods e.g. self.on_map(), self.on_focus_change().

• It calls process() to compute new down_state when the view is new OR is_damaged() OR when up_state changed compared to last_up_state.
• As usual, it always calls down_state.get(), returns the result to C, clearing the attributes it sent like _down_action_focus.

[cben]

Widget lifecycle

sequenceDiagram
    participant C
    participant wm as PyWM instance
    participant down_state as PyWMDownstreamState instance
    participant widget as SomeWidgetClass instance

    note over wm: create_widget(WidgetClass, ...)
    wm -->> widget: constructor(wm, ...)
    activate widget
    note over wm: _pending_widgets.push()
    C -) wm: _query_new_widget(handle)
    note over wm: _pending_widgets.pop(0)
    note over wm: _widgets.push()
    wm ->> widget: set ._handle
    wm -->> C: confirm handle got used
    activate C

    note over wm, widget: ... Widget exists ...
    loop
        C -) wm: _update_widget(handle)
        wm ->> widget: _update()
        opt if damaged
            widget ->> widget: process()
            widget -->>+ down_state: constructor(...)
        end
        widget ->> down_state: get(...)
        down_state -->>- C: data
    end

    note over widget: destroy()
    widget ->> wm: widget_destroy(widget)
    deactivate widget
    note over wm: _widgets.pop(...)
    note over wm: _pending_destroy_widgets.push()
    C -) wm: _query_destroy_widget()
    wm -->> C: handle
    deactivate C

[cben]

Another neat trick allowed by process() / down_state.get() split is that you may switch different downstream classes dynamically :bulb:.

• IIUC newm does this during animations, newm/interpolation.py defines several sub-classes ...DownstreamInterpolation whose get() methods compute momentary values during animations.
  (though the division of work there is perhaps a bit different? I got confused by _process vs process and the inheritance)