dreamcat4 commented 1 year ago

Transcript: of Skippy Testing (2) Videos - master branch, 11-july-2023

skippy videos folder - on google drive all videos
Link to: RAW markdown version. Of this full transcript
- To keep the markdown formatting: if doing copy/paste take from here --> to branch out each to separate issues

Overview

TLDR is: well... just watch the darn video(s)!
- There are 2 total videos. video 1 and video 2
- Everything else is: just sub-sections taken from those 2 full length videos
- However there are also 2 errata points (errors). They are documented within the body of this transcript here.
- There are also extra details / explanations, and links to further resources etc. All included as relevant sections.
- Each section cross referenced / timestamped to each various individual part(s) of the videos.
Format
- This is Full Transcript of all what is covered (multiple entirely different things)
- Items in Transcript are covered chronologically in same order as they appear in the videos.
- Probably requires 4k screen to view acceptably
- Otherwise not be able to see / view things. if too small / not clear
Content
- Videos are organized by multi-monitor config (since cannot change monitor positions within the same video)
- The individual bugs AND features are organized (within each video) by the main skippy features or modes (tooltip, paging, boxy, switch)
- This meta-issue references a full video link, for the entire session, and also entirely the full transcript (everything)
- Why? --> Because just was a lot easier / more time efficient way: to demo all testing together in 1 go (as 1 extended session).
- Can then branch off from here / raise each individually as seperate issues. For each individual matters.
- Can start issue names / make issue tags like: [multimon,paging] Issue Title etc.
- This being overall main index to these issues / tracking issue (or meta issue)
- Can just copy-paste sub-section(s) from this full transcript --> already documented to become each individual issue
- But don't have to! (for everything) ...maybe if you don't consider it as a bug, or cannot / don't wish to work on any specific sub-section(s). All fine.
- Maybe we can also cut up the video into shorter sections. For each individual part(s). (to put into individual issues).
  - Cmdline is: ffmpeg -i in.mkv -ss 00:00:00 -to 00:00:00 -vcodec copy -acodec copy out.mkv
Reproducability
- Some of these observed bugs only occur in specific circumstances
- So probably requires effort to make similar setups (maybe 2 total monitors... is the minimum needed)
- However (if can somehow get a similar setup)... THEN should be reproducable "pretty reliably" ?
Bugs priority
- Not all of these bugs may actually be worth fixing... but some are
- It also depends on difficulty level to make certain specific fixes
- Or might be several alternative solutions can work around. In other way(s)
- There is NO HURRY to fix any of these bugs: nor is any specific priority-ordering between them!
- However (clearly) some can be easy / tasty! And more tempting
- Other matters - i just run out of time. And cannot participate on skippy (for lack of total time to spend on it)

Video `#1`

video-1.reverse-Tee.FULL.mkv

Multi monitor configuration `#1` = aka "reverse Tee"

Video #1 - Segment #1 - 00m14s - 01m26s - What is 'reverse Tee' multi-monitor layout?
- video-1.reverse-Tee.segment-1.mkv
This is my main and everyday monitor arrangement
- The relative virtual positions of each monitor matches their relative positions in the real room. So it's designed the mouse spills onto the neighbouring monitor.
- The letter "P" marks the primary screen (an ultrawide, again not a conventional setup).
- "reverse tee" because it is like the tetris block letter "T" but upside down.

   [_]
 [_P_][_]

Skippy Version under test

Video #1 - Segment #2 - 02m28s - 03m32s
- video-1.reverse-Tee.segment-2.mkv
- Link to exact Skippy version being compiled, installed, tested: https://github.com/dreamcat4/skippy-xd/commit/5455985c674a1d1345b33ed0a95e3634f8d1c4d4

Tooltip issues (all modes)

Video #1 - Segment #3 - 03m32s - 13m34s
- video-1.reverse-Tee.segment-3.mkv
Tooltip relative co-ordinates position
- The tooltip for a highlighted clientwin preview is
- Supposed to be displayed either
  - Directly above
  - Directly below or
  - Directly on top of the clientwin preview
- However it's being displayed at some offset co-ordinates
- This suggests that --> the origin reference for the xorg 'screen' or canvas that these coords calculations are from the top monitor? starting position
- The precise nature of this bug is not known, however it also may exhibit not just for the tooltip itself... but also in other modes.
Update:
- This seems to indeed missing some necessary co-ords offset. To be added to existing (x,y) geometry calc
- Errata #1 - At 06m37s into Video #1 I incorrectly fail to draw on the screen where is the xorg screen canvas origin. Correction: It's actually located here (screencap)
- That within the whole entire xorg screen canvas (spanning all multiple monitors)
- Each monitor within the canvas has a necessary offset (x+y). To locate it within the wider canvas
- Skippy seems to be assuming: (at least for case of tooltip, not considering the other modes)
  - That monitor offset is always? (0,0), or it does not have such concept of monitor offset pre-existing
  - But this offset is depending upon which specific monitor the mouse was hovering in, at the time when:
  - skippy is either launched, or rather: at the time when daemon was triggered, and mainwin gets to be drawn / mapped from the main loop
Observation:
- We might expect or wonder if this 'wrong offset for tooltip position' is to do with primary monitor... but no:
- The offset does not seem to assumed taken of primary monitor because... this bug does in fact also happen on primary monitor too
- If skippy is invoked onto the primary monitor, but that monitor is also itself at some non-zero (x,y) offset too... (of the total screen canvas)
- I try to demonstrate (in this video) how to do those necessary geometry calculations on the tooltip when it's displayed
- By drawing an imaginary line downwards beneath the position where the tooltip wrongly appears
- And then cross checking that position with the value of that monitor's screen canvas offset (=the monitor skippy is currently being displayed onto)
Cmdline Tools:
- Here are some resource links to the necessary cmdline to run... to reveal such xorg screen canvas offsets
- display screen canvas x,y coo-rds in a terminal
- explains about screen canvas - multi monitor positions
- xrandr cmdline to move around relative positions of monitors
- xrandr --query will tell you these necessary screen canvas offsets for all monitors. On this line being output by the command:
  - DisplayPort-0 connected primary 3440x1440+0+1080 (normal left inverted right x axis y axis) 798mm x 335mm
  - We can see the 4 values width,height, x-offset, y-offset so here the last 2 values = +0+1080 at the end
  - It means: the monitor is x-offset starts at pixel 0 (the origin) on the leftmost side of the total screen canvas
  - then the last y starting offset means from the top of the screen. That the top-left corner begins at y=1090, so some ways down along the total screen canvas
    - however skippy assumes (always?) the screen canvas offset is 0,0 - which is only sometimes true. Depending on the relative positions of monitors within the screen canvas
  - DisplayPort-1 connected 1920x1080+1520+0 (normal left inverted right x axis y axis) 509mm x 286mm
  - In this example, the last 2 (x,y) offset coords are 1520+0... this means: the y-offset = 0 so this monitor is at the very top
  - However this time the x-offset is 1520 pixels into the screen. I did some simple match to determine that indeed... 3440-1920 = 1520. Voi-la
How to fix? ...not sure!
- Clearly, we also want to find out what xlibs / libraries api will tell us these screen canvas offsets, and which monitor skippy is getting displayed onto
- Not myself familiar with the necessary xorg apis. Or how to find out the necessary infos. (for within skippy program)
- We need to know:
  - Firstly, which monitor skippy mainwin routine is being drawn onto / mapped on (from mainloop or other places)
  - Then after knowing which monitor, to query the screen canvas x,y offsets the monitor's relative position
  - I have not looked anything into the code felix! So leaving this matter for you to investigate (at your leisure / as you see fit)
- The assumption is: that if xrandr program knows, then maybe skippy can find out too
  - Except the main uncertainty: if skippy can determine which specific monitor it's getting displayed onto?
  - IDK! ---> but surely we are entirely at the mercy of: xorg apis. hahaha
  - However what gives me hope: is that clearly the existin mainwin and clientwins (and transparency / highlight layer) are already being rendered to the correct monitor. So that would indicate the expected necessary missing geometry is already being done elsewhere in the code (for those other things). At least that would be a naiive expectation (it requires searching the code).
New Tooltip Question (not-a-bug):
- Can tooltip become show-all? (maybe not need to be individual option)
- At the moment, only the currently highlighted clientwin preview displays tooltip for
- However (potentially)... in future we could display all tooltips under each preview (at all times) ?
- The tracking issue was: Tooltips under window previews #69
- Maybe? the legacy skippy code only ever assumed and passes around 1 tooltip working object
  - Yet to implement this feature ("display all tooltops" / show all)...
  - It might require some new data structure, or "array/list of multiple tooltip objects", to be rendered as a layer
  - Which then incurrs extra data management risk/burden, to deal with allocating, freeing, etc.
  - Conclusion is: (while desirable feature) it may be best to defer / delay.
  - I.e. work on these other bugs first, then checkpoint and tag the code (as safe). Before attempting (potentially unsafe code for this)

Paging mode issues

Video #1 - Segment #4 - 13m32s - 20m14s
- video-1.reverse-Tee.segment-4.mkv

Note: I don't use paging mode (for myself), just reporting these things... I don't wish to go back and re-test. (Because it's takes time away from doing other things...)

Number of virtual desktops seems too many?
- Skippy seems to thinks the number of virtual desktops = 9 here
- Actual number of virtual destops = 5
- So for paging layout.. might expect a 2x3 style grid of 5 or 6 ? IDK
- This is giving a 3x3 grid instead. With the extra ones been blank
Indexing of virtual desktops starts at 0
- However a lot of DE/WMs might start at number "1", instead of "0"? (at least KDE here does)
- Might be nice to consider? I'll leave you to deal with that, since it's for Paging feature.
- Update: this is indeed correct starting offset value being reported by xorg as the desktop number.
However: Can virtual desktops be named with a text label?
- Already has pre-existing issue raised (by felix)
- Cmdline to show virtual desktop names: wmctrl -d Example
- Didn't bother to check what xlib / C libraries provides it (for retrieving these desktop text labels) but here is a naiive link
- However probably going to require new extra struct member, to point to additional char* string
- Maybe some new configuration flag to turn on/off
- And to make sure the string is memory managed properly (alloc and free so... can be some level of risk to add)
  - Conclusion being: (like tooltip-show-all) is something perhaps safer to defer at end of a block of work, since involves memory management
  - and git tag checkpoint the master before starting those work. (To give a safe place to rollback to in the event of potential breakage)
Co-ordinates offset bug (for paging preview elements)
- This is on "reverse tee" Multi-monitor layout
- This bug exhibits a bit similar to the other "tooltip offset coords" bug
- However they are different, maybe there is a clue here ?
- We can see there are 3 layers rendered
  - 1. Tooltip layer. Is offset some ways above each preview image
  - 1. Previews images of each virtual desktop
  - 1. Highlight shadow / transparency layer, for each virtual desktop
- But it's the 3. Highlight shadow that is correctly positioned to the currently in focus monitor
  - This you seemed to work on most recently. The other ones 1-2. are off different amounts.
Cancel on Esc key (not sure if correct here?)
- Always switches to the 1st desktop (='desktop 0')
- It seems like: when in paging mode, you can move (advance) the highlighted preview
- Then press Escape key --> The pager then always switches over to the 1st desktop (=desktop 0)
- However when the pager was initially launched. It was from the 2nd, 3rd desktop etc.

Boxy layout mode issues (expose mode)

Video #1 - Segment #5 - 20m18s - 24m43s
- video-1.reverse-Tee.segment-5.mkv

Note: I don't use the BOXY layout (for myself), just reporting these things... I don't wish to go back and re-test. (Because it's takes time away from doing other things...)

Example: Success case for how Boxy layout is "supposed to work"

"Diagonal Pattern" traversal between windows (above a certain number of open windows)

Isn't the layout supposed to be (generally speaking) grid aligned rectilinear?
- Might be a bit of a grey area - i do not wish to get caught up in figuring this all out!
- The assumption here is: That "boxy layout" is "somewhat" grid-aligned (aka "rectilinear")... so like a crossbar
- I believe the term you used was "groupings", and/or involving some representation of relative position.
- Observed behaviour (but not always) - boxy layout seems to have diagonal \ relationships between prev-next clientwins
- So (for example) when transitioning thru the list, the progression seems meandering \/\/\/\
However: bug does not always exhibit!
- Seems better behaved on single monitor layouts

The general observation is that: this "boxy layout" mode looks worse when

The current desktop has "too many" open windows
- This is considerably more likely to occur on multiple monitor layouts
- For example: having 3 monitors = potentially 3x the number of possible open windows (on a single desktop)
- Ideal (theoretical) best solution would be the way KDE handles expose:
- KDE will displays an Expose view on all of the monitors, and (for each one), shows only windows on that monitor
- However this would be difficlt to implement in skippy code, since skippt displays only 1x mainwin, at any given time
  - Therefore would take some substantial work for making 3x mainwin, so it is not recommended to bother
  - Also would be pretty risky / big changes. So (for a legacy app) can be difficult to justify

Video `#2`

video-2.regular-Tee.FULL.mkv

Multi monitor configuration `#2` = "Tee"

Video #2 - Segment #1 - 00m05s - 03m56s - What is 'regular Tee' multi-monitor layout?
- video-2.regular-Tee.segment-1.mkv
In this layout, the top-most monitor (one above) is then moved to be below the primary monitor.
So the shape is like a standard letter 'T'. Same tetris block but other way up.

 [_P_][_]
   [_]

Paging Mode

Primary Monitor

Video #2 - Segment #2 - 03m10s - 10m22s - Primary monitor
- video-2.regular-Tee.segment-2.mkv
Tooltip relative co-ordinates position
- Seems to be working as expected here? --> Yes
- This makes sense for 1st / PRIMARY monitor, but only when both x-axis and y-axis offset =0 (the 2 last numbers ?+?+0+0)

Paging mode is flickering

Flickering found in PAGING mode (was not documented previously) - 09m17s - 11m14s

Other 2nd, 3rd Monitors

Video #2 - Segment #3 - 11m14s - 24m47s - 2nd, 3rd monitors
- video-2.regular-Tee.segment-3.mkv
We still get some position issue on 2nd, 3rd monitors? --> yes correct
- There is also still same offset bugs in 'regular Tee' multi monitor configs
- For the transparency mask layer, or for the clientwin previews mask layer, (or both IDK)
- And tooltip too. Similar comments as per in previous video #1...
Errata #2: - During 15m00s - 17m21s - Wrong DUT (wrong device under test...)
- During this part of the video, was made a testing error whilst launching paging mode
- Paging mode ended up being opened onto the wrong monitor (the primary monitor again, which was already tested previously)
- Didn't notice this mistake while taking the video, then was observed "no obvious errors", but while expecting some...
- Resulting in a commentary that this was a strange and suspicious result...
- However the monitor that paging mode should have been tested on there was monitor 2 (right most in the 'Tee' shape)...
- So Monitor 2 was never tested. Skipped over here (at least for paging mode).
- However subsequently after the 17m21s mark of (full) Video #2, then Monitor 3 (bottom one) get tested.
  - And for Monitor 3 then the expected coords offset calc errors were then properly observed

Boxy layout mode

Video #2 - Segment #4 - 25m00s - 28m48s - 2nd, 3rd monitors
- video-2.regular-Tee.segment-4.mkv
Same as demonstrated previously (in video #1)
- Not likely to be related to multiple monitors
- Rather total number of clientwins is 'too many'

Other observed issues (unlikely to be multi-monitor related)

Video #2 - Segment #5 - 28m48s - 40m49s - Other 'new issues' or observations
- video-2.regular-Tee.segment-5.mkv
Tooltip is flickering
- Whenever moving between previews
- The flashing seems like a 'double flicker'
- Doesn't matter if keyboard or mouse
Successive alt+tab flickering - mainwin gets redrawn (unmapped, then re-mapped)
- However this becomes (in practical usage) distracting / flashing type behaviour
- It might be due to something of my skippy-xd-runner script
- Since executing the skippy-xd --switch cmdline successively (without runner wrapper) - it does not seem to exhibit that wa
- Not sure / have not investigated this matter
- Otherwise this might instead be related of new behaviour / fixings you added felix
- Since to swap the previous and next windows in the linked list around (first 2 entries?)
- Then it becomes necessary to redraw the whole mainwin, with the new linked list cw's
- It could be due to something else too though, I am no longer so certain best way to solve
- For example, if necessary to add some new configuration option
- Or I have mis-configured config (if this behaviour is a choice under configuration)

felixfung commented 1 year ago

Thanks @dreamcat4 for a very detailed report. It will take me some time to go through your findings.

I have created a quick PR on a bug I knew about also, please try it out.

Meanwhile, I have this thought for a while. You have previously offered ownership, and for a while I felt I am ready to take ownership. I did not raise this topic because you were about to report the bugs and I did not want you to take it the wrong way :)

If you are still inclined, I would be happy to take ownership.

dreamcat4 commented 1 year ago

perhaps the best way forwards will be to:

create a github organisation. with both of us having those necessary roles. this enables some advantages as follows:

distros can be more assured that it is the official and best version of skippy. that is most up to date. that they can trust more a github url like: skippy-xd/skippy-xd
much easier for future contributors and maintainers can be more easily added / future transfers of project ownership after we are no longer anything left to contribute
we can also consider if as a legacy and mature software (for xorg). then if some broader of more general foundation would be interest to keep the end point code. for example freedesktop .org. or otherwise... any one of a number of individual window manager / de's could alternatively take as a place to fork from.

so this i feel helps a lot to get your latest versions of skippy into the acceptance of modern distros and des. however with a good enough balance of not being so chained forever (because we all want to put down a project eventually, to move on). so that any future public position of skippy code really looks less like somebody's pet project. that isn't so easy to trust it won't end up changing hands (at least the download url / source code repo link). over and over.

and i feel this is my steady state position for skippy xd project. after a long enough periof of consideration. another big reason is that there isn't alternative project to skippy out there. for substitute (that is not linked to a specific wm/de or compositor).

so if you can get back to me later about project handover matters. (since i would prefer your buy-in and consent / agreement. and that there is no set timeframe for that move, it requires some overhead to move this repo over, and all of the issues, wiki etc.).

anyhow back to these bugs:

1 more possible improvement i did not mention before:

when adding or removing a monitor. then screen canvas changes
if skippy-xd --daemon is already loaded. then it might be possible to subscribe and pick up onto some xorg notifications those screen canvas changes
this then means user does not have to manually kill / restart background running skippy daemon
this also may exhibit as a bug for: when user is logging into their x session. because extra monitors may be getting initialized during / after certain login startup items. (in fact was case here in kde)

so this is similar matter. in related code. to maintain updated screen canvas origin points. between skippy invocations that is then mapping / unmapping the mainwin with stale xorg screen canvas data. again --> to make again as it's seperate issue. to track that one.

dreamcat4 commented 1 year ago

Another update / retrospective. For to try to help to prioritize (which) bugs. This one:

Video #2 Paging mode is flickering

Flickering found in PAGING mode (was not documented previously) - 09m17s - 11m14s (in the 2nd video)

It's perhaps instead some longstanding bug due to (and only exhibited on) AMD gpus, with the latest AMD open source driver (navi 1st generation / 5700xt).
I say this because a similar looking flickering also occurs from time to time (on my system here) with certain other specific GUI applications
For example Bitwig, which is known to do all of it's own UI drawing and draw calls. (thus putting it into a similar category as skippy, and not being GTK or QT based app).
Therefore, that 1 is recommended to ignore. Or de-prioritize. In particular if you cannot reproduce locally on other non-AMD gpu hardware.
If another user comes along and reports a similar general flickering (which is over the whole application window, rather than a specific UI element). Then would be worth to check and ask them to provide details of the graphics card. To determine if it's not only ever occur solely on AMD gpus
But for now: the recommendation is to ignore that one. As potentially might not actually be a fixable issue, then those time can be better spent elsewhere / on other matters.

felixfung commented 1 year ago

The flickering in paging is because each desktop window is unmapped, then rendered, then mapped in each refresh cycle. The reason for this to retain "live preview": e.g. when a video is playing during paging, it should show to be playing, rather than static. In my testing, which is single monitor, and I tested on many environments, there is only occasional mild flickering, which is to me completely acceptable. Sometimes, but not always, in GNOME and KDE, there is significant flickering. According to my very limited knowledge of X11, this is the best I can do.

Unless we turn on lazyTrans option. I think a more descriptive name for this config option is to rename pseudoTransparency = !lazyTrans. When lazyTrans = false, skippy assume there is no picom or similar, and first draws the background, and then the window content, which I think are often called pseudo transparency. This is why we need unmapping/remapping for paging desktop. When lazyTrans = true (pseudoTransparency = false), there is no first drawing of the background, hence the unmapping/remapping may not be needed, and flickering may be avoided.

Currently there is no implementation of the drawing of desktop windows without unmapping/remapping, and when lazyTrans = true, the flickering is so bad it is completely broken #153.

In summary, the flickering is real and right now cannot be avoided, but despite the flickering paging is usable, and perfectly so on environments outside of GNOME and KDE (and single monitor, #126). To elimniate flickering, #153 has to be completed.

felixfung commented 1 year ago

On the topic of whether a feature is usable/imperfection/broken, according to my testing on single monitor hardware, many window managers/desktop environments, pretty much all features are either perfect or very usable. I am still going through your detailed report, but from a brief look your setup skippy is not "broken", it is messed up but usable? (#153 is an example of what I would call broken, unusable)

Also yes, it is already known that multi-monitor setup has bug in terms of offset (#126). And yes that renders many features (e.g. tooltip, clientwin positions) "broken", very bad in usability. Their root cause and fix would be similar, if not identical. Since I do not have multiple hardware monitor I cannot dev/test. Would that be something you can work on?

dreamcat4 commented 1 year ago

ok, these matters are generally accepted... however i do have some observations around it:

if the flickering is worse on paging (than other modes)... can we make the paging mode have it's own independent setting for the update rate?
i dont have any strong opinions here:

so in all other respects i defer to your assessment of these matters all around lazytrans and dealing with these flickering matters. it seems well considered approach so please to go ahead with any of that (as you deem fit / at your leisure, including changing the config setting etc. all fine)

skippy is not "broken", it is messed up but usable? (https://github.com/dreamcat4/skippy-xd/issues/153 is an example of what I would call broken, unusable)

well yes. rather than resort too much vague terms, it was the reason to make the video. and be worth to communicate better. in effect (for myself) the main top (2) priorities for fixing would be:

relative tooltip position offset (multi-monitor)
the screen flashing on successive alt-tab

and that would probably be just enough fix skippy, (but only for myself, not necessarily for other users).

Since I do not have multiple hardware monitor I cannot dev/test. Would that be something you can work on?

yes it seems to be...

however more broadly speaking also requires some extra assessments and researching. so that wider matter (than individual bug) has also been opened up here https://github.com/dreamcat4/skippy-xd/issues/160

although i might also do some quick fixings (in the meantimes). for those (2) specific bugs / issues cited. which is a bit more self-serving matters.

felixfung commented 1 year ago

For your two concerns:

the screen flashing on successive alt-tab

I suspect it is as you say, it has something to do with the runner script and not skippy itself. skippy-xd --switch does not unmap/remap the main window.

relative tooltip position offset (multi-monitor)

The relevant code for tooltip is all within tooltip_move(). The xinerama/monitor dimensions is set in mainwin_update(). With a bit of printf (and mulitple screens!) I expect you to fix this relatively easily?

dreamcat4 / skippy-xd