search

chjj / compton

A compositor for X11.
Other
2.25k stars 500 forks source link

Release schedule discussion #56

Closed richardgv closed 10 years ago

richardgv commented 12 years ago

@chjj:

I created this issue report for discussions related to the release schedule.

I would highly discourage, repeat, highly discourage, to create a release right now with the latest commits from richardgv-dev branch.

  1. b78ab316fd is a experimental commit. It is known to cause drastic increase in CPU usage. I've indicated that point in my commit log and a few replies to issues. Currently I've only got feedback from a user (ichi-no-eda) and I need more information. My plan is to work on revising it in the near future, but I need time to figure out many details in Xorg's region calculation implementation to effectively fix the performance issue.
  2. Our documentation is terribly outdated. My plan is to revise the man pages, README, etc. in a few days. The documentation right now is very misleading, should I say. And we still need to write the typical NEWS files, etc.
  3. My belief is a commit needs to be tested in master branch for at least a week before it could be considered stable (unless it's a change in documentation). It does not sound like too great an idea to release a commit in my experimental branch only a single day after I pushed it.
  4. It's the first formal release, do you think we could afford the risk of showing everybody in the world how unstable the first formal release is?
  5. The VSync feature is currently broken. I need assistance from you or another Xorg expert to fix the feature.
  6. Both plain Makefile and CMakeLists.txt are available right now on compton. We have to make a final decision about which one to use. Also, CPackConfig.cmake probably needs an update.
  7. Considering releasing is a big thing, and some recent changes in code come from me, before you mark any sort of non-urgent release, I would prefer if you discuss with me. If you need some sort of private discussion, I'm sometimes available at #compton on FreeNode.

For now my recommendation is revoke 0.0.1 release immediately, before any package maintainers have time to push it out. I will revoke it myself if I'm not getting any messages from you within 24 hours.

By the way, I'm not sure about what your tag names mean.

chjj commented 12 years ago

A "formal" release might not be the right word, but I guess this is probably as formal as we'll get.

I periodically spend some time looking over the code and playing around with it a little bit. You're right, I haven't been paying as much attention as I should to the issues and development itself.

I was worried you had some changes ready and were waiting around for me to push them to master (my fault). If you feel your changes are ready, you can always message me on freenode or post an issue. If I'm not around, I obviously trust your judgement enough to make releases and commit to master if you'd like.

By the way, I'm not sure about what your tag names mean.

Hmm?

Anyway, I reverted to the last commit on master prior to this push. Since it hasn't been long, I'm guessing not many people have pulled it yet.

richardgv commented 12 years ago

Ah, apparently I made some sort of mistakes in my words, sorry. I didn't mean it's bad to merge the changes to master, considering master branch doesn't mean any sort of stability. And these changes should actually be merged into master, if that means more users will be playing with the recent changes so I get more feedback. I just don't like marking a 0.0.1 released, as I don't want those package maintainers to push the the very unstable recent changes out as a "version 0.0.1" of compton.

If I'm not around, I obviously trust your judgement enough to make releases and commit to master if you'd like.

Thanks for you trust. :-)

By the way, I'm not sure about what your tag names mean.

Hmm?

Oh, I understand those tag names now. They are probably just added a long time ago.

chjj commented 12 years ago

Our documentation is terribly outdated. My plan is to revise the man pages, README, etc. in a few days. The documentation right now is very misleading, should I say. And we still need to write the typical NEWS files, etc.

Totally agree. I really want to update the man page as its usually the first thing people look at. I'll grab the help info and update it.

The VSync feature is currently broken. I need assistance from you or another Xorg expert to fix the feature.

I noticed you were making a lot of GL calls there. Was the XFCE implementation any help? https://bugzilla.xfce.org/attachment.cgi?id=4433 https://bugzilla.xfce.org/show_bug.cgi?id=8898

b78ab31 is a experimental commit. It is known to cause drastic increase in CPU usage. I've indicated that point in my commit log and a few replies to issues. Currently I've only got feedback from a user (ichi-no-eda) and I need more information. My plan is to work on revising it in the near future, but I need time to figure out many details in Xorg's region calculation implementation to effectively fix the performance issue.

My belief is a commit needs to be tested in master branch for at least a week before it could be considered stable (unless it's a change in documentation). It does not sound like too great an idea to release a commit in my experimental branch only a single day after I pushed it.

We really need some kind of staging branch. A development branch is good, but it's difficult to go through code that's been refactored and cherry-pick the commits that we want to release. Maybe experimental features should be in separate branches and we could keep a staging branch. (Since the master branch is effectively the "release" branch, I'm guessing quite a few distros just pull from it without looking). The staging branch is where we merge the best commits and spend a week testing code ourselves along with whoever else would like to join in.

chjj commented 12 years ago

Ah, apparently I made some sort of mistakes in my words, sorry. I didn't mean it's bad to merge the changes to master, considering master branch doesn't mean any sort of stability. And these changes should actually be merged into master, if that means more users will be playing with the recent changes so I get more feedback. I just don't like marking a 0.0.1 released, as I don't want those package maintainers to push the the very unstable recent changes out as a "version 0.0.1" of compton.

No, you're right about this. I don't want to put something on master if it's severely unstable. Since the version tagging will be entirely new for this repo, package maintainers probably aren't looking for it anyway.

richardgv commented 12 years ago

Totally agree. I really want to update the man page as its usually the first thing people look at. I'll grab the help info and update it.

Ah, thanks. :-)

I noticed you were making a lot of GL calls there. Was the XFCE implementation any help?

The author of the Xfce patch says:

Tearing still happens about 10 pixels below the top of the display, but since this space is usually only occupied by the title bar, it's only visible when dragging a window across the drop of the screen.

Well, it probably isn't entirely impossible to do as good as the patch does, in most cases. Nonetheless I would presume the tearing line his solution produces will move depending on the CPU scheduler, system load, CPU power, screen resolution, etc., and it could move to an annoying position, like at the middle of the screen, if the machine changes. And anyway it would affect full screen video playback or gaming. This is possibly not the thing most users wish to see.

I wonder how the hell other compositing window managers are dealing with the problem. KWin and cairo-compmgr might be using some sort of double buffer, but my knowledge is limited and I don't know how exactly is that mechanism implemented. kwin is too big, while cairo-compmgr is written in a rather "object-oriented" way that makes its code harder to read. Hmm, I will see if I can get some more time to read their source code (my grandma got sick, a heart problem, occupying much of my time right now), but if I don't get anything from them... Do you have any ideas?

We really need some kind of staging branch. A development branch is good, but it's difficult to go through code that's been refactored and cherry-pick the commits that we want to release. Maybe experimental features should be in separate branches and we could keep a staging branch. (Since the master branch is effectively the "release" branch, I'm guessing quite a few distros just pull from it without looking). The staging branch is where we merge the best commits and spend a week testing code ourselves along with whoever else would like to join in.

The problem is my changes are usually applied to multiple parts of the program, so manually merging changes from different branches could be a hassle, and it requires additional testing to make sure things work well after the merge, effectively doubling the requires testing time. And a user (and me myself) won't be able to try all the cutting-edge features if he wants, unless he wishes to merge all those things manually. In view of the fact that we are not adding many features simultaneously right now, and the number of developers we have is not quite big enough to create a severe synchronization issue, and we don't have a planned feature that will stay in experimental state for a long period of time, I personally don't think there's an urgent need to split development of different features to different branches.

Hmm, as for the staging branch... Have you ever seen any of my commits introduces a big issue? :-) I'm usually pretty careful with commits even when I claim I did "little tests" in commit log, so I'm basically considering richardgv-dev as my own staging branch. IF you plan to greatly modify the code (not documentation) recently, it might be a good idea to create a staging branch for your work.

It's truly annoying that distros are pulling from master branch directly. I assume the situation will get a lot more better once we create a formal release, though.

By the way, how do you think about the plain Makefile vs CMake problem?

Another annoying technical problem is how to reduce the CPU usage caused by b78ab316fd. It generates many calls to XFixesIntersectRegion(), XFixesSubtractRegion(), and XFixesSetPictureClip() during painting, thus increasing the CPU usage of X and compton process. On my box with Intel Core i7 3770K, when glxgears is rendering with 18000 FPS, X is using roughly 80% of a single core, compton using 30% of another core, while under xcompmgr X uses only 30% of a single core (but with xcompmgr glxgears is running at 9000 FPS). I have a few thoughts in my mind:

  1. We could totally revert to pre-cdf7db750d painting sequence, the old xcompmgr style. The drawback is it cannot cache window ignore regions (w->border_clip), so the amount of calls to XFixesCreateRegion() and XFixesDestroyRegion() will increase much, and we won't be able to use the window ignore region for cutting damaged region (which will reduce the number of paints if the damaged window is hided behind a opaque window). It reduces program modularity because I have to split window painting to 2 parts.
  2. We could merge identical window ignore regions using a hash table. The effect of this approach depends on how much XFixesCopyRegion() is faster than XFixesSubtractRegion(). Well, I guess not much. Worse yet, if there are not many identical window ignore regions (usually, if there are not many semi-transparent or ARGB windows), it will have the opposite effect. Reduces program modularity, too, and requires a hash table implementation (looks like uthash is great).
  3. Remove the XFixesIntersectRegion() call which intersects the painting region with w->border_size if the window is not bounding shaped. Implemented, but didn't seem to help much. And it will break empty region detection in some cases.
  4. A bold plan: We could move the region calculation to compton process, i.e. do the region calculation X Fixes extension do in the compton process, by cloning some code from xorg-server, therefore saving the time of many roundtrips from compton process to X process. Oh, well, this is probably too ambitious.
  5. We could choose to leave it alone, as it's not terribly common to see compton painting 18,000 times per second, until our users come up complaining compton is causing crazy CPU usage in a particular case, which we can then look into.
  6. We could enable software VSync by default, limiting the number of paints per second. Also a cure for the OpenGL performance problem. The risk is if compton wasn't getting the correct refresh rate from X RandR extension, under some extreme cases, a user might see something fun.

Another thing I don't understand is why DEBUG_REPAINT or gdb reports no paint here when glxgears is running and compton should indeed be painting.

richardgv commented 12 years ago

@chjj:

Status update:

  1. The VSync problem, well, I asked on Xorg mailing list and nobody replied, and my fixes resulted in the same thing the Xfce patch achieves: a tearing line on the top of screen. At least it's some sort of VSync, and we may just have to live with it until somebody tells me a better fix.
  2. Nobody complained about the CPU usage so far, so we could just leave it alone.
  3. The DEBUG_REPAINT issue I mentioned on the bottom of the last reply: I heard nvidia drivers don't report damages correctly sometimes.

Current plan:

  1. No definite reports of breakages exist presently. I've merged your documentation changes from master to richardgv-dev. I suppose we are close to ready to merge richardgv-dev to master once I make sure what damarusama is complaining (#55) is not our fault. He provided too little information, well...
  2. Man pages need more work. I hate groff... do you think we can use DocBook or Asciidoc for writing man pages? Or Markdown, since you appears really enjoying it? I'm personally more familiar with AsciiDoc, however any a language will make more sense than groff, I bet. :-)
  3. 1-2 weeks after things get stabilized in master branch, if there's nothing wrong I hope there could be a formal release of compton. It's likely to attract more users. What's your thoughts about the version number? I thought about 0.1.0 or 1.0.0_beta. Last time you chose 0.0.1, but I feel that number is a bit too small considering the power and stability of compton, and your huge amount of work. :-) And it would be hard to add a minor revision: Will you call 0.0.1 + a minor bug fix 0.0.1.1 or 0.0.2? Both don't look too beautiful for me.
chjj commented 12 years ago

No definite reports of breakages exist presently. I've merged your documentation changes from master to richardgv-dev. I suppose we are close to ready to merge richardgv-dev to master once I make sure what damarusama is complaining (#55) is not our fault. He provided too little information, well...

Okay, I'll still play around with it a bit more and see if I notice higher cpu usage at all.

Man pages need more work. I hate groff... do you think we can use DocBook or Asciidoc for writing man pages? Or Markdown, since you appears really enjoying it? I'm personally more familiar with AsciiDoc, however any a language will make more sense than groff, I bet. :-)

Markdown would be really nice. It would be pretty easy to copy & paste from the readme then.

1-2 weeks after things get stabilized in master branch, if there's nothing wrong I hope there could be a formal release of compton. It's likely to attract more users. What's your thoughts about the version number? I thought about 0.1.0 or 1.0.0_beta. Last time you chose 0.0.1, but I feel that number is a bit too small considering the power and stability of compton, and your huge amount of work. :-) And it would be hard to add a minor revision: Will you call 0.0.1 + a minor bug fix 0.0.1.1 or 0.0.2? Both don't look too beautiful for me.

You're right, v0.0.1 probably would have been xcompmgr. ;p Well, I can't choose v0.0.1 anyway unless we forcibly overwrite the previous tag, which can screw up branches tracking master. v0.1.0 seems reasonable.

richardgv commented 12 years ago

Okay, I'll still play around with it a bit more and see if I notice higher cpu usage at all.

Oh, yep, that's a thing that could be troublesome. Plus I'm working on reducing the number of errors compton sometimes throws out when dealing with drag-and-drop windows, and inspecting #17 and #25, and I want a final confirmation of issue #6, #30, and funeral1998's comment about CPU load, lagging, and screen freeze. And #58, which doesn't look like our fault but could be.

Markdown would be really nice. It would be pretty easy to copy & paste from the readme then.

The benefit of Asciidoc is it has significantly more features and flexibility, including table support, mathematic formulas (which we most likely won't use :-D, but supported by troff, so Asciidoc is more likely able to cover all things troff allows), many options to customize the output to various backends (so it's easier to produce a beautiful & highly-customized HTML, PDF, or LaTeX version of our man page later), and it has a standard implementation that supports many backends. Although Docbook probably supports more things, its XML syntax means we have a few tons of extra characters to type. (As what Dan Allen said, "Writing in Docbook is just, inhumane.") As for the Readme, I guess Markdown and Asciidoc's syntax are so similar that conversation won't take too much more than search-and-replace. I personally prefer Asciidoc and is more familiar with it (all my past projects use Asciidoc Readmes). But it's you who makes the decision, we could go for Markdown if you really enjoy it.

You're right, v0.0.1 probably would have been xcompmgr. ;p Well, I can't choose v0.0.1 anyway unless we forcibly overwrite the previous tag, which can screw up branches tracking master. v0.1.0 seems reasonable.

Well, sorry, I've dropped the v0.0.1 tag on the server without asking you. I hope dropping a tag usually does not mess up the branches much. But it does mean there couldn't be a new v0.0.1 any longer.

By the way, do you think we could contact Crunchbang (which is using compton as the "official" compositor?) and somehow collect all the issue reports on its forums to a single official thread, so it's easier to keep track of the reports? The full discussion: https://github.com/chjj/compton/issues/55#issuecomment-9861854

By the way 2, I'm writing a (short) FAQ on the wiki, and I suppose a geek-ish joke on the bottom of the FAQ, as many projects have, would be great. Do you have one in mind? :-D

smlx commented 12 years ago

Just wanted to note: Pandoc does a nice conversion from Markdown to roff.

http://johnmacfarlane.net/pandoc/demo/pandoc.1.md

richardgv commented 12 years ago

Just wanted to note: Pandoc does a nice conversion from Markdown to roff.

http://johnmacfarlane.net/pandoc/demo/pandoc.1.md

Ah, thanks for the note. :-)

Two additional notes:

  1. Asciidoc supports man page as one of its standard document types, so you could say Asciidoc supports man page generation in design instead of "could generate man pages through a third-party converter".
  2. Git itself uses Asciidoc to generate its man pages, as well as its HTML versions of man pages, and its PDF user manual. I don't know if there's a software using markdown for the source of its man pages.
richardgv commented 12 years ago

I pushed a man page in Asciidoc to richardgv-dev branch, because chjj did not make a decision about what we should use in time. I hope it's not hard to convert Asciidoc to Markdown if you really need that.

Although GitHub kindly generated an HTML version of the man page for us, it doesn't look well. GitHub's implementation is using different styles than Asciidoc's standard man page and HTML implementation, and the "NAME" section disappeared. Well, at least it's better to have one than not.

I kept the old man page, too, so if you wish to use the new man page, you have to execute make docs before you do make install. make docs will also let Asciidoc generates a HTML version of the man page, but will not install it.

richardgv commented 12 years ago

Asciidoc-format man page for compton-trans pushed out, too, and slightly modified.

@chjj:

Do you think we should merge the fix of #61 (b15d40e) to master branch?

Over a month of work are living outside master branch, a total of 2,500+ lines of insertions. My recent changes aren't particularly risky, and there's no pending bugs as far as I could see (a few bug fixes have not been confirmed to work, though.) If all things go well, I hope we could merge richardgv-dev to master in a few days. Your opinion, please?

And how should we deal with #60? Honestly, I don't like it. :-D

richardgv commented 12 years ago

@chjj:

Current plan:

  1. I've written most code for what baskerville eventually asked for in #16, a window property _COMPTON_SHADOW, but I need extra time to test.
  2. There's a bug in my fix to #61. A minor one, though. Will fix it later.
  3. I plan to merge richardgv-dev into master in a few days unless I spot new bug reports. I'm still quite worried about the stability, but master branch is lagging too much behind, and there are not enough people testing richardgv-dev branch.

Opinions?

Things waiting for you to decide:

  1. Should we use Asciidoc man pages instead of roff or Markdown ones?
  2. Name and format of _COMPTON_SHADOW, discussed in #16?
  3. Should we merge #60?
  4. Is my modified focus tracking proper (#59)?
  5. Does my fix for #46 works for you?
  6. What do you think about the problem OrdinaryMagician just reported in #50? Dangerous, or not?
  7. Can't recall more for now...
richardgv commented 12 years ago

@chjj:

Current plan:

  1. Basically I want to repeat my last point in the last reply: merge richardgv-dev into master. Look, I pushed --unredir-if-possible (59e54b0665) to richardgv-dev, which could be pretty useful for many, and nobody is testing, nobody!

Problems:

  1. All the points I mentioned in the previous reply still make sense, although not vital.

  2. There is one big roadblock ahead, before I could merge richardgv-dev into master: Your c883cde. My rewritten compton-trans is incompatible with yours in a number of ways, that I would prefer not to change:

    1. It's possible to merge your -c none, but as compton (at far as I know) treats a 0xffffffff _NET_WM_OPACITY as non-existent, -c 100 and -c none have no differences at least for compton.
    2. We use different ways to determine the frame window of a specified window. Your version continuously generates xwininfo calls to look for the parent of a window, while mine calls xwininfo -root -tree once and look for the matching line. Your version has superior theoretical performance, but considering the cost of doing external calls and the efficiency of bash vs. C (xwininfo), it's pretty hard to predict which approach is generally faster. And my version always generates a finite number of xwininfo calls. Even if there's something wrong, 1000 loop cycles won't be wasted.
    3. I changed some of your CMD | grep ... | sed calls to CMD | sed. At least this part should be merged into master branch.
    4. And 3 days ago I pointed out two bugs in your version, they are still there.

    Decision, please?

  3. Should we now use Makefile or CMake? I have both in my branch, and they both work. Putting both into master branch could be puzzling.

I fully understand that you probably don't have time to commit much code, but I always hope you at least have some time to make a few vital decisions -- I always consider compton your project. Anyway, I will do what I consider appropriate, if I don't see your reply in 3 days.

By the way, is your nick on FreeNode chjj?

16:02 [FreeNode] -!- chjj [~c@c-24-118-232-172.hsd1.mn.comcast.net]
16:02 [FreeNode] -!-  ircname  : c
16:02 [FreeNode] -!-  server   : lindbohm.freenode.net [Stockholm, Sweden]
16:02 [FreeNode] -!-           : is using a secure connection
16:02 [FreeNode] -!-  account  : chjj
16:02 [FreeNode] -!- End of WHOIS
richardgv commented 12 years ago

Now I shall repeat, @chjj, if I don't see you replying in probably less than one day, I will merge richardgv-dev into master in the way I want. Please don't complain you have not been warned.

While the last reply discusses the present issues, this one discusses the future:

  1. I'm interested in porting compton to XCB. It's the better Xlib, at least it's what they claimed, and it makes error checking easier, from my observations right now, which hopefully will kill #52, and libxcb has some pretty cool documentation. Moreover, I hope libxcb is more effective against VSync issue, although I don't expect too much. However, the difficulty is obvious: XCB is more complicated to use than Xlib (less obscure logic, too, though), the size of the program may increase (but libX11.so is 10x bigger than libxcb.so here), and a whole set of dependency changes.
  2. I'm evaluating the possibility of a generic matching mechanism and a control socket for compton, mentioned in #16. Both require invention and implementation of new mini-languages, and my knowledge/experience in that field is 0...
richardgv commented 11 years ago

Status update:

  1. I've chosen to switch to libevent for main loop, as:
    1. poll() does not have the accuracy required for --sw-opti. We originally are using ppoll(), but as described in #74, ppoll() does not exist on *BSD. One possible alternative is select() / pselect(), but they have a rather ugly design and is slower. To write some BSD-specific code to handle this is possible, too, but I found no easy way to identify whether we are using glibc or BSD libc.
    2. libevent uses epoll and kqueue as backend, on different platforms. Both are more efficient than poll() or select().
    3. libevent is a pretty small library with speed in mind during design. Many lightweight apps (tmux, tor, etc.) use it, too.
    4. Abstraction of main loop means we at least won't encounter further platform-specific issues on this part.
    5. The incoming D-Bus support requires flexible timer handling in the main loop (DBusTimeout in libdbus). Using libevent will save some code and reduce the chances of bugs.
  2. The plan for introduction of new condition format and D-Bus support is again delayed by my bad cold, sorry. Well, I hope I could have at least one of them pushed out in 10 days.
  3. The plan for tagging is delayed, too, because of various bug reports we got right when I plan to add a tag.