Visible Virtual Treeview causes multiple problems in other controls in Lazarus/Cocoa

[TommySlokky]

case1.zip

Here is a demo project which demonstrates multiple issues when having a visible virtual treeview on Lazarus Mac/Cocoa (using fpcupdeluxe "trunk" which includes newest virtual treeview)

Issue was reported here: https://bugs.freepascal.org/view.php?id=34002

And together with Lazarus developers narrowed it down to be an issue with virtual treeview and cocoa.

The attached demo project demonstrates:

1) Toolbuttons becoming invisible until clicked 2) TPageControl tabsheets misdrawn until clicked 3) TEdit looking very weird with black background until clicked 4) Some captions/labels etc. are drawn upside down some places

All only when the two virtual treeviews are set to visible in the project.

...

To see all issues simply

• run project
• doubleclick window title bar (minimizes to dock)
• open from mac dock.

[blikblum]

VirtualTreeView does not uses cocoa directly, just LCL and lclextensions. Probably some of the LCL API used by VirtualTreeView is not mature enough on cocoa. The way to go is to isolate the API that is broken and fix it. In early days i did this for win32, qt and gtk2 widgetsets. I fixed a lot of LCL / widgetset. Unfortunately i don't have a mac.

If you want to isolate code from lclextensions, compile with USE_DELPHICOMPAT directive disabled (VTConfig.inc)

[TommySlokky]

Thank you for your answer. I try see if i can find time to study LCL/widgetset implementations, Cocoa and connect the dots with Virtual Treeview. It is an extremely important control (to me at least), so kinda critical it is not working on Cocoa :(

[blikblum]

As a start i really suggest compiling with USE_DELPHICOMPAT directive enabled. This will ensure only LCL functions are called. It should work normally but slower

[TommySlokky]

I will try that now.

Meanwhile I I have tried two non-virtual-treeview demos using graphic routines that appears to work on Cocoa - a when least standalone:

1) https://forum.lazarus.freepascal.org/index.php/topic,36871.msg246309.html#msg246309  Called SpriteTest2.Zip and it uses copyrect, mask, union

2) https://forum.lazarus.freepascal.org/index.php/topic,36871.msg246790.html#msg246790  Same thread, Mr.Madguy code listed uses intersect, fillrect, cliprect

[TommySlokky]

Offhand not possible to comment out the USE_DELPHICOMPAT define - it results in the following errors:

VirtualTree.pas

• SetBrushOrigin -- SetBrushOrgEx (Identifier not found)
• HandleMouseDown -- WM_LBUTTONDOWN (Identifier not found)

I did try comment out some other defines in VTConfig.inc - EnableAdvancedGraphics - but that did not make any difference.

...

Do you know what kind of graphic routines virtual treeview uses beyond those used in the two above demos? I am trying to create a demo application I can submit in my LCL cocoa bug report which reproduce the problem.

[blikblum]

Compiled now on linux and working fine. Should at least compile on mac

WM_LBUTTONDOWN (Identifier not found)

What version are you using? Do not use master, use lazarus-v4 or lazarus_v5 with lclextensions 0.6.1

There's no WM_LBUTTONDOWN

SetBrushOrgEx (Identifier not found)

This is protected by {$ifndef INCOMPLETE_WINAPI}. Also SetBrushOrigin was removed long time ago

[TommySlokky]

I am using the one that is included when using fpcupdeluxe/trunk / virtualtreeview-new (5.x)

Albeit currently lazarus now ships with virtual treeview package per default, just not installed - so fpcupdeluxe author states that is the newest version to use. (Not necessary to use fpcupdeluxe menu items for installing the virtualreeview-new (5x) package.)

You can see a problem report for lcl_extensions here: https://bugs.freepascal.org/view.php?id=33973 http://forum.lazarus.freepascal.org/index.php/topic,42226.0.html

Where fpcupdeluxe developer tracked it down to: "Fpcupdeluxe uses online sources to install packages.However, these sources are fast changing and moving all the time.Now it seems that these vtv-sources are no longer updated due to the fact that they are included (again) with Lazarus ! So, just install the latest vtv by installing the vtv that comes with Lazarus."

(So that is what I have done)

...

But if I understand you correctly, I should probably install manually from the sources downloaded manually from here. I will do that and report back (!)

[blikblum]

Fpcupdeluxe uses online sources to install packages.However, these sources are fast changing and moving all the time

lclextensions did not changed from a long time

But if I understand you correctly, I should probably install manually from the sources downloaded manually from here. I will do that and report back (!)

Yes. You can also use git and fork the repo. Use v4 or v5 branches:

https://github.com/blikblum/VirtualTreeView-Lazarus/tree/lazarus-v4 https://github.com/blikblum/VirtualTreeView-Lazarus/tree/lazarus-v5

[TommySlokky]

This bug report still applies then: https://bugs.freepascal.org/view.php?id=33973

delphicompat.pas(137,2) Fatal: Cannot open include file "uses.inc"

Looking in lclextensions-0/include you have subfolders for -carbon -generic -gtk -gtk2 -qt -win32 (where each contains uses.inc)

But none for Cocoa

I suspect that is the reason and fix is simple. I will test later

[TommySlokky]

Well, not so easy, but the latest version of LazarusTrunk includes a version of lclextensions where Cocoa version of delphicompat.inc (etc.) has been added.

So it would seem Lazarus people have made several additions without adding them to: https://github.com/blikblum/luipack/tree/master/lclextensions/include ?

[TommySlokky]

I uninstalled all old virtualtreeview... and fixed lclextensions

...

Then I downloaded from https://github.com/blikblum/VirtualTreeView-Lazarus/tree/lazarus-v5 using "Download or clone"

Then go into VTConfig.ini and switch off USE_DELPHICOMPAT define

and I still get error

VirtualTrees (5282,3) Error: Identifier not found "SetBrushOrgEx"

and

VirtualTrees (22188,38) Error: Identifier not found "VM_LButtonDown"

...

I can send screenshots...

[blikblum]

Please send screenshots of the places that the compiler warns about missing identifier

[TommySlokky]

na

[blikblum]

You are using lazarus-master branch which is not completed yet. Try these: https://github.com/blikblum/VirtualTreeView-Lazarus/releases

[TommySlokky]

Okay, so I on my Mac Lazarus Install:

• uninstalled old package
• downloaded and installed virtualtreeview-5-5.3-R1.zip
• comment off the define USE_DELPHICOMPAT

The same two issues but different line numbers. They are

VirtualTrees (5273,3) Error: Identifier not found "SetBrushOrgEx" and VirtualTrees (22179,38) Error: Identifier not found "VM_LButtonDown"

...

Just to confirm again I downloaded the file you suggest in Windows and opened in Notepad+ and navigated to the line numbers. I see them both there as well at the line numbers I report here.

...

And just to further confirm - I also tried download "source code (Zip)" https://github.com/blikblum/VirtualTreeView-Lazarus/archive/lazarus-5.5.3-R1.zip just in case you meant I should download that - and again I see them both there same line numbers

[blikblum]

Sorry, the v5 is missing some IFDEF. I will fix. Please try v4 for now: https://github.com/blikblum/VirtualTreeView-Lazarus/releases/download/lazarus-4.8.7-R4/virtualtreeview-4.8.7-R4.zip

[TommySlokky]

Thank you. Personally I think I will have to wait for 5.x since it has some functionality I depend on.

However, it might be worth pursuing if 4.8.7 exhibits/causes the same drawing/clipping issue in Cocoa which drew me down a very long pursuit (took time before I got the problem isolated to be related to virtual treeview)

...

When you fix 5.x will it be in the master branch https://github.com/blikblum/VirtualTreeView-Lazarus ? (just asking to make 100% sure I do not download wrong files)

...

What about the lcl-extensions issues for Cocoa? Do you want me to send you the files Lazarus distributes in its trunk for lcl-extensions-cocoa-include-files?

[blikblum]

When you fix 5.x will it be in the master branch

Done in 1bf04947d143237b4381226e4334f5d1e3a71ae9

Go to https://github.com/blikblum/VirtualTreeView-Lazarus/tree/lazarus-v5 and click on download

What about the lcl-extensions issues for Cocoa? Do you want me to send you the files Lazarus distributes in its trunk for lcl-extensions-cocoa-include-files?

I can take care of that

[TommySlokky]

Thank you. While the clipping issue remains, it will hopefully help tracking down the problem being able to comment out USE_DELPHICOMPAT

[blikblum]

I will help tracking the issue.

For start you can test the following bug on cocoa: https://bugs.freepascal.org/view.php?id=28689 Demo at: http://bugs.freepascal.org/view.php?id=25564

I have some LCL code that i fixed on other widgetsets to make VTV work. I will look at it

[blikblum]

You may want to test also ExcludeClipRect function

[blikblum]

Look here for a ExcludeClipRect demo:

https://bitbucket.org/blikblum/pascal-demos/src/16fc10cb12ed7a9fb1c4793bd592efb8ca198a9f/lazarus/LCLTests/ExcludeClipRect/?at=master

[TommySlokky]

I will test everything today - right now latest trunk of Lazarus/Cocoa/64bit is causing problems but looking into it, so I can get Lazarus working again for testing all the things you have posted ASAP :)

[TommySlokky]

I believe both your examples are buggy in Lazarus/Cocoa

I am attaching screenshots using the very latest Lazarus/Cocoa/64bit trunk

[blikblum]

Yes they are buggy

[TommySlokky]

Submitted bugs here https://bugs.freepascal.org/view.php?id=34196 https://bugs.freepascal.org/view.php?id=34197

...

Do you think there could be more potential issues?

The original problem was that if you added a virtual treeview to a Cocoa project like in this demo project: http://forum.lazarus.freepascal.org/index.php/topic,41930.msg292282.html#msg292282

1) Toolbuttons becoming invisible until clicked (just click on the top toolbar to expose them)

2) TPageControl tabsheets misdrawn until clicked

3) TEdit looking very weird with black background until clicked

4) Captions/labels etc. are drawn upside down some places

[blikblum]

I compiled the demo on windows and i have some remarks:

• By what i understand correctly, the issue does not exists when VTV is not visible, right?
• If i resize the window, the pink toolbar does not invalidate / repaint properly, regardless if VTV is visible or not
• The VTV components are children of the TForm with z-index on top. Try to put the VTV as child of the tabsheet
• There are many VTV components can you test with only one of them and than with no VTV component?
• The demo have many controls. Try to create a demo with only the minimal number of controls to demonstrate the issue

[blikblum]

I simplified a bit the demo.

It has only one VTV component.

Try to test the following scenarios and post the results:

• Make the "With VTV" tab active
• Make the "No VTV" tab active
• Make the VTV component not Visible by default. Test. Toggle Visible. Test again

case1.zip

[TommySlokky]

vtv-set-to-invisible--no-vtv-tab

vtv-set-to-visible--visible-vtv-tab--maximized

vtv-set-to-visible--visible-vtv-tab--just-resized-form

vtv-set-to-visible--visible-vtv-tab--just-resized-form--2--hover-over-button

inside-ide-reversed-text-and-icons

Here is a bunch of screenshots showing what different actions do - hopefully they will help

[blikblum]

vtv-set-to-invisible--no-vtv-tab > VTV is set to not Visible? Everything fine right?

When VTV is visible and With VTV is active the paint at first show is fine? The issue only manifests after minimizing / maximizing / resizing form right?

What happens if VTV is visible, but the No VTV Tab is active? Is the painting fine? If so, you can do two tests to check if the issue manifests:

• make the With VTV tab active (select it)
• add a button that calls tvVirtual_Sitemap.Handle -> this will force control initialization but without painting

This is important to know if the problem is triggered by the initialization or the paint code.

Alternatively, put a exit call in the start of TBaseVirtualTree.PaintTree

[blikblum]

Put a DebugLn inside TBaseVirtualTree.WMPaint and log the value of Message.PaintStruct^.rcPaint Also check if TBaseVirtualTree.WMPaint is being called after minimizing / maximizing / resizing (when the issue occurs).

My theory is that somehow the form is not being invalidated

[blikblum]

In CocoaPrivate inside

procedure TCocoaCustomControl.setFrame(aframe: NSRect); begin if NSEqualRects(aframe, frame) then Exit; if isdrawing>0 then faileddraw := true;

inherited setFrame(aframe); // it actually should come from a notifcation if Assigned(callback) then callback.frameDidChange(self); end;

Put a breakpoint at faileddraw := true; If hit, write the callstack here

[TommySlokky]

I will reply to all your questions during today and tomorrow.

...

I have also cleaned the demo a little so it is easier to see everything.

toggle-vtv-visible-then-reisze.zip

...

vtv-set-to-invisible--no-vtv-tab > VTV is set to not Visible? Everything fine right?

Adjusting the demo so it starts as invisible - no issue at startup or resize or maximize. -The moment it is set visible the toggle button becomes invisible. -maximixing then further causes issues in toolbar, edit etc.

...

add a button that calls tvVirtual_Sitemap.Handle

Next test: vtv starts up as invisible, but "with vtv" tab active. I click button which calls vtv.handle;

No visible change in anything. Also not when resized

[blikblum]

vtv starts up as invisible, but "with vtv" tab active. I click button which calls vtv.handle;

No visible change in anything. Also not when resized

Seems that the issue is triggered by the paint code.

The next steps are putting a breakpoint at TCocoaCustomControl.setFrame in the line that faileddraw is set to true and putting an Exit call at start of PaintTree and see what happens when VTV is Visible

[TommySlokky]

Inserting "Exit;" as first line in .PaintTree solves the problem - but obviously not a very good solution :)

[TommySlokky]

Faileddraw does not seem to trigger even if virtual treeview set to true, tab selected and I resize/maximize the window.

[blikblum]

Inserting "Exit;" as first line in .PaintTree solves the problem - but obviously not a very good solution :)

But show us how to find the issue:

Put Exit calls in different parts of .PaintTree until find the offending line

You can go from start to end in the method or use a strategy similar to bisect: put exit in the middle, if no bug then try between middle and end otherwise between start and middle and go on

I suggest keeping the used exit calls commented to mark already tested places

[TommySlokky]

Started doing that - then just realized I had made a grave error in my testing

...

I had inserted "Exit;" in

TBaseVirtualTree.VMPaint (solves issue) and not in TBaseVirtualTree.PaintTree (does not solve issue)

Just repeated the experiment multiple times to make sure it is done correct.

...

Anyhow, I will be happy to conduct more experiments - I will try see if I can diagnose it further. (If any ideas, let me know)

...

Until then I have submitted the two issues identified sofar here: https://bugs.freepascal.org/view.php?id=34196 https://bugs.freepascal.org/view.php?id=34197 (I have also uploaded a demo and screenshot for each, but if you think anything is missing please feel free to add)

I am not sure if those two issue can explain the whole issue demonstrated in toggle-vtv-visible-then-reisze.zip

[blikblum]

Started doing that - then just realized I had made a grave error in my testing

No problem, try to put Exit calls in TBaseVirtualTree.Paint

I am not sure if those two issue can explain the whole issue demonstrated in toggle-vtv-visible-then-reisze.zip

Probably not, but at least the SetWindowOrgEx bug must be resolved for a proper working VTV

[blikblum]

Also add a breakpoint to TBaseVirtualTree.WMPaint and step into TWinControl.WMPaint:

Msg.DC must be different 0 and PaintHandler should be called

[TommySlokky]

No problem, try to put Exit calls in TBaseVirtualTree.Paint

That solves the problem.... And seeing that it did not help exit PaintTree that really only leaves a limited amount of possible reasons left:

• ExcludeClipRect() - bug already found and reported per your guidance (the error probably ends in cocoawinapi.inc TCocoaWidgetSet.ExtSelectClipRGN and some of the stuff called there)
• FOffSetY
• FUpdateRect

Or can it be limited further?

(I do not think ComputeRTLOffset/RTLOffSet has any relevance since I am not using UseRightToLeftAlignment. I also do not think OffSetRect since its implementation is found in LCLProc and looks clean)

I guess nextup would be to confirm validity FOffSetY/FUpdateRect and then make a testcase for OffsetRect

Msg.DC must be different 0 and PaintHandler should be called

Msg.DC is different than 0 and PaintHandler is called

[blikblum]

ExcludeClipRect() - bug already found and reported per your guidance (the error probably ends in cocoawinapi.inc TCocoaWidgetSet.ExtSelectClipRGN and some of the stuff called there)

One way to test it is unsetting Header.Options.hoVisible since is only called when header is visible

FUpdateRect

It should have a value similar (in first paint, after setting visible) to

LEFT = 0, TOP = 0, RIGHT = 638, BOTTOM = 267,

It's valuabe also to check the value of FHeaderRect that is passed to ExcludeClipRect

It should be something like:

LEFT = 0, TOP = 0, RIGHT = 638, BOTTOM = 17,

[TommySlokky]

FUpdateRect

It should have a value similar (in first paint, after setting visible) to LEFT = 0, TOP = 0, RIGHT = 638, BOTTOM = 267,

With Header.Options.hoVisible = True; and Header.Options.hoVisible = False; values are: LEFT = 0, TOP = 0, RIGHT = 618, BOTTOM = 255,

It's valuabe also to check the value of FHeaderRect that is passed to ExcludeClipRect It should be something like: LEFT = 0, TOP = 0, RIGHT = 638, BOTTOM = 17,

With Header.Options.hoVisible = True; values are: LEFT = 0, TOP = 0, RIGHT = 618, BOTTOM = 17,

ExcludeClipRect()

One way to test it is unsetting Header.Options.hoVisible since is only called when header is visible

Yes!

Setting Header.Options.hoVisible also solves the issue in the toggle-vtv-then-resize.zip demo application

[blikblum]

There's a possible bug in TCocoaRegion.CombineWith when handling cc_DIFF (used by ExcludeClipRect)

If FShape is empty, a big one is created and stored in Shape var to be used in HIShapeCreateDifference, but Shape var is not used. The empty shape is used instead

      cc_DIFF:
      begin
        if HIShapeIsEmpty(FShape) then
          {HIShapeCreateDifference doesn't work properly if original shape is empty}
          {to simulate "emptieness" very big shape is created }
          Shape := HIShapeCreateWithRect(GetCGRect(MinCoord, MinCoord, MaxSize, MaxSize)); // create clip nothing.

        Shape := HIShapeCreateDifference(FShape, ARegion.Shape);
        Result := GetType;
      end;  

[TommySlokky]

Offhand this does not solve the issue:

cc_DIFF:
      begin
        if HIShapeIsEmpty(FShape) then
          begin
            {HIShapeCreateDifference doesn't work properly if original shape is empty}
            {to simulate "emptieness" very big shape is created }
            Shape := HIShapeCreateWithRect(GetCGRect(MinCoord, MinCoord, MaxSize, MaxSize)); // create clip nothing.
            Shape := HIShapeCreateDifference(Shape, ARegion.Shape);
          end
        else
          begin
            Shape := HIShapeCreateDifference(FShape, ARegion.Shape);
          end
        ;
        Result := GetType;
      end;  

No sure my "fix" is correct though by any means. I usually do not deal with graphics/GDI

[blikblum]

Offhand this does not solve the issue:

This was a blind shot, i don't ever know if in this case HIShapeIsEmpty(FShape) evaluates to true

It will be necessary to debug step by step the ExcludeClipRect code path by someone familiar with cocoa API

No sure my "fix" is correct though by any means.

It's correct

[blikblum]

Setting Header.Options.hoVisible also solves the issue in the toggle-vtv-then-resize.zip demo application

Great. Now let's discover exactly where the problem is.

In TBaseVirtualTree.Paint, comment separately FHeader.FColumns.PaintHeader and ExcludeClipRect calls, seems that one of the two are the culprit

[TommySlokky]

Inside TBaseVirtualTree.Paint:

Commenting out call to PaintHeader = problem continues Commenting out calls to ExcludeClipRect = problem solved in the demo app (!)

...

For reference the Cocoa maintainer "skalogryz" seems active in the forum thread: https://forum.lazarus.freepascal.org/index.php?topic=42246.msg296454

[blikblum]

I pushed a fix disabling ExcludeClipRect in cocoa.

Probably there will be some drawing issues with header

Please do some tests with examples that shows a header, like the ones in demos folder and report here your findings.

[TommySlokky]

Preliminary test seem very promising (!)

I will first get more time sunday - but hopefully I will be able to make a full report then.

It also seems the two reported bugs have been officially solved https://bugs.freepascal.org/view.php?id=34197 https://bugs.freepascal.org/view.php?id=34196