Question: InkCanvas inside Scrollviewer with touch drawing

[kellylawson]

I have a question about an inkCanvas inside a ScrollViewer. It works fine to support zoom and scroll on the CanvasControl until I set InputDeviceTypes to include Windows.UI.Core.CoreInputDeviceTypes.Touch on the inkPresenter.

I would have expected the canvas to draw with single finger touch, but multi touch gestures like pinch and two finger scroll should bubble to the ScrollViewer if multi point drawing is not enabled. What seems to happen instead is that multi-touch gestures are distilled down to a single touch point and do not bubble up.

I reproduced this easily with the following code in a simple project:

C#:

        public MainPage()
        {
            this.InitializeComponent();
            inkCanvas.InkPresenter.InputDeviceTypes = Windows.UI.Core.CoreInputDeviceTypes.Mouse | Windows.UI.Core.CoreInputDeviceTypes.Pen | Windows.UI.Core.CoreInputDeviceTypes.Touch;
            inkCanvas.InkPresenter.InputProcessingConfiguration.Mode = InkInputProcessingMode.Inking;
        }

xaml:

    <Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
        <ScrollViewer ZoomMode="Enabled" MinZoomFactor="0.25" HorizontalScrollBarVisibility="Hidden" VerticalScrollBarVisibility="Hidden" HorizontalScrollMode="Enabled" VerticalScrollMode="Enabled">
            <Grid Background="White">
                <canvas:CanvasControl x:Name="canvas" Draw="canvas_Draw"/>
                <InkCanvas x:Name="inkCanvas"/>
            </Grid>
        </ScrollViewer>
    </Grid>

My actual project is a little more complex, I'm using unprocessedInput on the inkPresenter and custom drawing everything, but the basic project above duplicates the behavior I'm seeing. I'm sure that I'm missing something here as it seems like this should be relatively simple... Is there a way to get the inkPresenter/CanvasControl to stop trapping multi touch gestures when touch input is enabled?

[shawnhar]

Hi, I will ask the DirectInk team about this. Note the question doesn't actually have anything to do with Win2D - you can remove the CanvasControl entirely and get the same result using only stock XAML controls.

[kellylawson]

Ok, thanks. I'll see if I can figure out a way to get it to work, I really appreciate you reaching out.

[kellylawson]

Feel free to close this issue since its external to Win2d. I'm working on another solution, catching mutli touch movements in pointer events and figuring out when it's a scale/pan event, then calling ChangeView on the scrollviewer to account for the gesture. I don't think there will be a better solution at present.

[shawnhar]

Thanks. I think you're right - there's no sense keeping this open as a Win2D issue since the issue is external to Win2D.

I did confirm with the inking team that this is a limitation of the current InkCanvas implementation. Once it has been set to listen to touch events, it will process all such events so they are no longer passed through to parent controls for manipulations such as zoom or pan. The relevant team is aware of your input so they can take that into account in future planning.

For now, I think your options are to process your own multitouch events as you mention above, or design the UI with different modes for pan/zoom vs. drawing ink.

Thanks for the report!

[FranciszMicrosoft]

Hi Kelly,

I'm the program manager for InkPresenter, and Shawn is indeed correct. Currently we don't support 1-finger inking, 2-fingers manipulate in InkPresenter, this is an item on our backlog. Our current recommendation for apps is to associate a UI control (typically a toggle button) with InkPresenter.InputDeviceTypes, so the button toggles whether CoreInputDeviceTypes.Touch is set or not.

Your proposed work around (catching multi-touch pointer events and figure out whether it's scale/pan) might work, but it'll require quite a bit of app code. When you use InkCanvas/InkPresenter, the OS gets first crack at all input set for inking (configured through InkputDeviceTypes). So if you set finger to ink, InkPresenter will get the touch input, and application won't see the touch events at all unless you intercept the input on the background inking thread through CoreInkIndependentInputSource handlers. And each finger will generate separate Pointer***ing events, without the ability to reference each other, so the application has to take care of tracking how many fingers are down, and mark PointerEventArgs.Handled to cancel the ink stroke already drawn by the first finger. In theory this should work, but like I said, it'll involve a lot of app code and will be complicated.

Ultimately we do want to make this a platform supported scenario, in the meantime I suggest you to go with manual mode toggle. If you have any questions, feel free to contact me at DirectInk@microsoft.com.

cheers!

[kellylawson]

Hi,

I greatly appreciate the confirmation. I did go down the path of getting the workaround going, and it functions but it's not as smooth as I would like so I would love to see native platform support in the future. Since I am not using the native inking from InkPresenter (I wanted to modify the texture applied to the line to get something like pencil on paper, so I am using Win2D to draw the textured strokes myself and setting the InkPresenter.InputProcessingConfiguration.Mode to None) I am getting the UnprocessedInput Pointer events already, but on the UI thread. I didn't ever try intercepting drawing events on the background thread since I'm killing that functionality, although if I understood that better there might be a more performant solution there somewhere.

I'm tracking the pointers and publishing the events to a GestureRecognizer when there are two pointers (I only support zoom and 2 finger pan). I noticed that the events for each pointer come separately, but the PointerPoints are organized in frames to make it easy to track. Unfortunately, the GestureRecognizer has a nasty habit of throwing an exception if the points from one pointer are processed before I try to process the corresponding points for the second pointer. I tried to structure my code to minimize the chances of this happening, but since the GestureRecognizer.ProcessMoveEvents doesn't seem to be completely synchronous it still intermittently throws an exception with the message "Input data cannot be processed in the non-chronological order". Found some links online from people with the same problem, but no solution other than burying the exception and moving on. I then manually manipulate the ScrollViewer to zoom or pan based on the input.

It's not perfect, but I created a gist for it here in case it helps someone else.

Thanks a ton for the involvement you guys have here, it's been invaluable for me in learning these APIs.

[mjracca]

Sorry for keeping this thread alive since I know it doesn't belong here.

I just wanted to say that this is an issue for us as well, and would love to see this working. @FranciszMicrosoft do you have a timeline for this?

@kellylawson thanks a lot for the code!

I also posted this question in the WinUI GitHub, if you want to follow up in something new. It is basically the same but in a ScrollViewer, since we currently have drawing disabled to be able to interact with it and not draw.

[EdiWang]

@FranciszMicrosoft

I have the code to just zoom the ink but not the InkCanvas, the ink will be moved with 1 finger or zoom / pan / rotate with 2 fingers. However I also want a way to use a scrollviewer to let the user zoom the inkcanvas, not the ink itself, so that they can do percise drawings. InkCanvas is not eating up the touch events regardless of it's ManipulationMode.

var scale = Matrix3x2.CreateScale(e.Delta.Scale);
var transform = Matrix3x2.CreateTranslation((float)-e.Position.X, (float)-e.Position.Y) *
                scale *
                Matrix3x2.CreateRotation((float)(e.Delta.Rotation / 180 * Math.PI)) *
                Matrix3x2.CreateTranslation((float)e.Position.X, (float)e.Position.Y) *
                Matrix3x2.CreateTranslation((float)e.Delta.Translation.X, (float)e.Delta.Translation.Y);

foreach (var stroke in Ink.InkPresenter.StrokeContainer.GetStrokes())
{
    var attr = stroke.DrawingAttributes;

    // Fix for pencil storke movement blowup. Avoid being 1 stared in the store.
    if (attr.Kind != InkDrawingAttributesKind.Pencil)
    {
        attr.PenTipTransform *= scale;
        stroke.DrawingAttributes = attr;
    }

    stroke.PointTransform *= transform;
}