make convenient mouse/keyboar/touch nodes

[joreg]

input could be windowinputsource, entity or layer in order to fetch the right device for the window in question. output: mousedevice and position, delta, buttons, while

[joreg]

see also vvvv/VL.StandardLibs#455

[tebjan]

Mouse Keyboard Touch nodes as entities. provide position in UV, hecto pixel top left, DIP? is Y up or down?

[azeno]

TextInput of SDL window not working because of mix of WinForms and SDL. I tested with a little example and once both run in the same thread the text input doesn't work anymore. So we have two options to proceed here:

• Go WinForms 1 ) We change the Stride window backend to WinForms, all events should now work again 2) We use the Stride input event abstraction as our new default -> replace INotification with IInputEvent. Events in windows forms based renderers (like Skia or OpenCV) would need to get translated (the opposite of what's happening now in the SkiaRenderer of VL.Stride)
• Go SDL 1) We need to get rid of the WinForms mainloop we setup once in our internal VLContext class. Instead we would probably setup a SDL mainloop. In any case, a WinForms based renderer wouldn't work anymore, we would need to rewrite Skia and OpenCV renderers. 2) We use the Stride input event abstraction as our new default -> replace INotification with IInputEvent. SDL events would need to be translated to the Stride events
• Go Stride window 1) Again, we'd need to get rid of the WinForms mainloop and rewrit the Skia and OpenCV renderers with stride windows? That even possible? It would at least safe us step 2)

UPDATE finally figured out how to properly bridge win forms and SDL, gives us some pause here at lease, see ae36b346cf552ff57cd1c0fb0edc909e93fad9a0

[tebjan]

This evaluation only focuses on text events but doesn't include other event sources like touch. The reason we used SDL in the first place is to have touch events, which is one of the first things requested by many users. This appears to be a more important event source compared to text input for testing and evaluating stride for starters and beginners at node.

It doesn't appear to be a wise decision to roll that back on the basis of an arguably less important issue. Also, Winforms is untested with our current multi-window renderers.

In general, the best solution is of course getting rid of Winforms, but it will also take more time than we have before node. So, I would postpone this for now and add it to the known bugs since it takes up precious developer time and we currently have more critical issues than text events.

The root of this problem is of course that we have one runtime thread for all running applications in the HDE. As soon as we have one thread or process per app, every app can have whatever event loop they want, as you would expect it from a developer perspective. This would make all this a non-issue.

We should also look into raw input and the custom message loop implementations in the Stride repo, before moving on.

[azeno]

Just for reference - this would be a nice approach maybe:

[gregsn]

• we decided to do this like seen in the screenshot above (as known from Skia Console node)
• decide upon ScreenSpace: orginal stride screen space (uv) or normalized ...
• skia: will IInputSource be transformed by layer-transfom-nodes? or is it the original input source
• dependency skia to stride.
• reevaluate when changing the windowing to stride windowing

[gregsn]

We should reevaluate if we really want to have SceneWindow, TextureWindow and RenderWindow, that share

• the property that they are complete windows
• the problem that they have to somehow push the input device upstream, which not only sounds wrong, but also comes with a lot of hacks and a node-set that isn't modular, but highly interconnected to the Entity, IRenderer, or even Texture

Nothing fits together and when you are done designing the nodes for the entity world, you need to start over for the next system.

However, if we just tear everything apart we don't end up with this mess: In this sketch we have nodes for different tasks, that play well together:

• Panels, Stacks, Window are some widgets from a UI framework. Not related to Stride
• The window, but also those widgets could come with their relative devices (only notifications that reach a panel will flow out of that panel with relative positions)
• we now can feed those input devices into our sub patches over one link of type IInputDevices, Cameras, but also easy-to-use nodes like Mouse would be connected upstream instead of downstream. Straightforward. ThusMouse node doesn't have an Entity or Texture output (makes sense)
• ScenSink, TextureSink are placeholder names for those nodes that take a panel and draw into them