Cocotron (AppKit) backend rework

LubosD commented 3 years ago

This issue documents how Cocotron's integration to lower API layers should work.

Backends for X11 and Wayland

AppKit should do CGS API calls to interface with the window server. This means that backends from `cocotron/AppKit/.backend` must go. There's a nice reversed collection of CGS API declarations here.
Implement new backends in cocotron/CoreGraphics/*.backend that implement the new CGS interfaces.
- The X11 backend shall use the XInput2 API to receive events.

Event Processing

All events must exist as CGSEventRecords prior to becoming CGEvents (CGEventCreateWithEventRecord()) and later NSEvents (for AppKit apps).

All CGS events run via an event port (CGSGetEventPort())

Call stack of event reception from the window server:

      CoreGraphics`CGEventCreateFromDataAndSource+0xbce
      CoreGraphics`CGSDecodeEventRecord+0x6a
      CoreGraphics`CGSDispatchDatagramsFromStream+0x28f
      CoreGraphics`snarfEvents+0x12a
      CoreGraphics`CGSGetNextEventRecordInternal+0x9f
      CoreGraphics`CGEventCreateNextEvent+0x2c
      HIToolbox`PullEventsFromWindowServerOnConnection(unsigned int, unsigned char)+0x58
      CoreFoundation`__CFMachPortPerform+0x75
      CoreFoundation`CFRunLoopRunSpecific+0xf51
      CoreFoundation`CFRunLoopRunInMode+0x58
      HIToolbox`RunCurrentEventLoopInMode+0x11b
      HIToolbox`ReceiveNextEventCommon+0x176
      HIToolbox`BlockUntilNextEventMatchingListInMode+0x6a
      AppKit`_DPSNextEvent+0x291
      AppKit`-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]+0x80
      AppKit`-[NSApplication run]+0x31b

Another call stack:

1 com.apple.framework.IOKit 0x98832c8e IOHIDEventCreateWithBytes + 136
2 com.apple.framework.IOKit 0x9882f772 IOHIDEventCreateWithData + 61
3 com.apple.CoreGraphics 0x9b145a47 CGEventCreateFromDataAndSource + 3165
4 com.apple.CoreGraphics 0x9b144d88 CGSDecodeEventRecord + 105
5 com.apple.CoreGraphics 0x9b144d15 event_datagram_handler + 71
6 com.apple.CoreGraphics 0x9b14433a CGSDatagramReadStream::dispatch_next_datagram() + 806
7 com.apple.CoreGraphics 0x9b144007 CGSDatagramReadStream::dispatch_datagrams() + 47
8 com.apple.CoreGraphics 0x9b143e19 CGSDatagramReadStreamDispatchDatagramsWithData + 160
9 com.apple.CoreGraphics 0x9b143b2e CGSSnarfAndDispatchDatagrams + 432
10 com.apple.CoreGraphics 0x9b143864 CGSGetNextEventRecordInternal + 84
11 com.apple.CoreGraphics 0x9b1437cb CGEventCreateNextEvent + 39
12 com.apple.HIToolbox 0x927497bd PullEventsFromWindowServerOnConnection(unsigned int, unsigned char, __CFMachPortBoost*) + 99
13 com.apple.HIToolbox 0x92749726 MessageHandler(__CFMachPort*, void*, long, void*) + 52
14 com.apple.CoreFoundation 0x91fadfe8 __CFMachPortPerform + 440
15 com.apple.CoreFoundation 0x91fade15 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 53
16 com.apple.CoreFoundation 0x91fadd7b __CFRunLoopDoSource1 + 523
17 com.apple.CoreFoundation 0x91f9e4b2 __CFRunLoopRun + 2130
18 com.apple.CoreFoundation 0x91f9d9ea CFRunLoopRunSpecific + 394
19 com.apple.CoreFoundation 0x91f9d84b CFRunLoopRunInMode + 123
20 com.apple.HIToolbox 0x92740b5d RunCurrentEventLoopInMode + 259
21 com.apple.HIToolbox 0x92740777 ReceiveNextEventCommon + 163
22 com.apple.HIToolbox 0x9278baca AcquireNextEventInMode + 75
23 com.apple.HIToolbox 0x928db9da _AcquireNextEvent + 58
24 com.apple.HIToolbox 0x928c9324 RunApplicationEventLoop + 225

Call stack of event reception by AppKit:

6   com.apple.CoreFoundation           0x00007fff8e1eded8 CFRunLoopRunSpecific + 296
7   com.apple.HIToolbox                0x00007fff99fa1935 RunCurrentEventLoopInMode + 235
8   com.apple.HIToolbox                0x00007fff99fa1677 ReceiveNextEventCommon + 184
9   com.apple.HIToolbox                0x00007fff99fa15af _BlockUntilNextEventMatchingListInModeWithFilter + 71
10  com.apple.AppKit                   0x00007fff8ba0defa _DPSNextEvent + 1067
11  com.apple.AppKit                   0x00007fff8ba0d32a -[NSApplication _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 454

HIToolbox (Carbon) APIs return the event as an EventRef, which is converted to CGSEventRecord by AppKit using _GetEventPlatformEventRecord() and then to NSEvent using NSEvent _initWithCGSEvent.

NSEvent most likely contains both CGSEventRecord and EventRef: +[NSEvent eventWithCGEvent:] calls CGEventRecordPointer() and CreateEventWithCGEvent() to pass the return values to -[NSEvent _initWithCGSEvent:eventRef:].

Event Lifecycle

The lifecycle of a HID event seems to be:

Receive event from X11/Wayland
Convert it into a CGSEventRecord
Post it to the Mach port returned by CGSGetEventPort()
When the event is read with CGEventCreateNextEvent(), convert it into a CGEventRef
Carbon then converts CGEventRef into an EventRef and stores it in an internal queue
When AppKit calls Carbon to get the next EventRef, it then converts the EventRef into a CGSEventRecord
The CGSEventRecord is converted to an NSEvent and passed through further processing

TheBrokenRail commented 3 years ago

Nice to see Wayland being worked on! What is the reasoning behind rewriting the X11 backend as part of CoreGraphics? Do some programs rely on internal CoreGraphics API calls?

LubosD commented 3 years ago

1) Yes 2) The main reason is to also properly integrate CGEvents and later Carbon.

noah-nuebling commented 1 year ago

Not sure this is useful but I just stepped through CreateEventWithCGEvent() and looking at the argument registers x0 - x8, one of the arguments is of type HIDEvent *, also known as its __bridged type IOHIDEventRef. Maybe that's the type of the event being created. You can find the source code for HIDEvent * in Apple's open source IOKit code.

darlinghq / darling