Synch aircraft across a networked multi-pc setup

[TwinFan]

Current Situation / Problem In a networked multi-PC setup several only loosely coupled instances of X-Plane run in parallel, usually to drive different screens. Running LiveTraffic or any other XPMP2-based plugin will not guarantee that all planes are always fully aligned and placed at the exact same location. This will look ugly when planes move across screens driven by different PCs.

Suggested Solution Synchronize plane objects across the network, driven by a master. Basic design ideas:

• [x] There are masters (the actual XPMP2-based plugins) disseminating position information via UDP multicst into the local network
• [x] There is one centrally provided client plugin per networked PC, to be developed as part of the XPMP2 project, that reads those multicast messages and displays the aircraft in the networked PCs.
• [x] The XPMP2 master shall be independent of the X-Plane master. This would allow to run the XPMP2 master on another of the networked PCs for load balancing. Or to use the same principle for networked gaming setup with several totally independent players, which still could share the same XPMP2-driven planes.
• [x] It shall be possible to collect multicast data from different XPMP2-driven plugins, even if they run on different PCs. So the collecting client needs to be able to identify different sources and map their aircraft ids in case they are in conflict. These are in principle ideas also valid for the TCAS Concentrator #16, so parts of the code will lay the basis for that one, too.
• [x] Standard setup shall be "zero-config": The masters will first listen to the multicast port. As soon as they receive something they will start broadcasting their own data, too. The clients will first send a "want-to-have-data" message on the multicast address and repeat that every once in a while until they eventually receive actual data.
• [x] Masters send their configuration regularly.
  • [x] The client needs to replicate relevant configuration settings like logging level, replace dataRefs/textures, label and map settings; as different masters could be configured differently the client needs to decide for a "best effort" config for each of these items
  • [x] The client needs to request TCAS/AI if any master had so
• [x] The network messages need to include per place:
  • [x] Position, attitude
  • [x] Model (the client needs to have the same set of CSL models installed, and it still needs to be able to do its own matching in case the requested model is not accessible)
  • [x] Configuration, ie. the array of animation dataRefs
  • [x] To save network bandwidth data compression shall be applied: Full data set only every once in a while (so that clients have a chance to fully synch with a new plane), diff data only inbetween, animation dataRefs only if they really change...
• [x] Special situations that require special handling
  • [x] Change of visibility
  • [x] Change of CSL model
  • [x] Removal of aircraft
  • [x] Stale aircraft (not updated in some time)
  • [x] Stale master (not sending in some time)
• [x] Read/write a config file to be able to change a few parameters and enforce network functionality
  • [x] Read by all XPMP2-based plugins, including Remote Client
  • [x] Example provided as Resources/XPMP2.example.prf
  • [x] Includes switch on/off/auto, port, multicast address, max msg length

Benefits Better looking support for a networked setup

Additional context Originally discussed as issue TwinFan/LiveTraffic#126, but now decided to be solved centrally in XPMP2 and hence moved here.

[TwinFan]

Technical proof of concept done: The plugin "XPMP2 Remote Client" receives aircraft information from other XPMP2-based plugins and displays planes using the same CSL model at the same location and attitude.

Some selected Log.txt lines involving the new "XPMP2 Remote Client", displaying the 3 circling planes of two XPMP2 Sample-Plugins, one locally, one remotely. Local machine is a Mac, remote machine is a Windows PC:

2:10:51.929 XPMP2_RC: XPMP2 Remote Client 0.01 starting up...
2:10:51.929 RC/XPMP2 DEBUG Remote.cpp:563/RmtRecvMain: Receiver started listening to 239.255.1.1:49788
...
2:10:54.175 XPMP2-Sample/XPMP2 DEBUG Aircraft.cpp:573/CreateInstances: Aircraft 0x000001: Instance created
2:10:54.175 XPMP2-Sample/XPMP2 DEBUG Remote.cpp:428/RmtSendMain: Listening to 239.255.1.1:49788, waiting for someone interested in our data...
2:10:54.175 XPMP2-Sample/XPMP2 DEBUG Aircraft.cpp:573/CreateInstances: Aircraft 0x123456: Instance created
2:10:54.175 XPMP2-Sample/XPMP2 DEBUG Aircraft.cpp:573/CreateInstances: Aircraft 0xABCDEF: Instance created
2:10:54.175 XPMP2-Sample/XPMP2 INFO  Remote.cpp:460/RmtSendMain: Received word from 192.168.178.71:31938, will start sending aircraft data
2:10:53.828 XPMP2_RC DEBUG Client.cpp:237/ClientProcSettings: Received settings from XPMP2-Sample (4) @ 192.168.178.36:31938
2:10:53.828 XPMP2_RC INFO  Client.cpp:248/ClientProcSettings: First data received from XPMP2-Sample @ 192.168.178.36:31938
2:10:56.076 RC/XPMP2 DEBUG Aircraft.cpp:573/CreateInstances: Aircraft 0x000001: Instance created
2:10:56.116 RC/XPMP2 DEBUG Aircraft.cpp:573/CreateInstances: Aircraft 0xABCDEF: Instance created
2:10:56.140 RC/XPMP2 DEBUG Aircraft.cpp:573/CreateInstances: Aircraft 0x123456: Instance created
...
2:11:11.184 XPMP2_RC DEBUG Client.cpp:237/ClientProcSettings: Received settings from XPMP2-Sample (5) @ 192.168.178.71:31938
2:11:11.184 XPMP2_RC INFO  Client.cpp:248/ClientProcSettings: First data received from XPMP2-Sample @ 192.168.178.71:31938
2:11:11.958 RC/XPMP2 DEBUG Aircraft.cpp:204/Create: Replaced duplicate _modeS_id 0x000001 with new unique value 0x000002
2:11:11.958 RC/XPMP2 DEBUG Aircraft.cpp:204/Create: Replaced duplicate _modeS_id 0x123456 with new unique value 0x000003
2:11:11.958 RC/XPMP2 DEBUG Aircraft.cpp:204/Create: Replaced duplicate _modeS_id 0xABCDEF with new unique value 0x000004
2:11:11.958 RC/XPMP2 DEBUG Aircraft.cpp:573/CreateInstances: Aircraft 0x000003: Instance created
2:11:11.958 RC/XPMP2 DEBUG Aircraft.cpp:573/CreateInstances: Aircraft 0x000004: Instance created
2:11:12.096 RC/XPMP2 DEBUG Aircraft.cpp:573/CreateInstances: Aircraft 0x000002: Instance created

I've also recompiled LiveTraffic (no code change necessary at all) and had it feed the busy traffic at KLAX...also works!

0:00:00.000 XPMP2_RC: XPMP2 Remote Client 0.01 starting up...
0:00:00.000 RC/XPMP2 DEBUG Remote.cpp:563/RmtRecvMain: Receiver started listening to 239.255.1.1:49788
...
0:02:05.140 XPMP2_RC DEBUG Client.cpp:237/ClientProcSettings: Received settings from LiveTraffic (3) @ 192.168.178.36:31938
0:02:05.140 XPMP2_RC INFO  Client.cpp:248/ClientProcSettings: First data received from LiveTraffic @ 192.168.178.36:31938
0:02:11.444 RC/XPMP2 DEBUG Aircraft.cpp:573/CreateInstances: Aircraft 0xA8B939: Instance created
0:02:11.444 RC/XPMP2 DEBUG Aircraft.cpp:573/CreateInstances: Aircraft 0xA97A67: Instance created

Don't worry...there's still a bit to go, especially more efficient position updates (currently it's only using full-details message for each position update) and handling of animation dataRefs, which aren't transferred at all at the moment, lot's of special situation handling (currently the planes never go away ;-) )...

[TwinFan]

Works, (bad quality) video available showing how planes move swiftly between screens.

@HellGL, would you be interested in Beta testing? If so, please contact me on Discord (TwinFan#7161, preferred, beta testing is coordinated in a channel of the LiveTraffic server), or via private message on X-Plane.org. I would appreciate the help!

[TwinFan]

Todo list after 8 hour overnight test with 2 X-Plane instances runnin a cross-feed scenario (ie. both instances send and receive: Win as Master, running LiveTraffic at KLAX, XPMP2-Sample, and XPMP2 Remote Client; Max as External Visual running XPMP2-Sample and XPMP2 Remote Client)

• [ ] On the PC side, the Remote Client has 2 occurences of loosing track to a large number of (potentially all?) LiveTraffic planes at 1:10:02 and at 8:12:22 (the latter was just 44 second before X-Plane shut down...but hey...it clearly was before shutdown)
• [ ] On the Mac side:
  • [ ] The Remote code does not handle changes in the network well. The network was, shortly before being shut down, disconnected from LAN at 8:14:11.594. Error messages unable to send broadcast are then expected and OK. But a reconnect via WLAN (different interface!) is not recognized. Suggested solution: Upon exception stop broadcasting altogether and start over so the multicast network object has a chance to reconnect via a different network interface. Need a retry-mechanism, ideally one which retries in growing time periods.
  • [ ] During the shutdown sequence at 8:15:10.711, X-Plane complains about Remote Client being "buggy". Is said "to turn off the Map layer for other planes, but it is not controlling the other planes", which I don't evenknow what that means, and "tried to control TCAS while not owning the AI planes". At that point the Remote Client had already ceded AI control (a few lines earlier in the log), so it is true that it doesn't own the AI planes...but I would need to figure out what it tries to do during shutdown that is not allowed here. This sequence of errors did not appear on the PC side.
• [x] For LiveTraffic (unrelated to actual XPMP2) suppress creation of aircraft, which are to beremoved immediately -> TwinFan/LiveTraffic#201

Logs_XPMP2_Overnight_20201118_19.zip

[HellGL]

I started my test yesterday with 4 clients and had massive problems in my network. After tracking the problem for one day it comes down to XPMP2 flooding the network with short (8 bytes of data, all the same) UDP multicast messages to 239.255.1.1:49788 (>100000 packets per second). Did anybody experience a similar problem? Maybe not even noticing it as a problem, because it just increases network load? Here it caused my networked hardware to loose connection which I immediately recognize.