Ensure Consistent Performance During Interop. Disconnect

[jvillemare]

Problem

Certain systems experience total crashes, or fail to resume after the connection to the Interoperability server has been disconnected.

Identification

Above is the list of the files in the repo identified to be dependent upon the Interop. server for information for obstacles, targets, and more.

What needs to be done now is testing to see how the four different systems, Image Processing, SDA, Competition Viewer, Manual Target Submission System (MTSS, aka GCS:WebUI), act after a disconnect.

What's Known

Three out of the four systems use flask_gcs.py in order to connect to the Interop. server, while Image Processing uses interop_client.py to directly connect to the Interop. server.

Possible Solutions

_For Image Processing that uses interop_client.py:_ @jmoxrox indicates that a possible solution follows the psuedocode of:

init():
   p = null

submit_x_target(): // submit_manual_target, or submit_autonomous_target
   if p is Process:
      if interop connected:
          p.append(target)
          p.submit_targets()
          p = null

   if interop connected:
       interop.submit(target)
   else:
       p = new Process(Queue)
       p.append(target)

Essentially, submitting targets when Interop. is available, and when it's not available, creating a queue of targets, and waiting for an Interop. connection to submit them.

MTSS' manual targets are sent to interop_client.py, and would also need a queue implemented and tested.

_For all other systems using flask_gcs.py:_ A unique solution may be required for all different subsystems that are started and fed information by flask_gcs.py.

• For SDA: @BrightStraightReality has indicated to me that since we're only avoiding stationary obstacles this year. So, an Interop. disconnect doesn't mean SDA should stop, as the information for the stationary obstacles is only gathered once.
• For Competition Viewer: Since this system displays real-time information on moving obstacles, it actively depends on a stable feed of information. What needs to be done is when there's an Interop. disconnect, the last known data from the Interop. is kept alive and displayed on-screen, a notification conveys to the user that Interop. has been disconnected, and keep the drone information (altitude, speed, etc) updated. Once Interop. has been reconnected, start displaying the new Interop. data.
• For MTSS: The control panel displays information about Interop's status (connected or disconnected,) and emergent, airdrop, and off-axis positions. Also, the act of submitting manual targets that are sent to Interop. may be a problem, but, manual targets are actually sent to interop_client.py, so, a queue fix in interop_client.py would resolve the Interop. dependency issue with Image Processing and MTSS.

Work

It would make the most sense for @jmoxrox to implement queues for autonomous (Image Processing) and manual image processing (MTSS) in interop_client.py, as he has made me aware that he is more familiar with creating processes and using managers.

@BrightStraightReality simply needs to ensure that SDA does not fail when Interop. is disconnected, and if it does, inform me what the best action would be.

I will work on fixing Competition Viewer's Interop. disconnect, as I have written all of it.

[jvillemare]

Unfortunately, due to the nature of the tasks I have, I cannot segment the work I need to do into different branches. I have since created the new branch interop-disconnect-and-duplicates-fixes which currently comprises the following changes:

• Ensure that interop_client.py doesn't crash upon a disconnect from Interop.
• Make Competition Viewer retain the mission data it does have after an Interop. disconnect, and update with new mission data upon reconnect. Notify user on disconnect
• Have MTSS (Manual Target Submission System) properly indicate when Interop. is disconnected (both in the control panel, and possibly in a toast notification.)

I plan to possibly enclose fixes for SDA so it continues to run on an Interop. disconnect, but, I'll need to work more with @BrightStraightReality on that in the very near future.

[jvillemare]

The four commits above this comment, 7e3554f, 9b7720f, 99e7d12, d7bb035, make the following changes:

• Properly working and tested disconnect and reconnect handling for interop_client.py,
• Adds HTML5 fix for IE9 below and related old browsers to the local; needs to be offline.
• Adds duplicate target modal and stub functions for MTSS.
• Removes commented out code in main.3.js and main.js.
• Improves Python Docstring documentation and formatting—adds more docs to JavaScript files.
• Tab-spaced formatting in competition_viewer_socket.py.

There are also a few code comments that are being kept as this is a WIP.

[jvillemare]

Major Status Update

Pull Request #137 was successfully merged today, and it has not only brought reconnect fixes to the Interop. client, but also a duplicates handling fix for MTSS. Both of these are explained below.

Interop. Disconnect Handler

Now, whenever there the connection to the Interop. server is disconnected, and some script attempts to use a method inside the interop_client.py file, the interop_client.py instance will wait an amount of time specified in SUASSystem's settings file, under the variable INTEROP_DISCONNECT_RETRY_RATE, and then re-instantiate the connection to the Interop. server and try again until the function is successful.

This has been tested, and works with SDA, Competition Viewer, and MTSS.

After an Interop. disconnect for each subsystem:

• With SDA, there are crashes, and the simulated drone continues flying to it's specified waypoints.
• With Competition Viewer, the mission data and drone data remains displayed; however, the drone's data stops being updated. This is because the competition_viewer_socket.py file that sends the mission and drone data to the Competition Viewer sends both the mission data and the drone data at the same time. The function that does this gets stuck waiting for the mission data from the Interop. client that has been disconnected from the Interop. server, and so, the drone's data is not sent up to the Viewer, as well as the mission data. A future fix for this might include separating the mission data and drone data into two different sockets, or using another websocket library that supports queuing and separate responses to unique websocket messages.
  • With MTSS (GCS:WebUI), the flask_gcs.py script is unaffected, and simply builds a queue of requests from the user's browsers for some Interop. mission data, and continues to allow the submission of targets, regardless. One problem that I was not able to solve in time was getting the Interop. Connection Status pane on the home window in the GCS:WebUI to update to "Disconnected" when Interop. was actually disconnected. A solution for this might involve adding an epoch time stamp on the /get/interop endpoint, and checking to see if it stops updating, and when it stops updating, state that Interop is "Disconnected", but, that will most likely have to be a fix for next year's team.

Image Processing was not tested because it was not fully integrated/working with GPS coordinates and filters until very recently. We are planning to do testing on Image Processing, and see how it handles an Interop. disconnect shortly in the future.

MTSS Duplicate Target Handler

Whenever a target is submitted through MTSS, the flask_gcs.py scripts checks the submitted target against the JSONs of the targets that have already been cropped and submitted to Interop. server, and if the flask_gcs.py finds any targets that match even one characteristic of a submitted target, a list of the previously submitted, potentially matching targets is sent to MTSS, and the Duplicates Targets Detected modal is displayed.

The modal is a little rough with some styling problems that I can't quite seem to fix, and I couldn't properly resize the preview of the current crop and the previously submitted crop to be of the same sizes (due to time and willpower constraints), but, it works, it's functional, and it's easy to use.